OSV mirror aborts and skips all remaining ecosystems when one configured ecosystem has no archive (404) #7909

Current Behavior

Version 5.2.1. The code on main (feba3276d) behaves the same.

The OSV mirror works through the configured ecosystems in list order. If one of them has no archive in the bucket, the download throws on the non-200 response and the whole activity fails. It retries 6 times and then fails terminally. The scheduled run fails the same way every day, and every ecosystem after the broken one in the list is never mirrored. Ecosystems before it keep their progress.

WARN [ActivityTaskWorker] Activity failed; Failing task terminally after 6 attempt(s) [activityName=mirror-vuln-data-source, ...]
java.lang.IllegalStateException: Failed to download advisory archive for ecosystem Docker Hardened Images: GET https://storage.googleapis.com/osv-vulnerabilities/Docker%20Hardened%20Images/all.zip responded with status code 404

Docker Hardened Images, Homebrew and WordPress are listed as defined ecosystems in https://ossf.github.io/osv-schema/#defined-ecosystems, which the config docs point to for the list of available ecosystems. But their archives return 404 and they are not in the bucket's ecosystems.txt. Their advisories are published in separate repositories.

Same cause as #7345 (Bioconductor), where only the error message was improved (#7385).

Steps to Reproduce

1. Enable the OSV data source.
2. Add PyPI, Docker Hardened Images and Wolfi (in this order) to the ecosystems.
3. Run the mirror.
4. PyPI is mirrored. Docker Hardened Images fails with 404, the mirror retries 6 times and fails terminally. Wolfi is never processed.

Expected Behavior

A missing archive for one ecosystem should not stop the others. Either validate the configured ecosystems against the bucket's ecosystems.txt, or log a warning for the 404 and continue with the next ecosystem. Please also document that the schema's ecosystem list is not the list that the bucket serves.

Dependency-Track Version

5.x

Browser

N/A

Checklist

• I have read and understand the contributing guidelines
• I have checked the existing issues for whether this defect was already reported