Skip to content

feat: improve NVD API & datafeed downloading to acknowledge and process downloads in order of completion - #8855

Open
chadlwilson wants to merge 1 commit into
dependency-check:mainfrom
chadlwilson:process-feed-downloads-on-arrival
Open

chadlwilson wants to merge 1 commit into
dependency-check:mainfrom
chadlwilson:process-feed-downloads-on-arrival

Conversation

@chadlwilson

@chadlwilson chadlwilson commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Description of Change

Currently all datafeed downloads are processed and awaited in the submitted order; and the submitted order is based on hash, not logical order.

There's no functional reason they need to be processed in submission order because each feed other than the modified one is for a separate year and subset of CVEs, and the submission order is also currently not consistent. We can acknowledge them and dispatch them for processing one-by-one as they complete using a CompletionService, rather than using static ordering of Futures.

The current behaviour causes really weird stuff like sitting there waiting for slow 2002 and 2003 downloads when the 2004 downloads have already failed, which is quite frustrating when you're trying to debug a problematic proxy or very slow NVD (as is the state right now).

This change

  • makes order of feed downloads deterministically sorted by year 2001, 2002, 2003....etc
  • simplifies the processing, by moving processing dispatch logic to the NvdApiDataSource itself
  • changes to make the downloads fail fast, and process in order of download completion (except for modified, processed at the end)
  • reverts to the previous default of 2 concurrent feed downloads (changed in 2021). The current unreliability does not seem to be affected by concurrency to either GHA (with a cache) or to NVD (direct upstream). With the current default value there is no value to the more complex threading code.
  • uses CompletionService to report NVD API progress more accurately

Related issues

N/A

Have test cases been added to cover the new functionality?

yes

@boring-cyborg boring-cyborg Bot added the core changes to core label Oct 21, 2026
@chadlwilson
chadlwilson requested a balanced review from Copilot October 22, 2026 06:53
@chadlwilson
chadlwilson marked this pull request as ready for review October 22, 2026 06:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Reversed executor pool sizes misapply the configured download limit and processing concurrency cap.

1 open issue
What changed in this PR

This PR refactors NVD datafeed updates to handle downloads in completion order, avoiding waits on earlier downloads while preserving modified-feed processing last.

Changes:

  • Use completion services to coordinate downloads and processing.
  • Separate downloading from processor submission.
  • Sort feed scheduling and mark modified-feed processors for deferred execution.
File Description
core/​src/​main/​java/​org/​owasp/​dependencycheck/​data/​update/​NvdApiDataSource.java Coordinates completion-order handling and processes the modified feed last.
core/​src/​main/​java/​org/​owasp/​dependencycheck/​data/​update/​nvd/​api/​NvdApiProcessor.java Adds a flag identifying processors that should be delayed.
core/​src/​main/​java/​org/​owasp/​dependencycheck/​data/​update/​nvd/​api/​DownloadTask.java Returns an unsubmitted processor after downloading.

Review effort: Balanced

Comment thread core/src/main/java/org/owasp/dependencycheck/data/update/NvdApiDataSource.java Outdated
@chadlwilson
chadlwilson force-pushed the process-feed-downloads-on-arrival branch from ea1f822 to 4c61e02 Compare October 22, 2026 07:03
@boring-cyborg boring-cyborg Bot added the utils changes to utils label Oct 22, 2026
@chadlwilson
chadlwilson force-pushed the process-feed-downloads-on-arrival branch 2 times, most recently from b600d37 to 8bca5fd Compare October 22, 2026 07:16
@chadlwilson
chadlwilson marked this pull request as draft October 22, 2026 07:45
@chadlwilson chadlwilson changed the title feat: improve datafeed downloading to acknowledge and process downloads in order that downloads complete feat: improve NVD API & datafeed downloading to acknowledge and process downloads in order of completion Oct 22, 2026
@chadlwilson
chadlwilson requested a balanced review from Copilot October 22, 2026 09:23
…ownloads as they complete

Currently all downloads are processed only in the submitted order. There's no reason for this because each are for a separate year. We can acknowledge them and process them as they complete using a CompletionService,
rather than using static ordering of Futures.

Signed-off-by: Chad Wilson <29788154+chadlwilson@users.noreply.github.com>
@chadlwilson
chadlwilson force-pushed the process-feed-downloads-on-arrival branch from 04b8beb to 3122eb3 Compare October 22, 2026 09:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Transitive npm findings can overwrite direct-dependency advisory versions, causing vulnerability associations to be lost.

1 open issue
1 resolved since last review

Review effort: Balanced

@chadlwilson
chadlwilson marked this pull request as ready for review October 22, 2026 09:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core changes to core utils changes to utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants