Skip to content

Collection cache to avoid checking the same links many times #86

Open

Description

@NicEastvillage

One of the slow operations is checking if the collection contains a specific task already. This requires communication with Redis and is done for every task we find. We may be able to improve the performance by given each worker a local cache with links that they have already confirmed are in the collection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestfuture-workFeature for the future

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions