Repository navigation
Data sources: display Source Health and Archive status #397
Description
Context
- We currently return all our validated data sources, and many of them have disappointing links. We're going to continue to do that, but surface the
source healthproperty to better communicate when URLs are poor - For each source, we have a primary URL, as well as archive URLs as a fallback.
- On the back end, we're doing work to keep track of a source's
url_healthandarchive_url. We can use these properties to infer whether we should be highlighting a URL's known archives, or calling it "archive only", or otherwise messaging differently.
Requirements
Back end
Front end
- Surface the source's URL health in search results + data source details, near the clickable URL
- consider color coding / icons as a shorthand
- show the range of possible source health options somewhere
- If the source's URL health is not good, and there exists an archive, we can infer that the source is "archive only".
- We should continue to surface these "broken" sources in search results if there is an
archive_url. - We should return the internet archive version of the URL in the API response as though it is the
source_url. This means the front end will not need to be adjusted for the URL to work or be returned in results.
Activity
maxachis commentedon Nov 24, 2025
@josh-chamberlain Here's my thinking on this right now.
In one of my updates to DS Dev, I added the internet archive URLs (where they exist) to both Meta URLs and Data Sources. I could similarly add these as an attribute when users retrieve Data Sources or Meta URLs.
Then, on the frontend, @joshuagraber can retrieve these and display them if the URL lists as broken.
What do you think?
josh-chamberlain commentedon Dec 18, 2025
@maxachis Yeah, I think this is a front end thing—based on the URL's source health being
poor, and an archive being present, it should be considered "archive only". Then, we just need to make sure it's gettable via the API. Either a dedicated GET for archive_only URLs (seems unnecessary, maybe) or they show up on the existing endpoint for GETing (seems like the way to go).So, this relies upon URL health more than it is part of it. Police-Data-Accessibility-Project/data-source-manager#341
maxachis commentedon Dec 19, 2025
@josh-chamberlain Aha! Then perhaps we should close this and make a corresponding issue in pdap.io?
[-]Set up logic for substituting broken URLs for last-functioning archive in data sources.[/-][+]Data sources: surface archive URLs for sources with poor URL health[/+][-]Data sources: surface archive URLs for sources with poor URL health[/-][+]Data sources: display Source Health and Archive status[/+]