feat: Add opam source (https://github.com/ocaml/security-advisories) to testing

[hannesm]

ref #4581

Now that #4582 has been merged...

[another-rex]

/gcbrun

[another-rex]

LGTM! Just a note on the OPAM database entries, it would be better to generate them with indentation rather than all on one line, as it makes it easy for consumers to see the changes to a record in the git history.

[another-rex]

@hannesm Seems like we are getting rate limited when enumerating versions.

A couple of options here:

1. It seems like the records we are getting are pre-enumerated already anyway, so the easiest way is to just skip enumeration for ocaml records and rely on the record itself being correctly enumerated.
2. Apparently we get higher rate limits if we have an API token, though it does mean a bit more complexity on our part.

Let me know if you're happy with option 1., otherwise we can look into option 2.

@michaelkedar fyi

[hannesm]

Thanks @another-rex.

@hannesm Seems like we are getting rate limited when enumerating versions.

I'm sorry to hear.

A couple of options here:

1. It seems like the records we are getting are pre-enumerated already anyway, so the easiest way is to just skip enumeration for ocaml records and rely on the record itself being correctly enumerated.

I'm not sure I fully understand what you mean, but indeed the osv data we generate carries an array of versions that are affected. And this is filled by the tooling that generates the json. So you can rely on that data.

2. Apparently we get higher rate limits if we have an API token, though it does mean a bit more complexity on our part.

Let me know if you're happy with option 1., otherwise we can look into option 2.

As mentioned, if I understand that correctly, I'm fine with 1. I don't know what needs to be done in the osv code to accommodate this option.

[hannesm]

I tried to find the test instance database where I'd be able to see the imported json data etc. -- any chance you can provide me with a way how to browse that data? At https://test.osv.dev I don't see any imported opam advisories.

Thanks.

[hannesm]

Hi, is there anything I can do in respect to:

@another-rex #4632 (comment)

skip enumeration for ocaml records and rely on the record itself being correctly enumerated

Esp. is there an example data source that skips enumeration?

@hannesm asking #4632 (comment)

is there a URL to the testing web UI?

[another-rex]

Sorry for the delay! Yes test.osv.dev should be where it will show up, looking into why this is not happening now.

[another-rex]

Found the issue, the affected_range.repo field is

git+https://github.com/ocaml/opam

which is not valid according to the schema, which has to be something that can be used with git clone. This is causing us to fail to clone the repository, so it's not being processed and published.

Changing it to https://github.com/ocaml/opam should fix this!

[hannesm]

Dear @another-rex, thanks for looking into this. I fixed this with ocaml/security-advisories#14.

not valid according to the schema

Maybe a good idea to explicitly check this in the schema (I've no idea how, since I've never worked on json schema). We validate all generated osv json with the schema before we publish that.

[another-rex]

Thanks!
Looks like everything's showing up now: https://test.osv.dev/list?q=&ecosystem=opam

Let me know if everything looks good to you and I'll merge the prod version.

---

Maybe a good idea to explicitly check this in the schema (I've no idea how, since I've never worked on json schema). We validate all generated osv json with the schema before we publish that.

I'll look into whether it's something we can add to the jsonschema, though I think that might be difficult to represent there. Either way we should add this to the osv-linter, which has a lot more checks that can't be encoded in the schema.

If you want you can replace the jsonschema check with the linter here: https://github.com/ossf/osv-schema/tree/main/tools/osv-linter for a more comprehensive check.

We also run the linter twice a day with the results being published here, sorted by source: https://test.osv.dev/linter

[hannesm]

Let me know if everything looks good to you and I'll merge the prod version.

This looks good to me, thanks a lot. Please merge the prod version.