[Feature Request] Filter some failed KaaS Tests

[guettli]

When running the SCS KaaS conformance Tests, some tests fail.

After discussion in today’s Container meeting HedgeDoc, we agreed that a fixed list of tests is allowed to fail.

For every item in that list, there should be a description of why it is okay to fail.

The network tests allow the argument --e2e-skip, but the conformance tests do not.

We suggest running the tests and doing some post-processing.

I had a short look at the code, and I am unsure where to add this.

@mbuechse (or someone else): can we join and find a way to solve that?

Currently, I have the vague idea of parsing JUnit XML files created by sonobuoy.

We could use the Python package junit2html to create a nice report and show which tests are fine, which are okay to fail, and which failed.

The default result of junit2html is not very usable at the moment, because so many tests get skipped (because only a few tests are part of the conformance test list) that looking at the report is not convenient.

We should implement it so that re-running the report generator after scs-compliance-check.py finished is possible. Otherwise, the development inner loop is way too long. I don't want to wait hours :-)

Additionally, I would like to output some timing information. I am curious why the test takes several hours and which test it is.

[mbuechse]

We already parse the XML file, see
https://github.com/SovereignCloudStack/standards/blob/main/Tests/kaas/sonobuoy_handler/sonobuoy_handler.py#L123

I think we should write up a YAML file that lists all tests that are to be downgraded to recommendations, together with a reason for each test, and then we add a command-line switch to supply the YAML file to the sonobuoy handler script.

The timing info would be very interesting indeed. I will have to look into the XML file to see what kind of information is provided there.

[mbuechse]

Just to be clear, we would then change these lines

• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compatible-kaas.yaml#L14
• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compatible-kaas.yaml#L34

so the respective YAML file will be given to the script.

[guettli]

@mbuechse the sonobuoy conformance tests do now allow skipping e2e tests. You get:

Error: mode flag and e2e-focus/skip flags both provided and may cause unintended behavior

Filtering works for the Network tests.

To avoid complex code I suggest that we focus only on post-processing (no e2e-skip).

[mbuechse]

Yes sure, let's do postprocessing. (Who knows what the results can be good for, even if demoted to recommendation or even optional.)

[guettli]

Yes sure, let's do postprocessing. (Who knows what the results can be good for, even if demoted to recommendation or even optional.)

Great, that you see it the same. Postprocessing has the benefit that the report-generation can be re-run after the long running test has finished.

We already parse the XML file, see ...

This code parses sonobuoy_results.yaml, not the junit xml file.

---

I had a the junit xml file.

<testcase 
    name="[It] [sig-network] Netpol NetworkPolicy between server and client should allow ingress access from updated pod [Feature:NetworkPolicy]"
    classname="Kubernetes e2e suite"
     status="passed"
     time="28.067231293">
</testcase>

Looks good. The timing info seems available.

---

BTW, Why not create a nice HTML report at the end?

[mbuechse]

1. You're right. I think we wanted to parse the XML at some point, and when that proved difficult, I found that a YAML file exists and opted for that one. I guess I can change that again -- no problem.
2. What I want in the end is that all relevant information is contained in the report.yaml generated by the main script and then sent to the compliance monitor, where it can then be requested and studied by interested parties. Maybe we can include better visualization into the compliance monitor. Or we say people have to run the conversion to HTML locally because it's a niche use-case -- I don't know.

[guettli]

@mbuechse sounds good. We can do it in these steps:

• long running tests get executed
• Script1 reads junit xml generates report.yaml
• optional: Script2 reads report.yaml and creates nice self-contained html.

Both scripts should be able to be executed again and again without the need to start the long running tests again.

Does this make sense to you?

BTW, currently the -o option is optional (to create the report). I think that should be mandatory. ... Or wait I had a different idea yesterday: The scs-compliance script gets a directory name. This directory must not exist, because we do not want to overwrite any files. In that directory there will be report.yaml. Then the number of arguments gets smaller, because -o is no longer needed.

Currently, you won't get a report.yaml, if you forgot the -o option ...

[mbuechse]

The main use-case for the whole testsuite and machinery is to have it run nightly (edit: regularly, can also be weekly or monthly) in a CI pipeline. Depending on the config, you will always have a dedicated directory for all kinds of artifacts. If, on the other hand, someone wants to test manually, I think it is on them to run the script in a way that suits their purposes. That being said, I do agree that we can (and should) improve the user experience for that scenario. However, from a more holistic perspective, I think the way we should approach these improvements would look a little differently and might need a bit more contemplation and discussion. I will be very happy to work on that topic.

[mbuechse]

(I should add that the Sonobuoy tests are the first tests integrated into this testsuite that run longer than a minute or two, and that produce such a large amount of output. I think overhauling the main script is in order one way or another...)

[janiskemper]

just a general comment: @mbuechse I believe that the tests currently run more often than necessary. I actually don't know the exact schedule right now, but I think this is a topic we can maybe bring up to the SIG standardization or so, to actually have clear picture there (if that clear picture doesn't exist yet).