conformance: tweak the HTML report to prepare for our future conformance report program #137
Conversation
Always include skipped tests in the HTML report so that we would be able to receive report with skipped tests from implementers in our future conformance report program. Signed-off-by: Arthur Outhenin-Chalandre <[email protected]>
Skipped tests doesn't include SpecRef because it's inside the Specify clause that may be skipped at the CLI level. This commit introduce a function SpecifyWithSpecRef that wrap the Specify function to include the spec in a global map and then restore the report entry within ReportBeforeEach. It doesn't change any functional things in the tests itself but we have to unfortunately indent everything to make up for this new arg being in a new line. Signed-off-by: Arthur Outhenin-Chalandre <[email protected]>
Make sure test report are sorted so that we can easily compare. Note that some description has leading whitespace so we have to trim it to have a sorting that makes sense. Signed-off-by: Arthur Outhenin-Chalandre <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: MrFreezeex The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
k8s-ci-robot
added
cncf-cla: yes
Signed-off-by: Arthur Outhenin-Chalandre <[email protected]>
|
cc @tpantelis @skitt I would imagine this would mainly be useful after we have discussed the details of the report program which not be happening before the 6th of January so no pressure to review this fast or anything like that 😅 (those are probably nice to have without a conformance program too though). And FYI here is an example of an HTML report with every test being skipped: Happy end of the year in the meantime! 🎉 |
2 participants
I am trying to make sure that conformance report that we would potentially start to collect in the future can be easily "compared"/are all coherents.
The main thing here is to be a able to show up skipped tests so that users checking out those conformance reports will be able to spot those instead of not displaying them at all. There was a slight complexity here because the spec ref to the kep was added within the Specify which would be entirely skipped! So I figured an alternative way (check out the second commit for more details).
And the last small tweak is to sort every tests by description instead of the "unstable" order that we currently have.
Also I included the test labels in the conformance report as a little bonus!