Add option to run tests in parallel (as default)

[gerbsen]

Since everyone complained about the tests being very slow, I managed to enable the parallel execution of all tests with very little intervention.

[mbuechse]

Removing doesn't work:

$ ./scs-compliance-check.py -v -a kubeconfig=path/to/kubeconfig.yaml -a subject_root=. -s SUBJECT -o report.yaml scs-compatible-kaas.yaml
WARNING: Config not found: path/to/kubeconfig.yaml
CRITICAL: Invalid kube-config file. Expected key current-context in kube-config
CRITICAL: 'e2e-parallel'
Traceback (most recent call last):
  File "/home/mbue/opensrc/scs/standards/Tests/./scs-compliance-check.py", line 376, in <module>
    sys.exit(main(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^
  File "/home/mbue/opensrc/scs/standards/Tests/./scs-compliance-check.py", line 353, in main
    runner.run(script, testcases=sorted(testcases))
  File "/home/mbue/opensrc/scs/standards/Tests/./scs-compliance-check.py", line 210, in run
    args = check.get('args', '').format(**assignment)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'e2e-parallel'

I will return with a proposal.

[mbuechse]

What is the reason for creating the switch? Why not just turn parallelization on once and for all? I think this reasoning should be added somewhere (in the README or as a comment in the YAML).

[gerbsen]

I don't like forcing people to do things the way I think they are "best". I want the option (without coding) to simply disable the parallel execution. if this makes sense or not, is for the user to decide.

I've now made the default to parallel execution. Non-parallel execution can be achieved via adding -a e2e-parallel="--e2e-parallel=false".

[gerbsen]

I don't really understand why this check needs to be done. In which cases would:

1. a "double" (not duplicate?) assignment be made?
2. be a problem? can't we not only propagate one value to the sonobuoy script?

[mbuechse]

This is not the place for defaults, because this script is agnostic of the actual tests (it works the same for iaas and kaas). If we want defaults, we have to place them in the respective yaml file, and that means we have to add this feature.

[gerbsen]

Ah interesting! Haven't really noticed this, but now that you mention it: what was the rational to test k8s and openstack with one script? Imho this adds an unnecessary level of complexity here (which made it quite hard for me to add this simple param). If I think about the discussion for #1100, adding one binary was considered too much there.

Also, why is this not a place for defaults? I saw things like self.quiet = False or self.critical_only = False which looked like reasonable defaults to me. Parallel execution could/should be done in k8s and in OS envs, right?

I don't really see the way forward now, to make this happen by default and be configurable. Could you point me to something acceptable?

[mbuechse]

That's like saying "why are we listing two directories with the same command ls"?
This script can run any test suite, and the test suite must be given via a yaml file.