[BUG] Test for SCS-0123 is not correctly implemented

[mauhau]

First: In the documented standard Swift is optional. But if not present, the test fails.

Second: If Swift is available beside S3 the test expects the same data backend for both APIs:

service-apis-check: PASS
DEBUG: ... conn
DEBUG: ** service-apis-check
DEBUG: ... service_apis_check
DEBUG: ... scs_0123_service_compute
DEBUG: ... services_lookup
DEBUG: ... scs_0123_service_identity
DEBUG: ... scs_0123_service_image
DEBUG: ... scs_0123_service_network
DEBUG: ... scs_0123_service_load_balancer
DEBUG: ... scs_0123_service_placement
DEBUG: ... scs_0123_service_object_store
DEBUG: ... scs_0123_storage_apis
DEBUG: ... scs_0123_swift_s3
ERROR: S3 buckets and Swift containers differ:
S3: ['scs-test-container']
SW: []

This is also not required regarding the standard documentation.

[mbuechse]

It's true that the test too strict. The reason for that is quite complicated, and it has something to do with the fact that presence of S3 is hard to verify automatically (see #1004). Long story short: all our test subjects so far have passed the overly strict test, and once we encounter a test subject that doesn't while being compliant with the text of the standard, we will have to adjust the test. You can tell us why you think that your environment is compliant with the standard and how this can be verified, and I will be happy to adjust the test.

[mbuechse]

As for no. 2, the reason for that is even more complicated -- I have no idea why this is tested. In the section on conformance testing, the standard merely states:

The test will further check whether the object-store endpoint is compatible to s3.

And I don't know what constitutes "compatibility" between object-store and S3. I suppose it could be construed in the way like it's done in the test. But that doesn't explain why this requirement isn't mentioned further up in the standard. It's debatable whether the section on conformance testing is supposed to contain any authoritative rules. On the other hand, it can be argued that the test has some authoritative value because it's been lived practice for so long.

[fkr]

thanks for raising this. We should make sure to address this. I've assigned this to @depressiveRobot in order to make sure this will be sorted out. As @mbuechse mentioned, there is already ongoing work regarding the discoverability of S3.

[depressiveRobot]

We are aware of this problem which is already discussed extensively in #1004. There is also a first draft to fix this in #1093. @mauhau please join the discussion there.

In the meantime, I created PR #1129 to fix the "object-store" test case to be in line with the standard.