Serialbox data versioning & backwards compatibility

[ofuhrer]

We try to keep up-to-date with the latest & greatest version of Serialbox (using dependabot). But we've been receiving...

collecting ... Serialbox: ERROR: error while parsing "/test_data/MetaData-Generator_rank0.json":
serialbox version of MetaData (2.6.1) does not match the version of the library (2.6.0)

...errors (and the opposite where the library is 2.6.1 and the data is 2.6.0) recently. This is a bit of a hassle, especially since actually the data files and json files between 2.6.0 and 2.6.1 are compatible and we are being overly restrictive.

We thus suggest to implement a versioning scheme where minor version (.X.) numbers are increased when data and/or json files are incompatible, but that patch versions (..X) are backwards compatible for reading / writing serialized data.

@havogt What do you think?

[havogt]

Related #111 and #103.

Reading again #103, the goal was actually to have compatibility for reading 2.6.0 archives from 2.6.1 serialbox, but it seems that's not the case.

Pragmatically, I'd go one step further and say minor versions are also compatible, i.e. all 2.x will be forward and backward compatible.

Reasons:

• I think we didn't change the archive in all 2.x versions. (Worst case, very old 2.x archives might not be compatible. But these archives shouldn't be around hopefully.)
• Because of limitation Limitation of version number #111 we can stay on 2.6 only 8 more patch versions.
• I would continue to use semantic versioning, i.e. increase minor version with new features.

Consequences of this approach: we have to move to 3.x for the next archive change. At that point I would implement #103, i.e. a separate version number for the archive. Then we can decide separately how archive versions are forward and backward compatible.

What do you think?

[ofuhrer]

Sorry, I didn't see the other issues and really should've checked.

I think decoupling the version number in the archives from the Serialbox version number is the best way forward. This also allows to quickly just to a diff -r dir1 dir2 in order to check whether data is the same. Introducing the Serialbox version number into the *.json renders this somewhat less straightforward. Let me know if you have cycles to do this, otherwise I could ask someone from the team to take it up.

@rheacangeo Also a good path forward from your point of view?

[havogt]

My proposal is, short term (2.6.2), make all 2.x compatible and only long term do the decoupling (in version 3.0). Ok? I think I could do that.

[ofuhrer]

Awesome!