Update benchmarking.md with paper content

[JanBobolz]

I'd suggest the following:

• don't tell people to use computeSync(). Instead, tell them to do a complete x.getRepresentation(); send(); x = fromRepresentation() cycle, simulating a real application.
• add (a short version of) the paper's discussion on the two benchmark types
• Add the JMH code example from the paper (but a full version of it) to the .md file.
• Maybe weave in some of the counting description from the paper, I thought that was pretty clear and concise.

[rheitjoh]

I just noticed that by putting in the signature = scheme.restoreSignature(signature.getRepresentation()); stuff into the benchmark method, I actually move the blocking computation via getRepresentation into the benchmark method. That might actually mess with the benchmark.

[rheitjoh]

Do you think actually mentioning the whole send and receive thing in there is helpful? You would not include networking in a benchmark like that, right? (I would do networking benchmarks separately). I think it is fine to mention the whole getRepresentation thing just as a mitigation for the lazy evaluation.

[JanBobolz]

I like the send/receive because that’s what you want to measure, right? How much time until there’s a string available to send und how much time between receiving that string und getting the (say, Boolean) result.

If the test case is „create a signature, then verify it“, that’s not actually a useful scenario because you know the verification result already. No need to check.

I also object to the framing that lazy computation is an issue that needs to be worked around by „forcing“ computation.
Our story can either be „we view test cases like this, where you imagine sending a value somewhere“ or „lazy evaluation sucks, so here is the workaround“. A middle ground is perhaps best, where we acknowledge that this is a likely source of benchmarking errors, but also explain how to think about the solution so that it makes sense even if lazy evaluation wasn’t a thing. I see the “send/receive” thing in that explainer role - computation isn’t done until you have bits in your hand.

[rheitjoh]

I like the send/receive because that’s what you want to measure, right? How much time until there’s a string available to send und how much time between receiving that string und getting the (say, Boolean) result.

If the test case is „create a signature, then verify it“, that’s not actually a useful scenario because you know the verification result already. No need to check.

I also object to the framing that lazy computation is an issue that needs to be worked around by „forcing“ computation.
Our story can either be „we view test cases like this, where you imagine sending a value somewhere“ or „lazy evaluation sucks, so here is the workaround“. A middle ground is perhaps best, where we acknowledge that this is a likely source of benchmarking errors, but also explain how to think about the solution so that it makes sense even if lazy evaluation wasn’t a thing. I see the “send/receive” thing in that explainer role - computation isn’t done until you have bits in your hand.

My benchmark example is not meant to measure verification as part of a larger application, though, just purely the verification time (like you might see in the paper for the signature scheme). In that context lazy computation is actually an issue we need to just work around, and trying to justify the mitigations by talking about sending values around seems inappropriate.

If I were to take more of a "Benchmark the signature as if it was part of some larger application" angle, then I would find that more fitting. Does that mean you would prefer the whole page to talk more from that perspective?

[JanBobolz]

But the „mitigation“ is also serializing and deserializing, no? I think all I want is not to have that sound unnatural, but weave it into a sensible story.
I think I fail to see the difference between benchmarking signing + verifying vs. benchmarking verifying. The code is the same in both scenarios (create sig, [serialize, deserialize], verify), just part that is „timed“ is different. Can you clarify? :)

[rheitjoh]

But the „mitigation“ is also serializing and deserializing, no? I think all I want is not to have that sound unnatural, but weave it into a sensible story.

That sounds like we are kind of swiping the issues under the rug, no? What if the user just wants to measure purely the verification algorithm and not the whole sign->serialize->deserialize->verify process? I think it comes down to the fact that we need to inform them in a way that guarantees they won't fall for the lazy evaluation issues. I think the most clear way to do that is to mention them very clearly, instead of creating a nice story where the actual issues might end up being hidden. I can of course write something like "These are the issues, but actually if you benchmark like this, you won't run into them", but then I still have to mention what happens if they don't benchmark like that, i.e. "you still have to do the whole serialize->deserialize thing in any case". Just makes the whole doc unnecessarily complicated in my opinion.

I think I fail to see the difference between benchmarking signing + verifying vs. benchmarking verifying. The code is the same in both scenarios (create sig, [serialize, deserialize], verify), just part that is „timed“ is different. Can you clarify? :)

The part that is timed being different is quite relevant depending on what the user wants to measure and how they want to represent it. Measuring separately gives you more information in case you have, for example, multiple parties verifying each signature. Then measuring signing + verifying as one thing is not very useful. We don't know what the user wants to measure and why.

[rheitjoh]

Ok, we had a discussion where we clarified what we wanted for the "Problem" section. I have integrated that feedback. Now it just remains to add some information on the new exp alg setter methods for DebugGroup etc.