feat: "hedging": issue two concurrent requests for HEAD and GET

[danking]

There's a reasonable argument that this should either (a) be implemented via an object_store::HttpService or (b) implemented as a reqwest middleware. Either of those approaches can be upstreamed or made a separate open source library.

From the AnyBlob paper, Durner et al. "Exploiting Cloud Object Storage for High-Performance Analytics". VLDB Vol 16 Iss 11:

2.7 Tail Latency & Request Hedging

Hedging against slow responses. Missing or slow responses from storage servers are a challenge for
users of cloud object stores. In our latency experiments, we see requests that have a considerable
tail latency. Some requests get lost without any notice. To mitigate these issues, cloud vendors
suggest restarting unresponsive requests,known as request hedging [ 10 , 34 ]. For example, the
typical 16 MiB request duration is below 600ms for AWS. However, less than 5% of objects are not
downloaded after 600ms. Missing responses can also be found by checking the first byte
latency. Similarly to the duration, less than 5% have a first byte latency above 200ms. Hedging
these requests does not introduce significant cost overhead.

[codspeed-hq]

Merging this PR will improve performance by 10.45%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
✅ 1160 untouched benchmarks
⏩ 1323 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	WallTime	u64_FoR[10M]	387.9 µs	351.2 µs	+10.45%

---

_{Comparing dk/hedge-object-store (70ce09f) with develop (83664a0)}

Footnotes

1. 1323 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[AdamGS]

happy to play with it here, but I'm in favor of making it its own crate so its easier to share around, happy to pair on this once I wrap up some other stuff.