Spring Webflux performance issues in 1.57

[mtomik]

Describe the bug

The new co.elastic.apm.agent.reactor.SubscriptionCancelInstrumentation has significant impact on performance in Spring Webflux project, especially when using alongside something that runs with low latency like Redis causing exceptions like this:

Unable to acquire connection! java.util.concurrent.CompletableFuture@18e0306a[Completed exceptionally: java.util.concurrent.CancellationException]Increase connection pool size. Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=null], command: (EXPIRE), params: [[115, 112, 114, 105, 110, 103, 58, 115, 101, 115, ...], 1209900] after 1 of 2 retry attempts
at org.redisson.command.RedisExecutor.lambda$scheduleRetryTimeout$0(RedisExecutor.java:309)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:717)
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:705)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:791)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:501)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:1474)

I tested the throughput of the "non-instrumentation" solution that we discussed here and I saw ~10x improvement in throughput or latency of http calls of the service

Steps to reproduce

Steps to reproduce the behavior:

1. Create Spring Boot 4 Webflux project with some simple API and use cache using Redis in it
2. Compare throughput of the service http calls with and without APM client version 1.57

Expected behavior

Throughput should be very similar ( not 10x diff )

[SylvainJuge]

Thanks for reporting this @mtomik , do you have a sample application to reproduce this ?

For redis, is that using any kind of reactive/asynchronous driver or is it using the simple synchronous client ?

[mtomik]

Thanks for reporting this @mtomik , do you have a sample application to reproduce this ?

For redis, is that using any kind of reactive/asynchronous driver or is it using the simple synchronous client ?

Not yet but sure I can prepare that. Right now, I am aware of only the reactive one. In that sample project I will try both

[mtomik]

here is the sample project - https://github.com/mtomik/apm-spring-benchmark
I have configured it to be able to test:

• reactive vs imperative spring
• lettuce ( default redis client ) vs redisson - that caused the exception above
• different versions of APM - in benchmark-runner/test/agents are all APM versions I used. there is 1.57.1-non-instrument that is just applied the commit mentioned above on latest version from main

then I run AI agent to create benchmark classes and I let him run them locally with all the combinations, and the output is in BENCHMARK_FINDINGS.md

the results table ( throughput - req/s ) shows few things:

• using APM in imperative doesnt represent any significant slowdown, compared to reactive one where the 50% reduction was even on 1.56.0 ( most probably because of long GC pauses due to that leaked subscriptions ) and another 50% in 1.57 due to that new SubscriptionCancelInstrumentation
• between lettuce and redisson client is no big difference
• the non-instrument version improved results even compared to 1.56.0 ( maybe just due to shorter GC pauses when it solved the leak ) but is still slowing down the whole service by ~25%
• Also note that in my API is just one redis call per request, so usually there are multiple redis calls per request, making the impact of it even worse