Skip to content

Fix k6 executor limitations#2005

Open
ushklianik wants to merge 3 commits into
Blazemeter:masterfrom
ushklianik:k6-executor-fixes
Open

Fix k6 executor limitations#2005
ushklianik wants to merge 3 commits into
Blazemeter:masterfrom
ushklianik:k6-executor-fixes

Conversation

@ushklianik

Copy link
Copy Markdown

The vendored K6Executor (bzt/modules/k6.py) was a thin module compared to JMeterExecutor, and broke on real-world usage during work to wire k6 into the same Azure DevOps release pipeline JMeter runs through today. Four issues, found and verified against a real k6 binary and a real production k6 script (not just synthetic fixtures):

  1. No configurable executable path. K6Executor hardcoded the literal command k6 in both its install-check and its k6 run invocation, ignoring any configured modules.k6.path (unlike JMeter, which reads modules.jmeter.path and uses tool_path throughout). This blocked pointing Taurus at a specific vendored k6 binary (e.g. /k6//k6) instead of relying on PATH.

  2. No native way to pass -e KEY=VALUE params or multiple --out targets. Only --vus/--duration/--iterations were generated from Taurus's generic load settings; anything else depended entirely on the fragile settings.cmdline raw-string escape hatch (naive space-split, breaks on values containing spaces). Added first-class modules.k6.env (dict) and modules.k6.outputs (list) settings.

  3. Taurus's generic load defaulting silently broke real k6 scripts that define their own options.scenarios. ScenarioExecutor.get_load() always defaults iterations to 1 when no concurrency/hold-for is configured, but k6 treats ANY CLI load flag (--vus/--duration/ --iterations/--stage) as a signal to discard the script's own options.scenarios in favor of an implicit scenario calling a function literally named default. Reproduced this against a real, production k6 TypeScript script (env-var-driven ramping-vus scenarios, no default export) - it failed with "function 'default' not found in exports" as soon as Taurus added the default --iterations 1. K6Executor.startup() now checks the RAW, un-defaulted load config (get_raw_load()) and skips the whole load-flag block entirely if the user configured nothing, mirroring the existing precedent in bzt/jmx/tools.py (JMXModifier.modify(), which skips rewriting JMX thread groups the same way to avoid clobbering a hand-authored .jmx). Verified live: the same real script now runs to completion end-to-end through bzt, hitting a real HTTPS endpoint with 0% failures and correct JUnit XML output.

  4. K6LogReader only parsed a handful of hardcoded HTTP metrics (http_reqs, http_req_duration/connecting/tls_handshaking/waiting, vus, data_received) from the --out csv= file. k6's own checks metric (pass/fail assertions inside the script) was completely ignored, so a failed check never surfaced through Taurus's passfail/junit-xml reporting. Added parsing of failed checks rows. Note this required two iterations: the first attempt tried to correlate a failed check to its originating http_reqs row via (timestamp, request name), matching how JMeter nests assertionResult inside the same JTL sample element - but empirical testing against a real k6 binary showed k6's checks CSV rows never carry a name tag at all, so that correlation is fundamentally impossible from this data source (and the first implementation also crashed on tailed/multi-batch reads due to position-tracking state being rebuilt on every read instead of persisted). The fix now surfaces each failed check as its own synthetic label (check: <check name>) in Taurus's result stream instead of attempting a correlation the data can't support. Verified live with a real k6 run containing a deliberately failing check: the JUnit XML now shows the failed check as its own testcase with an error, while the underlying HTTP request correctly shows as a clean success.

Custom Trend/Counter/Gauge metrics remain unparsed: KPISet.add_sample takes a fixed 8-field tuple with no slot for arbitrary named submetrics, so forcing custom metrics in would corrupt percentile/throughput math for real HTTP samples. Scoped out deliberately rather than forcing a bad fit.

Verified: 17/17 unit tests in tests/unit/modules/test_k6.py, plus live end-to-end runs via python3 -m bzt against a real k6 v2.0.0 binary - once against a synthetic script with a deliberately failing check hitting a public HTTP target.

Each PR must conform to Developer's Guide.

Uladzislau Shklianik and others added 2 commits July 11, 2026 12:13
The vendored K6Executor (bzt/modules/k6.py) was a thin module compared to
JMeterExecutor, and broke on real-world usage during work to wire k6 into
the same Azure DevOps release pipeline JMeter runs through today. Four
issues, found and verified against a real k6 binary and a real production
k6 script (not just synthetic fixtures):

1. No configurable executable path. K6Executor hardcoded the literal
   command `k6` in both its install-check and its `k6 run` invocation,
   ignoring any configured `modules.k6.path` (unlike JMeter, which reads
   `modules.jmeter.path` and uses `tool_path` throughout). This blocked
   pointing Taurus at a specific vendored k6 binary
   (e.g. PE_Install/k6/<version>/k6-win-v<version>.exe) instead of relying
   on PATH.

2. No native way to pass `-e KEY=VALUE` params or multiple `--out`
   targets. Only `--vus`/`--duration`/`--iterations` were generated from
   Taurus's generic load settings; anything else depended entirely on the
   fragile `settings.cmdline` raw-string escape hatch (naive space-split,
   breaks on values containing spaces). Added first-class
   `modules.k6.env` (dict) and `modules.k6.outputs` (list) settings.

3. Taurus's generic load defaulting silently broke real k6 scripts that
   define their own `options.scenarios`. ScenarioExecutor.get_load()
   always defaults `iterations` to 1 when no concurrency/hold-for is
   configured, but k6 treats ANY CLI load flag (--vus/--duration/
   --iterations/--stage) as a signal to discard the script's own
   `options.scenarios` in favor of an implicit scenario calling a
   function literally named `default`. Reproduced this against a real,
   production k6 TypeScript script (env-var-driven ramping-vus scenarios,
   no default export) - it failed with "function 'default' not found in
   exports" as soon as Taurus added the default `--iterations 1`.
   K6Executor.startup() now checks the RAW, un-defaulted load config
   (get_raw_load()) and skips the whole load-flag block entirely if the
   user configured nothing, mirroring the existing precedent in
   bzt/jmx/tools.py (JMXModifier.modify(), which skips rewriting JMX
   thread groups the same way to avoid clobbering a hand-authored .jmx).
   Verified live: the same real script now runs to completion end-to-end
   through `bzt`, hitting a real HTTPS endpoint with 0% failures and
   correct JUnit XML output.

4. K6LogReader only parsed a handful of hardcoded HTTP metrics
   (http_reqs, http_req_duration/connecting/tls_handshaking/waiting, vus,
   data_received) from the `--out csv=` file. k6's own `checks` metric
   (pass/fail assertions inside the script) was completely ignored, so a
   failed check never surfaced through Taurus's passfail/junit-xml
   reporting. Added parsing of failed `checks` rows. Note this required
   two iterations: the first attempt tried to correlate a failed check to
   its originating http_reqs row via (timestamp, request name), matching
   how JMeter nests assertionResult inside the same JTL sample element -
   but empirical testing against a real k6 binary showed k6's `checks`
   CSV rows never carry a `name` tag at all, so that correlation is
   fundamentally impossible from this data source (and the first
   implementation also crashed on tailed/multi-batch reads due to
   position-tracking state being rebuilt on every read instead of
   persisted). The fix now surfaces each failed check as its own
   synthetic label (`check: <check name>`) in Taurus's result stream
   instead of attempting a correlation the data can't support. Verified
   live with a real k6 run containing a deliberately failing check: the
   JUnit XML now shows the failed check as its own testcase with an
   error, while the underlying HTTP request correctly shows as a clean
   success.

Custom Trend/Counter/Gauge metrics remain unparsed: KPISet.add_sample
takes a fixed 8-field tuple with no slot for arbitrary named submetrics,
so forcing custom metrics in would corrupt percentile/throughput math for
real HTTP samples. Scoped out deliberately rather than forcing a bad fit.

Verified: 17/17 unit tests in tests/unit/modules/test_k6.py, plus live
end-to-end runs via `python3 -m bzt` against a real k6 v2.0.0 binary -
once against a production www-test script hitting a real candidate
environment, once against a synthetic script with a deliberately failing
check hitting a public HTTP target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ushklianik ushklianik changed the title Fix k6 executor limitations blocking ADO pipeline integration Fix k6 executor limitations Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant