Fix k6 executor limitations#2005
Open
ushklianik wants to merge 3 commits into
Open
Conversation
The vendored K6Executor (bzt/modules/k6.py) was a thin module compared to JMeterExecutor, and broke on real-world usage during work to wire k6 into the same Azure DevOps release pipeline JMeter runs through today. Four issues, found and verified against a real k6 binary and a real production k6 script (not just synthetic fixtures): 1. No configurable executable path. K6Executor hardcoded the literal command `k6` in both its install-check and its `k6 run` invocation, ignoring any configured `modules.k6.path` (unlike JMeter, which reads `modules.jmeter.path` and uses `tool_path` throughout). This blocked pointing Taurus at a specific vendored k6 binary (e.g. PE_Install/k6/<version>/k6-win-v<version>.exe) instead of relying on PATH. 2. No native way to pass `-e KEY=VALUE` params or multiple `--out` targets. Only `--vus`/`--duration`/`--iterations` were generated from Taurus's generic load settings; anything else depended entirely on the fragile `settings.cmdline` raw-string escape hatch (naive space-split, breaks on values containing spaces). Added first-class `modules.k6.env` (dict) and `modules.k6.outputs` (list) settings. 3. Taurus's generic load defaulting silently broke real k6 scripts that define their own `options.scenarios`. ScenarioExecutor.get_load() always defaults `iterations` to 1 when no concurrency/hold-for is configured, but k6 treats ANY CLI load flag (--vus/--duration/ --iterations/--stage) as a signal to discard the script's own `options.scenarios` in favor of an implicit scenario calling a function literally named `default`. Reproduced this against a real, production k6 TypeScript script (env-var-driven ramping-vus scenarios, no default export) - it failed with "function 'default' not found in exports" as soon as Taurus added the default `--iterations 1`. K6Executor.startup() now checks the RAW, un-defaulted load config (get_raw_load()) and skips the whole load-flag block entirely if the user configured nothing, mirroring the existing precedent in bzt/jmx/tools.py (JMXModifier.modify(), which skips rewriting JMX thread groups the same way to avoid clobbering a hand-authored .jmx). Verified live: the same real script now runs to completion end-to-end through `bzt`, hitting a real HTTPS endpoint with 0% failures and correct JUnit XML output. 4. K6LogReader only parsed a handful of hardcoded HTTP metrics (http_reqs, http_req_duration/connecting/tls_handshaking/waiting, vus, data_received) from the `--out csv=` file. k6's own `checks` metric (pass/fail assertions inside the script) was completely ignored, so a failed check never surfaced through Taurus's passfail/junit-xml reporting. Added parsing of failed `checks` rows. Note this required two iterations: the first attempt tried to correlate a failed check to its originating http_reqs row via (timestamp, request name), matching how JMeter nests assertionResult inside the same JTL sample element - but empirical testing against a real k6 binary showed k6's `checks` CSV rows never carry a `name` tag at all, so that correlation is fundamentally impossible from this data source (and the first implementation also crashed on tailed/multi-batch reads due to position-tracking state being rebuilt on every read instead of persisted). The fix now surfaces each failed check as its own synthetic label (`check: <check name>`) in Taurus's result stream instead of attempting a correlation the data can't support. Verified live with a real k6 run containing a deliberately failing check: the JUnit XML now shows the failed check as its own testcase with an error, while the underlying HTTP request correctly shows as a clean success. Custom Trend/Counter/Gauge metrics remain unparsed: KPISet.add_sample takes a fixed 8-field tuple with no slot for arbitrary named submetrics, so forcing custom metrics in would corrupt percentile/throughput math for real HTTP samples. Scoped out deliberately rather than forcing a bad fit. Verified: 17/17 unit tests in tests/unit/modules/test_k6.py, plus live end-to-end runs via `python3 -m bzt` against a real k6 v2.0.0 binary - once against a production www-test script hitting a real candidate environment, once against a synthetic script with a deliberately failing check hitting a public HTTP target. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The vendored K6Executor (bzt/modules/k6.py) was a thin module compared to JMeterExecutor, and broke on real-world usage during work to wire k6 into the same Azure DevOps release pipeline JMeter runs through today. Four issues, found and verified against a real k6 binary and a real production k6 script (not just synthetic fixtures):
No configurable executable path. K6Executor hardcoded the literal command
k6in both its install-check and itsk6 runinvocation, ignoring any configuredmodules.k6.path(unlike JMeter, which readsmodules.jmeter.pathand usestool_paththroughout). This blocked pointing Taurus at a specific vendored k6 binary (e.g. /k6//k6) instead of relying on PATH.No native way to pass
-e KEY=VALUEparams or multiple--outtargets. Only--vus/--duration/--iterationswere generated from Taurus's generic load settings; anything else depended entirely on the fragilesettings.cmdlineraw-string escape hatch (naive space-split, breaks on values containing spaces). Added first-classmodules.k6.env(dict) andmodules.k6.outputs(list) settings.Taurus's generic load defaulting silently broke real k6 scripts that define their own
options.scenarios. ScenarioExecutor.get_load() always defaultsiterationsto 1 when no concurrency/hold-for is configured, but k6 treats ANY CLI load flag (--vus/--duration/ --iterations/--stage) as a signal to discard the script's ownoptions.scenariosin favor of an implicit scenario calling a function literally nameddefault. Reproduced this against a real, production k6 TypeScript script (env-var-driven ramping-vus scenarios, no default export) - it failed with "function 'default' not found in exports" as soon as Taurus added the default--iterations 1. K6Executor.startup() now checks the RAW, un-defaulted load config (get_raw_load()) and skips the whole load-flag block entirely if the user configured nothing, mirroring the existing precedent in bzt/jmx/tools.py (JMXModifier.modify(), which skips rewriting JMX thread groups the same way to avoid clobbering a hand-authored .jmx). Verified live: the same real script now runs to completion end-to-end throughbzt, hitting a real HTTPS endpoint with 0% failures and correct JUnit XML output.K6LogReader only parsed a handful of hardcoded HTTP metrics (http_reqs, http_req_duration/connecting/tls_handshaking/waiting, vus, data_received) from the
--out csv=file. k6's ownchecksmetric (pass/fail assertions inside the script) was completely ignored, so a failed check never surfaced through Taurus's passfail/junit-xml reporting. Added parsing of failedchecksrows. Note this required two iterations: the first attempt tried to correlate a failed check to its originating http_reqs row via (timestamp, request name), matching how JMeter nests assertionResult inside the same JTL sample element - but empirical testing against a real k6 binary showed k6'schecksCSV rows never carry anametag at all, so that correlation is fundamentally impossible from this data source (and the first implementation also crashed on tailed/multi-batch reads due to position-tracking state being rebuilt on every read instead of persisted). The fix now surfaces each failed check as its own synthetic label (check: <check name>) in Taurus's result stream instead of attempting a correlation the data can't support. Verified live with a real k6 run containing a deliberately failing check: the JUnit XML now shows the failed check as its own testcase with an error, while the underlying HTTP request correctly shows as a clean success.Custom Trend/Counter/Gauge metrics remain unparsed: KPISet.add_sample takes a fixed 8-field tuple with no slot for arbitrary named submetrics, so forcing custom metrics in would corrupt percentile/throughput math for real HTTP samples. Scoped out deliberately rather than forcing a bad fit.
Verified: 17/17 unit tests in tests/unit/modules/test_k6.py, plus live end-to-end runs via
python3 -m bztagainst a real k6 v2.0.0 binary - once against a synthetic script with a deliberately failing check hitting a public HTTP target.Each PR must conform to Developer's Guide.