chore(operations): Compare test harness output with nightly/latest#2204
Conversation
Signed-off-by: binarylogic <bjohnson@binarylogic.com>
| args: | | ||
| bash -o pipefail -xc "cd /vector-test-harness \ | ||
| && bin/compare -s vector -c default $(echo '${{ github.event.comment.body }}' | head -n 1 | sed 's|^/test ||') | tee \"$GITHUB_WORKSPACE/output\"" | ||
| && bin/compare -s vector -c default -v nightly/latest -v ${{ steps.cloned-repo-info.outputs.test_harness_vector_version }} $(echo '${{ github.event.comment.body }}' | head -n 1 | sed 's|^/test ||') | tee \"$GITHUB_WORKSPACE/output\"" |
There was a problem hiding this comment.
@MOZGIII I'm curious how you think we should go about carrying over the -c flag to here?
There was a problem hiding this comment.
If we make -c flag settable via env vars, we'll be able to assign it by default to default just like at the test command above.
I'm also thinking about adding a command to the test harness repo that would be dedicated to handling the Github Action invocations. This would better decouple the test harness logic from the Github Action logic - I believe it'd be better if the latter is scoped to just triggering test harness, so that we offload unrelated logic from it as much as possible.
There was a problem hiding this comment.
If we make
-cflag settable via env vars, we'll be able to assign it by default todefaultjust like at the test command above.
Btw, the same applies to -s.
MOZGIII
left a comment
MOZGIII
left a comment
There was a problem hiding this comment.
Looks good! Let's address the default config issue via a separate PR. This change is contained and looks correct, I see no reason why we should hesitate with merging it.
binarylogic
added
domain: ci
…ectordotdev#2204) Signed-off-by: binarylogic <bjohnson@binarylogic.com> Signed-off-by: Brian Menges <brian.menges@anaplan.com>
3 participants
This defaults the test harness output to compare to the
nightly/latestversion. The only outstanding issue here is the use-c defaultsince this is not always the case. For example, #2145 (comment) run a test with the-c big-vmsflag which also needs to be passed to thebin/comparecommand.