macOS tests failed but didn't retry

[dagood]

macOS tests failed (assuming flakiness for now, I hit retry), but the built-in retry (test 2) didn't run. Not sure why that happened:

(There is some special logic involved, this screenshot is probably not enough to diagnose.)

Originally posted by @dagood in #2262 (comment)