Skip to content

RUBRIC.md

Latest commit

 

History

History
93 lines (74 loc) · 5.09 KB

RUBRIC.md

File metadata and controls

93 lines (74 loc) · 5.09 KB
Raw

Interviewer Guide — rlfan Technical Screen

Format: 60 min total. ~5 min intro + brief walkthrough, ~45–50 min coding, ~5–10 min discussion/wrap. The candidate gets CANDIDATE.md + starter/. The solution/ folder is for your calibration only.

Setup checklist (before the interview)

  • Decide and state the AI-tools policy up front; note it on the scorecard either way.
  • Default to --mock mode for the whole session — it's deterministic and fast. If you want a live-API finale, use a sandbox key with tight limits and budget the last 5 minutes for it; never let the interview block on provisioning latency.
  • Have them run the starter once (python fanout.py --count 1 --command "echo hi" --mock) in the first 3 minutes to confirm the environment works.
  • Note: with RLFAN_MOCK_FAIL_RATE set, the mock can also fail shutdown calls. A candidate who retries shutdown (like the reference solution) passes the leak check; one who tries exactly once may get flagged unluckily. Treat a leak warning under failure injection as a discussion prompt, not an automatic ding.

Expected timeline (calibrated against the reference solution)

Minute A strong candidate has...
0–10 Read run_one, asked 1–2 clarifying questions, chosen threads or asyncio, sketched the plan
10–25 Basic concurrent fan-out working on the happy path
25–40 Per-box error isolation + guaranteed teardown; tested with RLFAN_MOCK_FAIL_RATE=0.3
40–50 Timeout handling, and possibly one stretch goal

Finishing core requirements 1–4 with tested failure handling is a hire-signal performance. Stretch goals separate strong from exceptional; nobody needs all of them.

What to probe while they work

Ask these when natural pauses occur; how they reason matters as much as the code.

  1. Concurrency model. Why threads (or asyncio)? Do they know the SDK offers AsyncRunloop? Do they understand where the blocking calls are? Red flag: sequential loop presented as done, or asyncio wrapped around blocking calls.
  2. Teardown guarantees. "What happens if I Ctrl-C your tool mid-run?" Strong answers involve tracking every provisioned ID the moment it exists (not after the command finishes), teardown in finally, and acknowledging the truly unkillable cases (SIGKILL) — great candidates mention a reconciliation sweep, e.g. devboxes.list(status="running") + name-prefix filter on next startup.
  3. Failure semantics. Does a failed box produce a result row or vanish? Do they distinguish "command exited nonzero" from "API call failed"? Do retries (if attempted) distinguish retryable errors (429, connection) from others?
  4. Timeout design. Is it a whole-run deadline or per-box? What happens to in-flight work at deadline — do they still tear down? (Reference solution: deadline on collection, teardown of everything provisioned in finally.)
  5. Reading unfamiliar code. Did they actually read run_one and the mock, or reinvent things the starter already gave them?

Scoring (4 axes, 1–4 each)

Coding & correctness. 4: core reqs done, code runs, failure paths tested with the fail-rate knob. 3: core done with minor gaps (e.g. timeout rough). 2: happy path only, or concurrency subtly broken (shared-state races, swallowed errors). 1: didn't get a concurrent version working.

Systems judgment. 4: teardown is airtight and they can articulate its limits; error taxonomy handled deliberately; sensible answers to probes 2–4 unprompted. 3: sound choices when prompted. 2: hand-waves cleanup or retries-everything-blindly. 1: no failure thinking.

Tool/SDK fluency. 4: navigates api.md/types quickly, uses the right calls (create_and_await_running, execute_sync), leverages the starter. 3: gets there with some flailing. 2: fights the SDK, ignores provided scaffolding. 1: stuck on mechanics throughout.

Communication. 4: narrates tradeoffs, asks sharp clarifying questions about the underspecified parts (per-box vs global timeout, what "success" means). 3: clear when asked. 2: silent grinding. 1: can't explain own code.

Suggested bar: hire at ≥12/16 with no axis at 1; strong hire at ≥14 with Systems judgment = 4.

Common candidate paths (so you can calibrate live)

  • ThreadPoolExecutor + as_completed — the mainstream good answer (see reference).
  • asyncio + AsyncRunloop — equally good; watch that they use the async client rather than blocking calls inside coroutines.
  • run_one in a loop — works but serial; if presented as final, probe whether they understand why requirement 2 fails.
  • Teardown inside the worker only — misses boxes provisioned when the run fails (timeout, Ctrl-C). The distinguishing question: "who owns cleanup, the worker or the orchestrator?"

After the interview

Run their final code yourself with: RLFAN_MOCK_SEED=42 RLFAN_MOCK_FAIL_RATE=0.4 python fanout.py --count 6 --command "echo hi" --mock Deterministic seed → comparable across candidates. Check: exit code correct, one row per requested box, no !! LEAKED line (modulo the shutdown-failure caveat above).