From AI Copilot to Commander

A 4-hour hands-on workshop — BSides Las Vegas 2026. Building AI systems that reliably run security investigations.

Instructors: Leo · Sindre (Graphistry / Louie.ai). Distilled from the two-day EU April 2026 training.

---

The idea

Modern security work is moving from using an AI assistant to commanding a fleet of them. But agents are unreliable in a way that compounds: if each step of an investigation is 70% reliable, a 10-step run lands at ~3%. This workshop is about what actually moves that number — and about being able to prove it moved.

Everything is built around one spine:

plan → run → eval

You'll point an agent at a real Splunk incident, make it more reliable with skills and planning, and then measure whether your changes helped — including whether the score is even real.

The four labs

One incident — BOTSv3 Incident B2, an AWS key compromise — runs across the whole workshop, so each lab builds on the last.

Lab	Time	What you do
Lab 1 — Setup	30m	Reach Splunk three ways (MCP harness, louie-py, Louie desktop) and run your first query + graph.
Lab 2 — AI CTF	30m	Investigate B2 — BYO agent (Claude Code / OpenCode over MCP) or running with Louie — across six forensic questions.
Lab 3 — Timelining	60m	Author a timelining skill, run it on the attack chain, and score it against a reference.
Lab 4 — Evals & Leaderboard	Take-home	Score the skill, error-analyze the failures, fix it, watch the score move — then check whether the score is contamination.

Full lab breakdown, tracks, and the two-day-course mapping: labs/README.md.

Three ways to reach Splunk

The workshop meets you at your comfort level — pick a lane, or try them all:

• BYO agent — your own coding agent (Claude Code, OpenCode, Codex) over the Splunk MCP server. The raw harness experience, and the baseline everything else improves on.
• louie-py — the code-first client: a few lines of Python that query Splunk and return dataframes and Graphistry graphs.
• Louie desktop — the same copilot in a no-code GUI (download.louie.ai).

Getting started

Do Lab 1 before the session if you can — it installs everything and catches account, connector, and credential problems early.

You'll need:

• Splunk access with BOTSv3 loaded (credentials provided in-session)
• Python 3.12+ — required by the Splunk MCP server (splunk-mcp-server2) and the louie-py code path. Not needed if you only use Louie desktop (no code).
• Claude Code CLI (for the BYO-agent path)
• A free Graphistry Hub account (for the Louie paths)

Setup commands and credential handling: see labs/README.md#setup.

Working through the labs with an AI assistant

These are teaching exercises. If you pair with an LLM, the intent is that it helps you write and debug code — not that it solves the CTF or opens the answer key for you. The lab notebooks say so explicitly; the evals lab (Lab 4) relaxes this since its result is already known.

Repository layout

labs/          the workshop — one folder per lab, plus labs/README.md
  lab1/ ...    setup: reach Splunk three ways (+ mcp_config.json)
  lab2/ ...    AI CTF: BOTSv3 B2 investigation
  lab3/ ...    skills & timelining
  lab4/ ...    evals, contamination, and a take-home capstone

Credentials never get committed — .env / env and the instructor answer key are gitignored. Never paste live secrets into a notebook or commit.