ZCP API Smoke Test Design

[ditahkk]

ZCP API Smoke Test Design

Owner: Jennifer (QA Architect) — design and sign-off only
Implementer: Engineer (TBD) — builds from this spec
Schedule: Nightly at 03:00 ET
API base: https://api.zcp.zsoftly.ca/api
Swagger reference: https://api.zcp.zsoftly.ca/api/docs/nimbo

---

Jennifer's role on this project

Jennifer designs, explains, and signs off. She does not write code.

Jennifer does	Engineer does
Define what to test and why	Write the TypeScript test files
Specify expected behaviour per endpoint	Build the shared helpers and fixtures
Explain to the team what a failure means	Wire up the GitHub Actions pipeline
Review PRs for coverage gaps	Maintain the test infrastructure
Approve the phase before the next begins	Fix flaky tests and timeouts

Each phase ends with Jennifer reviewing the results and giving a go/no-go for the next phase.

[ditahkk]

---

Tool: Playwright (@playwright/test)

These are pure HTTP calls — no browser. APIRequestContext is functionally identical to curl.
It sends GET /virtual-machines with a Bearer token and checks the JSON response. No browser
launches. Playwright is chosen as the runner (not Postman) because the tests live as .ts files
in a repo, get reviewed in PRs, and are maintained by engineers who already write TypeScript.
JSON Postman collections are hard to diff and review. If the team prefers Postman for the design
phase, that is fine — but the implementation lives in TypeScript.

---

Running subsets of tests

Not every run needs the full 50-test suite. Tests are tagged so any combination can be run.

Tags

Tag	What it covers	When to run
@critical	Auth + envelope contract (5 tests)	Every deploy, every PR
@read	GET-only tests — no resources created or deleted	Safe to run anytime
@compute	Virtual machine tests	When compute changes ship
@storage	Block volumes + object storage	When storage changes ship
@networking	VPC + public network	When networking changes ship
@kubernetes	Cluster tests (slow — allow 5 min)	Nightly only
@dns	Domain and record tests	When DNS changes ship
@smoke	All 50 tests	Nightly

Run commands

# Only auth (fast — 5 tests, ~10s)
npx playwright test --grep @critical

# Only read-only tests (no side effects — safe on prd)
npx playwright test --grep @read

# Specific service
npx playwright test --grep @compute
npx playwright test --grep @storage
npx playwright test --grep "@networking|@dns"

# Full nightly suite
npx playwright test

# Skip slow tests (everything except kubernetes)
npx playwright test --grep-invert @kubernetes

---

Repository structure

zcp-smoke-tests/
  tests/
    01-auth.spec.ts
    02-discovery.spec.ts
    03-compute.spec.ts
    04-storage.spec.ts
    05-networking.spec.ts
    06-kubernetes.spec.ts
    07-dns.spec.ts
  support/
    client.ts        # typed HTTP wrapper (thin layer over APIRequestContext)
    assertions.ts    # assertEnvelope, assertArray — reusable expect helpers
    cleanup.ts       # resource registry — collects created slugs, deletes in afterAll
    fixtures.ts      # Playwright fixtures: authenticated context, project slug
  playwright.config.ts
  .env.example
  package.json
  tsconfig.json

---

Phase overview

Phase	Days	What Jennifer delivers	What engineer builds
1	Day 1	Auth spec + shared design patterns	support/ helpers + 01-auth.spec.ts + 02-discovery.spec.ts
2	Day 2	Compute test spec	03-compute.spec.ts
3	Day 3	Storage + networking test spec	04-storage.spec.ts + 05-networking.spec.ts
4	Day 4	Kubernetes + DNS test spec	06-kubernetes.spec.ts + 07-dns.spec.ts
5	Day 5	CI/CD review + sign-off	GitHub Actions pipeline + selective run jobs

---

Phase 1 — Foundation and auth

Jennifer's deliverable: Confirm the shared design patterns below are understood. Walk the
engineer through the envelope contract and explain what each helper does and why. Sign off on the
support layer before any test specs are written.

Engineer builds:

• support/client.ts — typed HTTP wrapper
• support/assertions.ts — envelope and array helpers
• support/cleanup.ts — resource registry
• support/fixtures.ts — authenticated request fixture
• playwright.config.ts
• 01-auth.spec.ts (tests 1–5)
• 02-discovery.spec.ts (tests 6–10)

Gate: All 10 tests pass. Jennifer reviews that the envelope assertion, cleanup pattern, and tag
structure are correct before Phase 2 starts.

---

Shared design patterns (Jennifer explains to engineer)

Why a typed client wrapper

Every test calls the API. Without a wrapper, every test manually constructs the URL, sets headers,
and checks the status. With the wrapper, a test reads:

const env = await client.get<VM[]>('/virtual-machines');
assertArray(env);

Instead of:

const res = await request.get(`${BASE}/virtual-machines`, { headers: { ... } });
const env = await res.json();
expect(res.status()).toBe(200);
expect(env.status).toBe('Success');
expect(Array.isArray(env.data)).toBe(true);

The wrapper keeps tests focused on behaviour, not plumbing.

Why a cleanup registry

Tests that create resources must delete them. If the test fails halfway, the cleanup must still
run. The registry collects every created resource path during the test and runs DELETE on all of
them in afterAll, even if earlier tests threw.

// In the test
const vm = await client.post('/virtual-machines', payload);
cleanup.register(`/virtual-machines/${vm.data.slug}`);

// afterAll runs this automatically — test pass or fail
await cleanup.run(client);

Why tags

A nightly full run takes ~20 minutes because Kubernetes cluster creation is slow. A PR check only
needs the 5 @critical auth tests (~10 seconds). Tags let the same test files serve both use
cases with one CLI flag. Jennifer defines which tag goes on which test — the engineer adds them
during implementation.

---

Phase 1 test specs

Auth & contract — 01-auth.spec.ts — tag: @critical

#	Test name	Method	Endpoint	Pass condition
1	valid token is accepted	GET	/regions	HTTP 200, status=Success
2	missing token is rejected	GET	/regions (no header)	HTTP 401, status=Error
3	malformed token is rejected	GET	/regions (Bearer bad)	HTTP 401, status=Error
4	response envelope has all required fields	GET	/regions	status, message, timezone, data all present
5	token authorizes compute endpoints	GET	/virtual-machines	HTTP 200, status=Success

Why these matter: If test 1–3 fail, the token is wrong, revoked, or the auth service is down.
If test 4 fails, the API contract has changed and all other tests will produce garbage results.
Tests 1–5 run on every deploy before any other tests.

---

Discovery — 02-discovery.spec.ts — tags: @read @smoke

#	Test name	Method	Endpoint	Pass condition
6	regions list is reachable	GET	/regions	array, each item has slug and name
7	OS templates are available	GET	/templates	non-empty array
8	compute plans are available	GET	/plans/service/compute	array, each item has slug, cpu, ram, disk
9	smoke-tests project exists	GET	/projects	array includes project with slug smoke-tests
10	networks list is reachable	GET	/networks	HTTP 200, array

Why these matter: Discovery endpoints are prerequisites for compute tests. If /templates
returns empty, POST /virtual-machines will always fail — knowing this upfront saves debugging
time. These are read-only and safe to run against any environment.

---

Phase 2 — Compute

Jennifer's deliverable: Review the 12 compute test specs below. Walk the engineer through the
create → operate → delete lifecycle. Explain what each negative test (tests 14, 15, 17) is
checking and why the error code matters, not just the fact of failure.

Engineer builds: 03-compute.spec.ts (tests 11–22)

Gate: All 22 tests pass (Phase 1 + Phase 2). Cleanup is verified: after the run, no
smoke-vm-* resources remain in the test project.

---

Compute — 03-compute.spec.ts — tags: @compute @smoke

The spec creates smoke-vm-01 in beforeAll, runs all operation tests against it, then deletes
it in afterAll. Tests 14, 15, 17 are negative tests — they must return an error, not a 200.

#	Test name	Method	Endpoint	Pass condition
11	list all VMs	GET	/virtual-machines	HTTP 200, array
12	list VMs filtered by project	GET	/virtual-machines?project=smoke-tests	HTTP 200, array
13	create VM with valid payload	POST	/virtual-machines	HTTP 200/201, data.slug present, data.state present
14	create VM missing required field	POST	/virtual-machines (no name)	HTTP 400/422, status=Error
15	create VM with duplicate name	POST	/virtual-machines (name already exists)	HTTP 409/422, status=Error
16	get VM by slug	GET	/virtual-machines/smoke-vm-01	HTTP 200, data.slug=smoke-vm-01
17	get VM with unknown slug	GET	/virtual-machines/does-not-exist	HTTP 404, status=Error
18	stop VM	PUT	/virtual-machines/smoke-vm-01/stop	HTTP 200, status=Success
19	start VM	PUT	/virtual-machines/smoke-vm-01/start	HTTP 200, status=Success
20	reboot VM	PUT	/virtual-machines/smoke-vm-01/reboot	HTTP 200, status=Success
21	get VM console URL	GET	/virtual-machines/smoke-vm-01/console	HTTP 200, data.url is a non-empty string
22	delete VM	DELETE	/virtual-machines/smoke-vm-01	HTTP 200/204, status=Success

Note on lifecycle order: Tests 18–20 (stop → start → reboot) are order-dependent within this
spec. Run them sequentially, not in parallel. The VM must be running before stop, stopped before
start, running before reboot.

---

Phase 3 — Storage and networking

Jennifer's deliverable: Review the storage and networking specs. Explain to the engineer that
block volumes and object storage are independent services — a failure in one should not block the
other. Walk through the VPC subnet dependency (subnet test depends on VPC existing).

Engineer builds: 04-storage.spec.ts (tests 23–33), 05-networking.spec.ts (tests 34–43)

Gate: All 43 tests pass. No smoke-vol-*, smoke-obj-*, smoke-vpc-*, smoke-net-*
resources remain after the run.

---

Block storage — 04-storage.spec.ts — tags: @storage @smoke

#	Test name	Method	Endpoint	Pass condition
23	list volumes	GET	/volumes	HTTP 200, array
24	create volume	POST	/volumes	HTTP 200/201, data.slug present
25	get volume by slug	GET	/volumes/smoke-vol-01	HTTP 200, correct slug
26	create snapshot from volume	POST	/snapshots	HTTP 200/201, data.slug present
27	list snapshots	GET	/snapshots	HTTP 200, array
28	delete volume	DELETE	/volumes/smoke-vol-01	HTTP 200/204

Object storage — 04-storage.spec.ts (continued) — tags: @storage @smoke

#	Test name	Method	Endpoint	Pass condition
29	list object storage instances	GET	/object-storage	HTTP 200, array
30	create object storage	POST	/object-storage	HTTP 200/201, data.slug present
31	get object storage instance	GET	/object-storage/smoke-obj-01	HTTP 200
32	list access keys	GET	/object-storage/smoke-obj-01/access-keys	HTTP 200, array
33	delete object storage	DELETE	/object-storage/smoke-obj-01	HTTP 200/204

VPC — 05-networking.spec.ts — tags: @networking @smoke

#	Test name	Method	Endpoint	Pass condition
34	list VPCs	GET	/vpcs	HTTP 200, array
35	create VPC	POST	/vpcs	HTTP 200/201, data.slug present
36	get VPC by slug	GET	/vpcs/smoke-vpc-01	HTTP 200, correct slug
37	add subnet to VPC	POST	/vpcs/smoke-vpc-01/subnets	HTTP 200/201
38	list public IPs on VPC	GET	/vpcs/smoke-vpc-01/public-ips	HTTP 200, array
39	delete VPC	DELETE	/vpcs/smoke-vpc-01	HTTP 200/204

Public network — 05-networking.spec.ts (continued) — tags: @networking @smoke

#	Test name	Method	Endpoint	Pass condition
40	list public networks	GET	/networks?type=public	HTTP 200, array
41	create public network	POST	/networks	HTTP 200/201, data.slug present
42	list public IPs on network	GET	/networks/smoke-net-01/public-ips	HTTP 200, array
43	delete public network	DELETE	/networks/smoke-net-01	HTTP 200/204

---

Phase 4 — Kubernetes and DNS

Jennifer's deliverable: Flag to the engineer that Kubernetes cluster creation is async and
slow — tests 45–47 need a 5-minute timeout and a polling wait until state=Running before
proceeding. Confirm the DNS test domain with the platform team before implementing.

Engineer builds: 06-kubernetes.spec.ts (tests 44–47), 07-dns.spec.ts (tests 48–50)

Gate: All 50 tests pass. Jennifer reviews the full nightly run report and signs off.

---

Kubernetes — 06-kubernetes.spec.ts — tags: @kubernetes @smoke

Use test.setTimeout(300_000) (5 minutes) for tests 45–47. Add a polling helper that retries
GET /kubernetes/smoke-k8s-01 until data.state === 'Running' before the stop test.

#	Test name	Method	Endpoint	Pass condition
44	list clusters	GET	/kubernetes	HTTP 200, array
45	create cluster	POST	/kubernetes	HTTP 200/201, data.slug present
46	get cluster — wait for running state	GET	/kubernetes/smoke-k8s-01 (polled)	data.state=Running within 5 min
47	stop cluster	PUT	/kubernetes/smoke-k8s-01/stop	HTTP 200, status=Success

DNS — 07-dns.spec.ts — tags: @dns @smoke

Confirm the test domain with the platform team. Use an internal domain
(e.g., smoke.test.internal) — not a real TLD.

#	Test name	Method	Endpoint	Pass condition
48	list domains	GET	/dns	HTTP 200, array
49	add test domain	POST	/dns	HTTP 200/201, domain slug present
50	list records on domain	GET	/dns/smoke-test-domain/records	HTTP 200, array

---

Phase 5 — CI/CD pipeline

Jennifer's deliverable: Review the pipeline design below. Confirm the job names and schedules
match what the team expects. Confirm the Slack channel for failure alerts. Sign off on the
complete suite before the pipeline is enabled.

Engineer builds: The full GitHub Actions pipeline from this spec.

Gate: One successful nightly run in GitHub Actions with the HTML report artifact attached.
Jennifer reviews the report and confirms everything is green.

---

Pipeline jobs

Three separate jobs — each runnable independently.

Job 1: critical      @critical only  — every push to main, every PR       ~10s
Job 2: service-smoke any single tag  — manually triggered, or on deploy    ~5 min
Job 3: nightly       @smoke (all 50) — every night at 03:00 ET             ~20 min

GitHub Actions workflow

# .github/workflows/smoke.yml
name: ZCP API Smoke Tests

on:
  push:
    branches: [main]
  pull_request:
  schedule:
    - cron: '0 7 * * *'   # 03:00 ET (UTC-4)
  workflow_dispatch:
    inputs:
      tag:
        description: 'Test tag to run (e.g. @compute, @storage, @networking, @smoke)'
        required: false
        default: '@smoke'

jobs:
  critical:
    name: Auth (every push)
    runs-on: ubuntu-latest
    if: github.event_name == 'push' || github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci
      - run: npx playwright test --grep @critical
        env:
          ZCP_SMOKE_TOKEN: ${{ secrets.ZCP_SMOKE_TOKEN }}
          ZCP_SMOKE_PROJECT: ${{ secrets.ZCP_SMOKE_PROJECT }}

  service-smoke:
    name: Service smoke (manual or deploy)
    runs-on: ubuntu-latest
    if: github.event_name == 'workflow_dispatch'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci
      - run: npx playwright test --grep "${{ github.event.inputs.tag }}"
        env:
          ZCP_SMOKE_TOKEN: ${{ secrets.ZCP_SMOKE_TOKEN }}
          ZCP_SMOKE_PROJECT: ${{ secrets.ZCP_SMOKE_PROJECT }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: smoke-report-${{ github.run_id }}
          path: playwright-report/
          retention-days: 14

  nightly:
    name: Full nightly suite
    runs-on: ubuntu-latest
    if: github.event_name == 'schedule'
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci
      - run: npx playwright test --grep @smoke
        env:
          ZCP_SMOKE_TOKEN: ${{ secrets.ZCP_SMOKE_TOKEN }}
          ZCP_SMOKE_PROJECT: ${{ secrets.ZCP_SMOKE_PROJECT }}
          ZCP_SMOKE_REGION: yow
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: nightly-report-${{ github.run_id }}
          path: playwright-report/
          retention-days: 30
      - name: Notify on failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "ZCP nightly smoke FAILED — <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|view report>"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_QA_WEBHOOK }}

Secrets to configure in GitHub

Secret	What it holds
ZCP_SMOKE_TOKEN	API token from the smoke-test account (Profile → API Tokens)
ZCP_SMOKE_PROJECT	Project slug: smoke-tests
SLACK_QA_WEBHOOK	Incoming webhook URL for the QA alerts channel

---

.env.example

ZCP_API_BASE=https://api.zcp.zsoftly.ca/api
ZCP_SMOKE_TOKEN=your-api-token-here
ZCP_SMOKE_PROJECT=smoke-tests
ZCP_SMOKE_REGION=yow

---

Implementation notes for the engineer

1. Verify every endpoint path against the Swagger UI at https://api.zcp.zsoftly.ca/api/docs/nimbo
   before writing any test. The paths in this document come from the public docs and CLI reference
   — the Swagger is the source of truth.

2. All created resources use the smoke- prefix. After every local run verify manually that
   no smoke-* resources remain: GET /virtual-machines, GET /volumes, GET /vpcs, etc.

3. Kubernetes polling. Add a waitForState(client, path, targetState, timeoutMs) helper in
   support/client.ts that polls every 15 seconds. Use it before the stop test and in cleanup.

4. Test ordering within spec files. Tests inside a spec run sequentially. Never use
   test.describe.parallel inside a CRUD spec — the order is load-bearing (create must precede
   get, get must precede delete).

5. DNS domain. Confirm the test domain with the platform team before Phase 4. Do not use a
   real TLD.

[jenniferoko]

PDF work for the Design for Developer.

ZCP_QA_Comprehensive Framework.pdf