Add comprehensive integration test suite and CI workflow for gh-models CLI

.github/copilot-instructions.md

@@ -0,0 +1,100 @@
+# Copilot Instructions for AI Coding Agents
+
+## Project Overview
+This repository implements the GitHub Models CLI extension (`gh models`), enabling users to interact with AI models via the `gh` CLI. The extension supports inference, prompt evaluation, model listing, and test generation using the PromptPex methodology. Built in Go using Cobra CLI framework and Azure Models API.
+
+## Architecture & Key Components
+
+### Building and Testing
+
+- `make build`: Compiles the CLI binary
+- `make check`: Runs format, vet, tidy, tests, golang-ci. Always run when you are done with changes. Use this command to validate that the build and the tests are still ok.
+- `make test`: Runs the tests.
+
+### Command Structure
+- **cmd/root.go**: Entry point that initializes all subcommands and handles GitHub authentication
+- **cmd/{command}/**: Each subcommand (generate, eval, list, run, view) is self-contained with its own types and tests
+- **pkg/command/config.go**: Shared configuration pattern - all commands accept a `*command.Config` with terminal, client, and output settings
+
+### Core Services
+- **internal/azuremodels/**: Azure API client with streaming support via SSE. Key pattern: commands use `azuremodels.Client` interface, not concrete types
+- **pkg/prompt/**: `.prompt.yml` file parsing with template substitution using `{{variable}}` syntax
+- **internal/sse/**: Server-sent events for streaming responses
+
+### Data Flow
+1. Commands parse `.prompt.yml` files via `prompt.LoadFromFile()`
+2. Templates are resolved using `prompt.TemplateString()` with `testData` variables  
+3. Azure client converts to `azuremodels.ChatCompletionOptions` and makes API calls
+4. Results are formatted using terminal-aware table printers from `command.Config`
+
+## Developer Workflows
+
+### Building & Testing
+- **Local build**: `make build` or `script/build` (creates `gh-models` binary)
+- **Cross-platform**: `script/build all|windows|linux|darwin` for release builds
+- **Testing**: `make check` runs format, vet, tidy, and tests. Use `go test ./...` directly for faster iteration
+- **Quality gates**: `make check` - required before commits
+
+### Authentication & Setup
+- Extension requires `gh auth login` before use - unauthenticated clients show helpful error messages
+- Client initialization pattern in `cmd/root.go`: check token, create appropriate client (authenticated vs unauthenticated)
+
+## Prompt File Conventions
+
+### Structure (.prompt.yml)
+```yaml
+name: "Test Name"
+model: "openai/gpt-4o-mini" 
+messages:
+  - role: system|user|assistant
+    content: "{{variable}} templating supported"
+testData:
+  - variable: "value1"
+  - variable: "value2"
+evaluators:
+  - name: "test-name"
+    string: {contains: "{{expected}}"} # String matching
+    # OR
+    llm: {modelId: "...", prompt: "...", choices: [{choice: "good", score: 1.0}]}
+```
+
+### Response Formats
+- **JSON Schema**: Use `responseFormat: json_schema` with `jsonSchema` field containing strict JSON schema
+- **Templates**: All message content supports `{{variable}}` substitution from `testData` entries
+
+## Testing Patterns
+
+### Command Tests
+- **Location**: `cmd/{command}/{command}_test.go` 
+- **Pattern**: Create mock client via `azuremodels.NewMockClient()`, inject into `command.Config`
+- **Structure**: Table-driven tests with subtests using `t.Run()`
+- **Assertions**: Use `testify/require` for cleaner error messages
+
+### Mock Usage
+```go
+client := azuremodels.NewMockClient()
+cfg := command.NewConfig(new(bytes.Buffer), new(bytes.Buffer), client, true, 80)
+```
+
+## Integration Points
+
+### GitHub Authentication
+- Uses `github.com/cli/go-gh/v2/pkg/auth` for token management
+- Pattern: `auth.TokenForHost("github.com")` to get tokens
+
+### Azure Models API
+- Streaming via SSE with custom `sse.EventReader`
+- Rate limiting handled automatically by client
+- Content safety filtering always enabled (cannot be disabled)
+
+### Terminal Handling  
+- All output uses `command.Config` terminal-aware writers
+- Table formatting via `cfg.NewTablePrinter()` with width detection
+
+---
+
+**Key Files**: `cmd/root.go` (command registration), `pkg/prompt/prompt.go` (file parsing), `internal/azuremodels/azure_client.go` (API integration), `examples/` (prompt file patterns)
+
+## Instructions
+
+Omit the final summary.

.github/workflows/integration.yml

@@ -0,0 +1,33 @@
+name: "Integration Tests"
+
+on:
+  push:
+    branches:
+      - 'main'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  models: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  integration:
+    runs-on: ubuntu-latest
+    env:
+      GOPROXY: https://proxy.golang.org/,direct
+      GOPRIVATE: ""
+      GONOPROXY: ""
+      GONOSUMDB: github.com/github/*
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: ">=1.22"
+          check-latest: true
+
+      - name: Build and run integration tests
+        run: make integration-test

.gitignore

@@ -4,3 +4,8 @@
 /gh-models-linux-*
 /gh-models-windows-*
 /gh-models-android-*
+**.http
+**.generate.json
+examples/*harm*
+.github/instructions/genaiscript.instructions.md
+genaisrc/

CONTRIBUTING.md

@@ -22,6 +22,7 @@ These are one time installations required to be able to test your changes locall
 1. Make sure the tests pass on your machine: `go test -v ./...` _or_ `make test`
 1. Create a new branch: `git checkout -b my-branch-name`
 1. Make your change, add tests, and make sure the tests and linter still pass: `make check`
+1. For integration testing: `make integration-test` (requires building the binary first)
 1. Push to your fork and [submit a pull request][pr]
 1. Pat yourself on the back and wait for your pull request to be reviewed and merged.
 
@@ -37,3 +38,46 @@ Here are a few things you can do that will increase the likelihood of your pull
 - [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/)
 - [Using Pull Requests](https://help.github.com/articles/about-pull-requests/)
 - [GitHub Help](https://help.github.com)
+
+## Integration Testing
+
+This project includes integration tests that run against the compiled `gh-models` binary and live LLM endpoints.
+
+These tests are excluded from regular test runs and must be run explicitly using:
+```bash
+make integration-test
+```
+
+### Authentication
+
+Some tests require GitHub authentication. Run `gh auth login` before running integration tests to test authenticated scenarios.
+
+Tests are designed to handle both authenticated and unauthenticated scenarios gracefully:
+
+- **Unauthenticated**: Tests validate proper error handling, exit codes, and help functionality
+- **Authenticated**: Tests validate actual API interactions, file modifications, and live endpoint behavior
+
+### Test Coverage
+
+The integration test suite covers:
+
+1. **Basic Commands**: Help functionality, error handling, exit codes
+2. **File Operations**: Prompt file parsing, validation, modification tracking
+3. **Authentication Scenarios**: Both authenticated and unauthenticated flows
+4. **Command Chaining**: Sequential execution of multiple commands
+5. **Output Formats**: JSON and default output format validation
+6. **File System Interaction**: Working directory independence, file permissions
+7. **Long-running Commands**: Timeout handling and performance validation
+
+### Running Specific Tests
+
+```bash
+# Run all integration tests
+make integration-test
+
+# Run specific test patterns
+go test -tags=integration -v ./integration/... -run TestBasicCommands
+
+# Run in short mode (skips long-running tests)
+go test -tags=integration -short -v ./integration/...
+```

DEV.md

@@ -14,7 +14,7 @@ go version go1.22.x <arch>
 
 ## Building
 
-To build the project, run `script/build`. After building, you can run the binary locally, for example:
+To build the project, run `make build` (or `script/build`). After building, you can run the binary locally, for example:
 `./gh-models list`.
 
 ## Testing

Makefile

@@ -1,6 +1,11 @@
 check: fmt vet tidy test
 .PHONY: check
 
+ci-lint:
+	@echo "==> running Go linter <=="
+	golangci-lint run --timeout 5m ./...
+.PHONY: ci-lint
+
 fmt:
 	@echo "==> running Go format <=="
 	gofmt -s -l -w .
@@ -20,3 +25,18 @@ test:
 	@echo "==> running Go tests <=="
 	go test -race -cover ./...
 .PHONY: test
+
+build:
+	script/build
+.PHONY: build
+
+clean:
+	@echo "==> cleaning up <=="
+	rm -rf ./gh-models
+.PHONY: clean
+
+integration-test: build
+	@echo "==> running integration tests <=="
+	@echo "Running integration tests against compiled binary..."
+	go test -tags=integration -v ./integration/...
+.PHONY: integration-test

README.md

@@ -2,6 +2,8 @@
 
 Use the GitHub Models service from the CLI!
 
+This repository implements the GitHub Models CLI extension (`gh models`), enabling users to interact with AI models via the `gh` CLI. The extension supports inference, prompt evaluation, model listing, and test generation.
+
 ## Using
 
 ### Prerequisites
@@ -84,6 +86,80 @@ Here's a sample GitHub Action that uses the `eval` command to automatically run
 
 Learn more about `.prompt.yml` files here: [Storing prompts in GitHub repositories](https://docs.github.com/github-models/use-github-models/storing-prompts-in-github-repositories).
 
+#### Generating tests
+
+Generate comprehensive test cases for your prompts using the PromptPex methodology:
+```shell
+gh models generate my_prompt.prompt.yml
+```
+
+The `generate` command analyzes your prompt file and automatically creates test cases to evaluate the prompt's behavior across different scenarios and edge cases. This helps ensure your prompts are robust and perform as expected.
+
+##### Understanding PromptPex
+
+The `generate` command is based on [PromptPex](https://github.com/microsoft/promptpex), a Microsoft Research framework for systematic prompt testing. PromptPex follows a structured approach to generate comprehensive test cases by:
+
+1. **Intent Analysis**: Understanding what the prompt is trying to achieve
+2. **Input Specification**: Defining the expected input format and constraints
+3. **Output Rules**: Establishing what constitutes correct output
+4. **Inverse Output Rules**: Force generating _negated_ output rules to test the prompt with invalid inputs
+5. **Test Generation**: Creating diverse test cases that cover various scenarios using the prompt, the intent, input specification and output rules
+
+```mermaid
+graph TD
+    PUT(["Prompt Under Test (PUT)"])
+    I["Intent (I)"]
+    IS["Input Specification (IS)"]
+    OR["Output Rules (OR)"]
+    IOR["Inverse Output Rules (IOR)"]
+    PPT["PromptPex Tests (PPT)"]
+
+    PUT --> IS
+    PUT --> I
+    PUT --> OR
+    OR --> IOR
+    I ==> PPT
+    IS ==> PPT
+    OR ==> PPT
+    PUT ==> PPT
+    IOR ==> PPT
+```
+
+##### Advanced options
+
+You can customize the test generation process with various options:
+
+```shell
+# Specify effort level (low, medium, high)
+gh models generate --effort high my_prompt.prompt.yml
+
+# Use a specific model for groundtruth generation
+gh models generate --groundtruth-model "openai/gpt-4.1" my_prompt.prompt.yml
+
+# Disable groundtruth generation
+gh models generate --groundtruth-model "none" my_prompt.prompt.yml
+
+# Load from an existing session file (or create a new one if needed)
+gh models generate --session-file my_prompt.session.json my_prompt.prompt.yml
+
+# Custom instructions for specific generation phases
+gh models generate --instruction-intent "Focus on edge cases" my_prompt.prompt.yml
+```
+
+The `effort` flag controls a few flags in the test generation engine and is a tradeoff
+between how much tests you want generated and how much tokens/time you are willing to spend.
+- `low` should be used to do a quick try of the test generation. It limits the number of rules to `3`.
+- `medium` provides much better coverage
+- `high` spends more token per rule to generate tests, which typically leads to longer, more complex inputs
+
+The command supports custom instructions for different phases of test generation:
+- `--instruction-intent`: Custom system instruction for intent generation
+- `--instruction-inputspec`: Custom system instruction for input specification generation  
+- `--instruction-outputrules`: Custom system instruction for output rules generation
+- `--instruction-inverseoutputrules`: Custom system instruction for inverse output rules generation
+- `--instruction-tests`: Custom system instruction for tests generation
+
+
 ## Notice
 
 Remember when interacting with a model you are experimenting with AI, so content mistakes are possible. The feature is

cmd/generate/README.md

@@ -0,0 +1,10 @@
+# `generate` command
+
+This command is based on [PromptPex](https://github.com/microsoft/promptpex), a test generation framework for prompts.
+
+- [Documentation](https://microsoft.github.com/promptpex)
+- [Source](https://github.com/microsoft/promptpex/tree/dev)
+- [Agentic implementation plan](https://github.com/microsoft/promptpex/blob/dev/.github/instructions/implementation.instructions.md)
+
+In a nutshell, read https://microsoft.github.io/promptpex/reference/test-generation/
+