test: add comprehensive E2E test suite for CLI wrapper #7

beonde · 2025-12-27T14:22:17Z

Summary

Adds comprehensive E2E test suite for the CLI wrapper, validating all P0 CLI commands against the actual capiscio-core binary.

P0 E2E Tests Added

Test File	Tests	Coverage
`test_validate_e2e.py`	8	Validation command
`test_badge_e2e.py`	9	Badge issue/verify commands
`test_score_e2e.py`	4	Scoring (via validation)
`test_status_e2e.py`	8	CLI status/version checks

Total: 29 new E2E tests

Infrastructure

pytest.ini: E2E test configuration with markers
tests/e2e/conftest.py: Shared fixtures and CLI runner
tests/e2e/fixtures/: Test agent cards (valid, invalid, malformed)
.github/workflows/e2e.yml: E2E test workflow
.github/workflows/test.yml: Updated unit test workflow

Changes

Reorganized tests into tests/unit/ and tests/e2e/ directories
Added tests/README.md with test documentation
Updated pyproject.toml with test dependencies

Testing

# Run E2E tests
uv run pytest tests/e2e/ -v

# Run with specific marker
uv run pytest -m "e2e" -v

P0 E2E Tests Added: - test_validate_e2e.py: 8 tests for validation command - test_badge_e2e.py: 9 tests for badge issue/verify commands - test_score_e2e.py: 4 tests for scoring (via validation) - test_status_e2e.py: 8 tests for CLI status/version checks Infrastructure: - pytest.ini: E2E test configuration with markers - tests/e2e/conftest.py: Shared fixtures and CLI runner - tests/e2e/fixtures/: Test agent cards (valid, invalid, malformed) - .github/workflows/e2e.yml: E2E test workflow - .github/workflows/test.yml: Updated unit test workflow Reorganized: - tests/unit/: Moved existing unit tests - tests/README.md: Test documentation Total: 29 new E2E tests

Copilot

Pull request overview

This PR adds a comprehensive E2E test suite for the CLI wrapper, validating P0 CLI commands against the actual capiscio-core binary. The implementation includes 29 new E2E tests across validation, badge, scoring, and status commands, along with supporting test infrastructure and CI/CD workflows.

Key Changes

Added 29 E2E tests organized into 4 test files covering validate, badge, score, and status commands
Reorganized tests into tests/unit/ and tests/e2e/ directories with comprehensive unit tests for CLI and manager modules
Created test infrastructure including pytest configuration, fixtures, and GitHub Actions workflows for both unit and E2E testing

Reviewed changes

Copilot reviewed 13 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
uv.lock	Complete dependency lock file with all test and development dependencies
pytest.ini	Pytest configuration with markers for E2E and unit tests
pyproject.toml	Added test dependencies section for pytest and pytest-cov
tests/README.md	Comprehensive documentation for running and configuring tests
tests/unit/test_manager.py	Unit tests for the manager module covering platform detection, binary management, and execution
tests/unit/test_cli.py	Unit tests for CLI module covering argument passing, wrapper commands, and exit codes
tests/e2e/conftest.py	Shared E2E test fixtures for server connection, API configuration, and test data
tests/e2e/fixtures/*.json	Test data files including valid, invalid, and malformed agent cards
tests/e2e/test_validate_e2e.py	E2E tests for validate command (8 tests)
tests/e2e/test_score_e2e.py	E2E tests for score command (4 tests)
tests/e2e/test_badge_e2e.py	E2E tests for badge issue/verify commands (9 tests)
tests/e2e/test_status_e2e.py	E2E tests for status commands (8 tests)
.github/workflows/test.yml	Unit test workflow for multiple Python versions and OS platforms
.github/workflows/e2e.yml	E2E test workflow with local server and optional dev environment testing

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pytest.ini

pyproject.toml

tests/e2e/test_badge_e2e.py

tests/e2e/conftest.py

- Remove test_score_e2e.py (CLI has no 'score' command) - Remove test_status_e2e.py (CLI has no 'status' command) - Update conftest.py to remove server dependencies - Fix test_validate_e2e.py to use --schema-only for offline validation - Fix test_badge_e2e.py to use --self-sign and --accept-self-signed - Update valid-agent-card.json to match A2A v1.3.0 schema - Simplify e2e.yml workflow (no longer needs server/postgres) Tests now run entirely offline using CLI features: - badge issue --self-sign - badge verify --accept-self-signed --offline - validate --schema-only

github-actions · 2025-12-27T15:52:19Z

✅ All checks passed! Ready for review.

Copilot

Pull request overview

Copilot reviewed 11 out of 14 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pyproject.toml

tests/e2e/fixtures/malformed.txt

.github/workflows/e2e.yml

tests/README.md

tests/e2e/conftest.py

tests/e2e/test_badge_e2e.py

- Remove unused imports (os, json) from test files - Rename malformed.json to malformed.txt to avoid linter issues - Update tests/README.md to match actual test coverage

github-actions · 2025-12-27T18:08:17Z

✅ All checks passed! Ready for review.

The test was failing on Windows CI because Path('/bin/capiscio') normalizes to backslashes on Windows. Compare using str(binary_path) to ensure consistent comparison across platforms.

github-actions · 2025-12-27T20:14:13Z

✅ All checks passed! Ready for review.

Copilot

Pull request overview

Copilot reviewed 12 out of 14 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-27T20:17:17Z

tests/README.md

+- ✅ JSON output format
+- ✅ Help command
+
+###     ├── test_validate_e2e.sue self-signed badge


Line 58 contains a broken line that appears to be incorrectly formatted: "### ├── test_validate_e2e.sue self-signed badge". This appears to be a copy-paste error or incomplete edit. The section header is malformed and the content doesn't make sense ("sue self-signed badge"). This should likely be removed or replaced with proper documentation about the Badge Command test coverage.

Suggested change

### ├── test_validate_e2e.sue self-signed badge

### Badge Command (`test_badge_e2e.py`)

Copilot · 2025-12-27T20:17:18Z

tests/README.md

+### Run Specific Test File
+
+```bash
+pytest tests/e2scio --cov-report=html


There's an error in the example command. Line 35 shows pytest tests/e2scio --cov-report=html where "e2scio" should be "e2e". This is a typo in the documentation that would cause the command to fail.

Suggested change

pytest tests/e2scio --cov-report=html

pytest tests/e2e/ --cov-report=html

Copilot · 2025-12-27T20:17:18Z

.github/workflows/e2e.yml

+      - name: Run unit tests
+        run: |
+          pytest tests/unit/ -v --cov=capiscio --cov-report=xml --cov-report=term
+      
+      - name: Run E2E tests
+        run: |
+          pytest tests/e2e/ -v --tb=short
+      
+      - name: Upload coverage reports
+        if: matrix.python-version == '3.11'
+        uses: codecov/codecov-action@v4
+        with:
+          file: ./coverage.xml
+          flags: unittests
+          name: codecov-umbrella
+          fail_ci_if_error: false


The E2E workflow is running unit tests (lines 32-34) in addition to E2E tests. This duplicates the work done in the separate unit test workflow (test.yml). The E2E workflow should only run E2E tests to avoid redundancy and reduce CI time.

Suggested change

- name: Run unit tests

run: |

pytest tests/unit/ -v --cov=capiscio --cov-report=xml --cov-report=term

- name: Run E2E tests

run: |

pytest tests/e2e/ -v --tb=short

- name: Upload coverage reports

if: matrix.python-version == '3.11'

uses: codecov/codecov-action@v4

with:

file: ./coverage.xml

flags: unittests

name: codecov-umbrella

fail_ci_if_error: false

- name: Run E2E tests

run: |

pytest tests/e2e/ -v --tb=short

Copilot AI review requested due to automatic review settings December 27, 2025 14:22

Copilot started reviewing on behalf of beonde December 27, 2025 14:22 View session

Copilot AI reviewed Dec 27, 2025

View reviewed changes

pytest.ini Outdated Show resolved Hide resolved

pyproject.toml Show resolved Hide resolved

tests/e2e/test_badge_e2e.py Outdated Show resolved Hide resolved

tests/e2e/conftest.py Outdated Show resolved Hide resolved

beonde added 2 commits December 27, 2025 10:48

fix(config): correct pytest.ini syntax (use INI format, not TOML)

fc86f5b

Copilot AI review requested due to automatic review settings December 27, 2025 15:50

Copilot started reviewing on behalf of beonde December 27, 2025 15:50 View session

fix(e2e): handle CLI download messages in badge tests

a73fe18

Copilot AI reviewed Dec 27, 2025

View reviewed changes

refactor(e2e): address copilot review feedback

f57f31d

- Remove unused imports (os, json) from test files - Rename malformed.json to malformed.txt to avoid linter issues - Update tests/README.md to match actual test coverage

fix(tests): use platform-agnostic path comparison in unit test

a2b6b27

The test was failing on Windows CI because Path('/bin/capiscio') normalizes to backslashes on Windows. Compare using str(binary_path) to ensure consistent comparison across platforms.

Copilot AI review requested due to automatic review settings December 27, 2025 20:13

Copilot started reviewing on behalf of beonde December 27, 2025 20:14 View session

beonde merged commit 4c905c6 into main Dec 27, 2025
23 checks passed

Copilot AI reviewed Dec 27, 2025

View reviewed changes

	### ├── test_validate_e2e.sue self-signed badge
	### Badge Command (`test_badge_e2e.py`)

	pytest tests/e2scio --cov-report=html
	pytest tests/e2e/ --cov-report=html

test: add comprehensive E2E test suite for CLI wrapper #7

test: add comprehensive E2E test suite for CLI wrapper #7

Uh oh!

Conversation

beonde commented Dec 27, 2025

Summary

P0 E2E Tests Added

Infrastructure

Changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 27, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 27, 2025

Uh oh!

github-actions bot commented Dec 27, 2025

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants