Implement structured evidence in rules and results #2

Compliance scanning results often provide a simple "yes/no" or "pass/fail" answer, which isn't always helpful for governance, risk management, and operational response. We need to evolve our Rule and CheckResult structs to include evidence that explains why a check passed or failed, transforming these results into actionable intelligence.

This will significantly benefit both technical and non-technical users. Auditors and compliance officers will gain context, while system administrators will be able to quickly zero in on problems.

Two-Part Solution

To provide a complete picture, the evidence should be delivered in two formats:

1. Human-Readable Evidence: A concise, clear explanation of the check's outcome. This is especially useful for auditors and non-technical stakeholders who need to understand the "why" behind a failure. For example, a human-readable message could be "Some pods are missing required CPU resource limits."

2. Structured Evidence: This is machine-readable data, such as a Kubernetes manifest, that provides the exact details needed for remediation. This is crucial for a technical audience, like system or cluster administrators, who need to identify and fix the specific issue. This structured data can document either compliant or non-compliant resources.

Use Cases and Examples

We can support two primary use cases by providing evidence for both passing and failing checks.

Case 1: Failure Remediation

For checks that fail, the structured evidence will identify the specific non-compliant resources. This allows administrators to quickly find and fix the problem. The human-readable failureReason gives the context, while the nonCompliantResources provides the technical details.

id: pods-must-have-cpu-limits
status: FAIL
failureReason: "Some pods are missing required CPU resource limits."
structuredEvidence:
  nonCompliantResources:
    - apiVersion: v1
      kind: Pod
      metadata:
        name: frontend
        namespace: production
      spec:
        containers:
          - name: frontend
            image: nginx:1.21
            resources: {} 

Case 2: Proving Compliance

For checks that pass, the structured evidence can document the compliant resources. This is valuable for auditors and compliance officers who need to verify completeness and prove that an environment meets a certain standard. The compliantResources field would list all resources that passed the check.

id: pods-must-have-cpu-limits
status: PASS
structuredEvidence:
  compliantResources:
    - apiVersion: v1
      kind: Pod
      metadata:
        name: frontend
        namespace: production
      spec:
        containers:
          - name: frontend
            image: nginx:1.21
            resources:
              limits:
                cpu: 500m
    - apiVersion: v1
      kind: Pod
      metadata:
        name: backend
        namespace: production
      spec:
        containers:
          - name: backend
            image: postgres:16
            resources:
              limits:
                cpu: 4000m

Proposed Field Structure

We should update the Rule and CheckResult structs to include two new fields:

• failureReason: A human-readable message explaining why a check failed.
• structuredEvidence: A machine-readable field containing either compliantResources or nonCompliantResources.

While keeping errorMessage reserved specifically for technical issues (e.g., a connection error during scanning or permission issues).

By implementing this, we can ensure that every scan result provides the necessary context and technical data for security teams to create actionable tickets, enabling administrators to easily remediate issues and maintain a compliant state.