-
Notifications
You must be signed in to change notification settings - Fork 10
Add Kbs events #360
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
yairpod
wants to merge
4
commits into
trusted-execution-clusters:main
Choose a base branch
from
yairpod:kbs_events
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Add Kbs events #360
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "features": { | ||
| "ghcr.io/devcontainers/features/go:1": { | ||
| "version": "1.3.4", | ||
| "resolved": "ghcr.io/devcontainers/features/go@sha256:d85e921f91b41340055bb12b325d9d551170ed04b3b832e33530bf42f167c032", | ||
| "integrity": "sha256:d85e921f91b41340055bb12b325d9d551170ed04b3b832e33530bf42f167c032" | ||
| }, | ||
| "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": { | ||
| "version": "1.3.1", | ||
| "resolved": "ghcr.io/devcontainers/features/kubectl-helm-minikube@sha256:bbe8adf6b37fff8c67412ab0a4579f4c2f30bbaba1d9a5cebd9e38bade54025b", | ||
| "integrity": "sha256:bbe8adf6b37fff8c67412ab0a4579f4c2f30bbaba1d9a5cebd9e38bade54025b" | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| { | ||
| "name": "TEC Operator Dev Container", | ||
| "image": "mcr.microsoft.com/devcontainers/rust@sha256:b12e288bacbb5633f9b7655ba1a2cf69dd2fecef662346f2a9754f019af82483", | ||
| "features": { | ||
| "ghcr.io/devcontainers/features/go:1": {}, | ||
| "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": {} | ||
| }, | ||
| "postCreateCommand": "curl -sL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz | sudo tar xz -C /usr/local/bin oc kubectl && curl -fsSL 'https://azurecliprod.blob.core.windows.net/$root/deb_install.sh' | sudo bash", | ||
| "customizations": { | ||
| "vscode": { | ||
| "settings": {}, | ||
| "extensions": [ | ||
| "rust-lang.rust-analyzer", | ||
| "golang.go" | ||
| ] | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| # KBS Event Proxy | ||
|
|
||
| ## Overview | ||
|
|
||
| The KBS event proxy is a reverse proxy sidecar that runs alongside the KBS (Key Broker Service) container in the Trustee pod. It intercepts RCAR attestation HTTP traffic between nodes and KBS and emits Kubernetes events for attestation activity. | ||
|
|
||
| Without the proxy, attestation outcomes are only visible in KBS pod logs. The proxy surfaces these outcomes as first-class Kubernetes events on Machine and TrustedExecutionCluster resources. | ||
|
|
||
| ## Problem | ||
|
|
||
| Trustee has no webhook, callback, or audit log for attestation outcomes. An administrator cannot answer "Did machine X attest successfully?" without reading KBS container logs. Kubernetes events provide a standard, queryable interface for this information. | ||
|
|
||
| ## Architecture | ||
|
|
||
| The proxy runs as a sidecar container in the same pod as KBS. The Kubernetes Service routes external traffic to the proxy on port 8080. The proxy forwards all requests to KBS on localhost port 8081. | ||
|
|
||
| ``` | ||
| Nodes --> Service:8080 --> Proxy:8080 --(TLS)--> KBS:8081 (localhost) | ||
| | | ||
| inspects request/response | ||
| | | ||
| emits K8s events | ||
| ``` | ||
|
|
||
| Both hops use TLS. The proxy terminates external TLS from nodes, then connects to KBS via HTTPS on localhost. Both containers mount the same TLS secret volume. | ||
|
|
||
| ### Why a reverse proxy | ||
|
|
||
| A metrics-based sidecar polls Prometheus counters and sees counter deltas, not individual events. The reverse proxy provides: | ||
|
|
||
| - Real-time event emission per attestation attempt | ||
| - Distinction between attestation failure (401) and resource policy denial (403) | ||
| - Session correlation across the three RCAR protocol steps | ||
| - Per-machine event attribution for resource requests | ||
|
|
||
| ## RCAR Protocol | ||
|
|
||
| The RCAR (Remote CoCo Attestation and Retrieval) protocol has three HTTP steps. The proxy tracks sessions via the `kbs-session-id` cookie. | ||
|
|
||
| | Step | Endpoint | What the proxy observes | | ||
| |---|---|---| | ||
| | Auth | `POST /kbs/v0/auth` | TEE type from request body. Session cookie in response. | | ||
| | Attest | `POST /kbs/v0/attest` | 200 = attestation passed. Non-200 = failure. Session cookie identifies the session. | | ||
| | Resource | `GET /kbs/v0/resource/default/{id}/root` | Machine ID from URL path. 200 = key released. 403 = policy denied. 401 = rejected. | | ||
|
|
||
| ### Session tracking | ||
|
|
||
| The proxy maintains an in-memory HashMap that maps session IDs (from the `kbs-session-id` cookie) to session metadata: | ||
|
|
||
| ``` | ||
| session_id -> SessionInfo { tee_type, created } | ||
| ``` | ||
|
|
||
| Sessions expire after 5 minutes (matching the KBS session timeout). The proxy cleans up expired sessions after each request. | ||
|
|
||
| ## Events emitted | ||
|
|
||
| | Reason | Event type | Target resource | Trigger | | ||
| |---|---|---|---| | ||
| | `AttestationSucceeded` | Normal | Machine | Resource endpoint returns 200 for `default/{machine-id}/root` | | ||
| | `AttestationFailed` | Warning | TrustedExecutionCluster | Attest endpoint returns non-200 | | ||
| | `AttestationFailed` | Warning | Machine | Resource endpoint returns 401 for `default/{machine-id}/root` | | ||
| | `ResourcePolicyDenied` | Warning | Machine | Resource endpoint returns 403 for `default/{machine-id}/root` | | ||
|
|
||
| The proxy emits `AttestationFailed` on the TrustedExecutionCluster (not on a Machine) at the attest step because the RCAR protocol does not carry a machine identifier at that point. The session carries only the TEE type. | ||
|
|
||
| The resource step does carry the machine ID in the URL path. The proxy resolves machine IDs to Machine custom resources via the Kubernetes API. | ||
|
|
||
| ## Deployment | ||
|
|
||
| ### Pod spec changes | ||
|
|
||
| The operator modifies the Trustee pod spec in `operator/src/trustee.rs`: | ||
|
|
||
| 1. KBS container listens on `127.0.0.1:8081` (internal only) | ||
| 2. Proxy container listens on `0.0.0.0:8080` (exposed via Service) | ||
| 3. Both containers mount the TLS secret volume | ||
| 4. The pod uses the `trusted-cluster-operator` ServiceAccount for RBAC | ||
| 5. The proxy receives `CONTROLLER_POD_NAME` via the downward API for event reporting | ||
|
|
||
| ### Image resolution | ||
|
|
||
| The operator resolves the proxy image from the `RELATED_IMAGE_KBS_EVENT_PROXY` environment variable. If unset, it falls back to `{TEC_REGISTRY}/kbs-event-proxy:{COMPONENT_VERSION}`. | ||
|
|
||
| ### RBAC | ||
|
|
||
| The proxy reuses the `trusted-cluster-operator` ServiceAccount. The ClusterRole includes: | ||
|
|
||
| - `events.k8s.io` API group: `create`, `patch` (for emitting events via the `events.k8s.io/v1` API) | ||
|
|
||
| The proxy also reads Machine and TrustedExecutionCluster resources to resolve object references for event targets. These permissions are already present in the operator's ClusterRole. | ||
|
|
||
| ## TLS | ||
|
|
||
| The proxy accepts invalid TLS certificates when connecting to KBS on localhost. This is safe because the connection stays within the same pod on the loopback interface. The KBS TLS certificate contains the external hostname, not `127.0.0.1`, so strict validation would reject the connection. | ||
|
|
||
| The external-facing TLS termination uses the same certificate and key that KBS previously used directly. Nodes see no change in TLS behavior. | ||
|
|
||
| ## Code structure | ||
|
|
||
| The proxy source is in `kbs-event-proxy/src/main.rs`, organized into six sections: | ||
|
|
||
| 1. **Types and state**: CLI arguments, session info, proxy state with HTTP client, Kubernetes client, event recorder, and session map | ||
| 2. **Request/response parsing**: Extract session IDs from Cookie and Set-Cookie headers, extract machine IDs from URL paths | ||
| 3. **Kubernetes object lookups**: Resolve TrustedExecutionCluster and Machine custom resources to ObjectReferences for event targets | ||
| 4. **RCAR attestation event handlers**: One handler per RCAR step (auth, attest, resource) that inspects the forwarded response and emits events | ||
| 5. **Reverse proxy core**: Request forwarding, error responses, and the main handler that dispatches to event handlers based on URL path | ||
| 6. **Entry point**: Client initialization, TLS configuration, and server startup | ||
|
|
||
| ## Dependencies | ||
|
|
||
| The proxy reuses workspace dependencies: | ||
|
|
||
| - `axum` and `axum-server`: HTTP server and TLS termination (also used by register-server and attestation-key-register) | ||
| - `reqwest`: HTTP client for forwarding requests to KBS | ||
|
alicefr marked this conversation as resolved.
|
||
| - `kube` and `k8s-openapi`: Kubernetes API access and event recording | ||
| - `trusted-cluster-operator-lib`: Shared types (`Machine`, `record_event`, `get_trusted_execution_cluster`) | ||
|
|
||
| ## Verification | ||
|
|
||
| After deployment, verify events with: | ||
|
|
||
| ```bash | ||
| kubectl get events.events.k8s.io -n <namespace> | ||
| kubectl describe machine <machine-name> | ||
| ``` | ||
|
|
||
| A successful attestation produces an `AttestationSucceeded` event on the Machine resource. A failed attestation produces an `AttestationFailed` warning on the TrustedExecutionCluster resource. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # SPDX-FileCopyrightText: Yair Podemsky <ypodemsk@redhat.com> | ||
| # | ||
| # SPDX-License-Identifier: CC0-1.0 | ||
|
|
||
| [package] | ||
| name = "kbs-event-proxy" | ||
| version = "0.1.0" | ||
| edition.workspace = true | ||
| rust-version.workspace = true | ||
|
|
||
| [dependencies] | ||
| anyhow.workspace = true | ||
| axum.workspace = true | ||
| axum-server.workspace = true | ||
| clap.workspace = true | ||
| env_logger.workspace = true | ||
| http.workspace = true | ||
| k8s-openapi.workspace = true | ||
| kube.workspace = true | ||
| log.workspace = true | ||
| reqwest = { version = "0.12", default-features = false, features = ["native-tls"] } | ||
| serde.workspace = true | ||
| serde_json.workspace = true | ||
| tokio.workspace = true | ||
| trusted-cluster-operator-lib = { path = "../lib" } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now we don't have it yet, but in the future the uuid of the machine should be provided as initdata as part of the attestation. In this way, it should be possible to extrapolate which machine failed the attestation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So we can create a ticket to continue this work when the uuid is provided.