From dde08a8e9a02a6c34a2c6b2a4b53fe45a924cbf7 Mon Sep 17 00:00:00 2001
From: Kumar Pratyush <kpratyush@splunk.com>
Date: Fri, 7 Aug 2026 17:30:41 +0530
Subject: [PATCH 1/3] docs: restructure external Splunk integration guide

---
 .../EXTERNAL_SPLUNK_INTEGRATION.md            | 585 ++++++------------
 1 file changed, 189 insertions(+), 396 deletions(-)

diff --git a/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md b/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
index ecc45a0d..201c9114 100644
--- a/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
+++ b/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
@@ -1,479 +1,272 @@
-# External Splunk Integration with Splunk AI Platform
+# On-Premises Splunk Enterprise Integration with Splunk AI Platform
 
-Runbook for connecting an **externally-hosted Splunk Enterprise instance** (outside the
-k0s cluster) to the Splunk AI Platform backend (SAIA). Covers every failure mode
-encountered in practice, in the order you are likely to hit them.
+This runbook connects an on-premises Splunk Enterprise search head to the Splunk
+AI Platform backend (SAIA). It covers a single search head running outside the
+k0s cluster. Search-head clusters require a separate deployer and configuration
+bundle procedure.
 
-Use this when:
-- Splunk Enterprise runs on a separate host (bare-metal, EC2, VM) — not the bundled
-  in-cluster Splunk standalone deployed by the installer.
-- The SAIA backend (`AIService`) must validate JWT tokens issued by that external Splunk.
+The request flow is:
 
----
-
-## Table of Contents
-
-- [Architecture Overview](#architecture-overview)
-- [Prerequisites](#prerequisites)
-- [Step 1 — Fix JWT Signing Key Error](#step-1--fix-jwt-signing-key-error)
-- [Step 2 — Fix 401 Unauthorized from SAIA Backend](#step-2--fix-401-unauthorized-from-saia-backend)
-- [Step 3 — Fix Browser Mixed-Content Block](#step-3--fix-browser-mixed-content-block)
-  - [Option A — Disable Splunk Web SSL (temporary workaround)](#option-a--disable-splunk-web-ssl-temporary-workaround)
-  - [Option B — TLS Termination via Load Balancer or Ingress (production fix)](#option-b--tls-termination-via-load-balancer-or-ingress-production-fix)
-- [Step 4 — Fix "Issuer Not Allowed" from SAIA Backend](#step-4--fix-issuer-not-allowed-from-saia-backend)
-- [Step 5 — Restart Splunk Correctly](#step-5--restart-splunk-correctly)
-- [Step 6 — Final Verification](#step-6--final-verification)
-- [Cleanup](#cleanup)
-- [Troubleshooting Quick Reference](#troubleshooting-quick-reference)
-
----
-
-## Architecture Overview
-
-```
-Browser (Splunk AI Assistant)
-  │
-  ├─── HTTP/HTTPS :8000 ──────────────────────────────────────────────────────►
-  │                                                                Splunk Enterprise
-  │                                                          (external host, e.g. 43.203.164.228)
-  │◄── JWT token (issuer = https://<issuer_uri>:8089) ─────────────────────────
-  │
-  │  XHR / EventSource — SAIA API calls with Bearer token
-  │  (browser calls SAIA directly — NOT through Splunk)
-  ▼
-SAIA Backend (k0s cluster, e.g. 15.164.171.171)
-  │  1. Validates JWT: issuer must be in SPLUNK_ISSUERS (ConfigMap <name>-saia-config)
-  │  2. Fetches JWKS from https://<issuer_uri>:8089/.well-known/oauth2_keys
-  │     → port 8089 must be reachable from k0s cluster nodes
-  ▼
-Ray inference / LLM
+```text
+Browser -> Splunk Web -> issues JWT
+Browser -> SAIA API -> validates JWT
+SAIA API -> Splunk management port 8089 -> downloads JWKS signing keys
 ```
 
-**Important:** The browser calls SAIA directly using the URL in `saia_sok_url`. Splunk
-only issues the JWT token — it is not a proxy for SAIA requests. Firewall/SG rules must
-allow access from the **browser's network** (not just from the Splunk host) to the SAIA
-endpoint.
+## Before you begin
 
-Key constraint: the SAIA backend fetches the public signing keys from
-`<issuer_uri>/.well-known/oauth2_keys` at JWT validation time. The `issuer_uri`
-in the token must exactly match an entry in the SAIA backend's `SPLUNK_ISSUERS`
-ConfigMap key — patched directly as described in Step 4.
+The customer needs:
 
----
+- Administrator and shell access to the Splunk Enterprise host.
+- `kubectl` access to the namespace containing the SAIA `AIService`.
+- Splunk Enterprise and Splunk AI Assistant versions validated to issue
+  interactive JWTs and expose `/.well-known/oauth2_keys`. Static or ephemeral
+  token support alone is not sufficient.
+- A Splunk management FQDN that resolves from the SAIA pods and has a certificate
+  trusted by those pods.
+- Port `8089` open from the SAIA cluster to the Splunk host.
+- An HTTPS URL for SAIA that is reachable from the customer's browser.
 
-## Prerequisites
+Use the following values throughout the procedure:
 
-- SSH access to the external Splunk host
-- `kubectl` access to the k0s cluster running SAIA
-- The external Splunk host's public IP or FQDN (used as `issuer_uri`)
-- Port **8089** (Splunk management) open from the k0s cluster nodes to the Splunk host
+| Value | Description | Example |
+|---|---|---|
+| `ISSUER_URI` | Splunk management URL used as the JWT issuer | `https://splunk.example.com:8089` |
+| `NS` | Namespace containing SAIA | `ai-platform` |
+| `AISERVICE` | SAIA `AIService` name | `production-ai-platform-saia` |
+| `SAIA_URL` | Browser-reachable SAIA URL | `https://saia.example.com` |
+| `SPLUNK_WEB_ORIGIN` | Scheme, host, and port used for Splunk Web | `https://splunk.example.com:8000` |
+| `SPLUNK_HOME` | Splunk Enterprise installation directory | `/opt/splunk` |
+| `SPLUNK_RUN_USER` | Actual operating-system account that runs `splunkd` | `splunk` |
 
----
+Set them once in each administration shell:
 
-## Step 1 — Fix JWT Signing Key Error
+Use the service unit or live `splunkd` process to identify `SPLUNK_RUN_USER`; do
+not assume that the example account is correct.
 
-**Symptom:** Splunk log shows:
-
-```
-Unable to load keys for signing interactive JWT
+```bash
+NS=ai-platform
+AISERVICE=production-ai-platform-saia
+ISSUER_URI=https://splunk.example.com:8089
+SAIA_URL=https://saia.example.com
+SPLUNK_WEB_ORIGIN=https://splunk.example.com:8000
+SPLUNK_HOME=/opt/splunk
+SPLUNK_RUN_USER=splunk
 ```
 
-**Root cause:** The `[oauth2_settings]` stanza in `authentication.conf` is missing
-or empty — `AuthenticationRSAKeysManager` has no certificate to sign tokens with.
-
-**Fix:**
-
-1. SSH into the Splunk host.
-
-2. Confirm the error:
-
-    ```bash
-    grep "Unable to load keys for signing interactive JWT" \
-      $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -5
-    ```
-
-3. Edit `$SPLUNK_HOME/etc/system/local/authentication.conf` and add:
+Find the SAIA `AIService` name if it is not already known:
 
-    ```ini
-    [oauth2_settings]
-    issuer_uri = https://<PUBLIC_IP_OR_FQDN>:8089
-    certFile = $SPLUNK_HOME/etc/auth/server.pem
-    sslPassword = <passphrase>
-    ```
-
-    To find the passphrase, check `server.conf`'s `[sslConfig]` stanza, or decrypt it:
+```bash
+kubectl get aiservice -n "$NS" -l feature=saia
+```
 
-    ```bash
-    $SPLUNK_HOME/bin/splunk show-decrypted --value '<encrypted value from server.conf>'
-    ```
+## Step 1 - Configure JWT signing in Splunk Enterprise
 
-    A typical default passphrase for a fresh Splunk install is `password`.
+### What the customer should do
 
-4. Restart Splunk (see [Step 5](#step-5--restart-splunk-correctly) for the correct procedure).
+Configure Splunk Enterprise to issue JWTs with a routable issuer URI and signing
+certificate. Restart Splunk as the operating-system user that runs `splunkd`.
 
-5. Verify the key is now loaded:
+### File to edit
 
-    ```bash
-    grep -E "oauth2|JWT|signing" $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -10
-    ```
+```text
+$SPLUNK_HOME/etc/system/local/authentication.conf
+```
 
----
+Back up the file and preserve unrelated stanzas.
 
-## Step 2 — Fix 401 Unauthorized from SAIA Backend
+### Values to provide
 
-**Symptom:** SAIA returns `401 Unauthorized`. Splunk AI Assistant log shows
-token fetch succeeds but SAIA rejects it.
+```ini
+[oauth2_settings]
+issuer_uri = https://splunk.example.com:8089
+certFile = /opt/splunk/etc/auth/server.pem
+sslPassword = <certificate-private-key-password>
+```
 
-**Root cause:** `issuer_uri` is set to `https://127.0.0.1:8089`. The SAIA
-backend can't reach `127.0.0.1` on the Splunk host — it resolves to its own
-loopback. The JWT validation (JWKS fetch) fails.
+- `issuer_uri` must be the exact `ISSUER_URI`; do not use `127.0.0.1` or
+  `localhost`.
+- `certFile` must be an absolute path to a supported PEM containing the signing
+  certificate and accessible private-key material.
+- `sslPassword` must be the actual PEM password. Do not store it in source
+  control or shell history.
 
-**Fix:**
+If the management certificate uses a private CA, add that CA to the SAIA pod's
+trust store before testing.
 
-1. Change `issuer_uri` in `authentication.conf` to the public IP or FQDN:
+Use the configured service manager when one exists. For a CLI-managed install,
+apply and verify the change as follows:
 
-    ```ini
-    [oauth2_settings]
-    issuer_uri = https://<YOUR_PUBLIC_IP>:8089
-    certFile = $SPLUNK_HOME/etc/auth/server.pem
-    sslPassword = <passphrase>
-    ```
+```bash
+sudo -H -u "$SPLUNK_RUN_USER" \
+  "$SPLUNK_HOME/bin/splunk" btool authentication list oauth2_settings --debug
 
-2. Verify port 8089 is reachable **from the k0s cluster** (this is the path that
-   actually performs JWT validation — a check from your laptop can pass while
-   SAIA pods are still blocked by a security group or firewall):
+BEFORE_PID="$(pgrep -u "$SPLUNK_RUN_USER" -x splunkd || true)"
+sudo -H -u "$SPLUNK_RUN_USER" "$SPLUNK_HOME/bin/splunk" restart
+AFTER_PID="$(pgrep -u "$SPLUNK_RUN_USER" -x splunkd || true)"
+printf 'before=%s after=%s\n' "$BEFORE_PID" "$AFTER_PID"
+```
 
-    ```bash
-    # Run from any k0s cluster node (e.g. the installer or controller)
-    nc -zv <public-ip> 8089
-    # or
-    curl -sk https://<public-ip>:8089/services/server/info | grep -c "<title>"
-    ```
+The restart must succeed, and the before/after PIDs must differ.
 
-    If the check passes from your laptop but fails from the cluster, update the
-    Splunk host's security-group inbound rules to allow port 8089 from the
-    cluster nodes' IP range.
+From the SAIA runtime, confirm that the JWKS endpoint is reachable and contains
+at least one key:
 
-3. Restart Splunk (see [Step 5](#step-5--restart-splunk-correctly)).
+```bash
+kubectl exec -n "$NS" deployment/"${AISERVICE}-saia-deployment" -- \
+  python -c 'import json,sys,urllib.request; print(len(json.load(urllib.request.urlopen(sys.argv[1]))["keys"]))' \
+  "${ISSUER_URI%/}/.well-known/oauth2_keys"
+```
 
-4. Confirm a new token carries the correct issuer:
+The command must return a number greater than zero.
 
-    ```bash
-    # Grab a fresh token from Splunk AI Assistant log
-    grep "Successfully retrieved interactive token" \
-      $SPLUNK_HOME/var/log/splunk/splunk_ai_assistant.log | tail -3
-    ```
+## Step 2 - Configure SAIA to trust the issuer
 
----
+### What the customer should do
 
-## Step 3 — Fix Browser Mixed-Content Block
+Set `SPLUNK_ISSUERS` to the same value used for `issuer_uri`, then recreate the
+SAIA pods so they load the updated value.
 
-**Symptom:** Browser dev tools → Network tab shows `blocked:mixed-content`.
-The SAIA API call is blocked before it even leaves the browser.
+### Resource to edit
 
-**Root cause:** Splunk Web is served over **HTTPS** (port 8000), but the SAIA
-backend URL configured in the AI Assistant app uses plain **HTTP**
-(e.g. `http://15.164.171.171:30080`). Browsers enforce mixed-content policy
-and block HTTPS pages from making HTTP sub-requests.
+```text
+ConfigMap/${AISERVICE}-saia-config
+data.SPLUNK_ISSUERS
+```
 
-**Options (choose one):**
+Do not change `AIPlatform.spec.splunkConfiguration.endpoint`; that field is the
+HEC telemetry endpoint, not the JWT issuer.
 
-| Option | When to use |
-|--------|-------------|
-| [Option A — Disable Splunk Web SSL](#option-a--disable-splunk-web-ssl-temporary-workaround) | Testing / short-term debugging only |
-| [Option B — TLS Termination via Load Balancer or Ingress](#option-b--tls-termination-via-load-balancer-or-ingress-production-fix) | Production — eliminates root cause |
+### Value to provide
 
----
+The value must match the JWT `iss` claim exactly, including scheme, hostname,
+port, case, and any trailing slash.
 
-### Option A — Disable Splunk Web SSL (temporary workaround)
+This sets one on-premises issuer. If another issuer must remain trusted, use the
+multi-issuer format supported by the deployed SAIA version.
 
-1. Edit `$SPLUNK_HOME/etc/system/local/web.conf`:
+```bash
+kubectl patch configmap "${AISERVICE}-saia-config" -n "$NS" --type merge \
+  -p "{\"data\":{\"SPLUNK_ISSUERS\":\"${ISSUER_URI}\"}}"
 
-    ```ini
-    [settings]
-    enableSplunkWebSSL = 0
-    ```
+kubectl get configmap "${AISERVICE}-saia-config" -n "$NS" \
+  -o jsonpath='{.data.SPLUNK_ISSUERS}{"\n"}'
+```
 
-2. Restart Splunk (see [Step 5](#step-5--restart-splunk-correctly)).
+Recreate the SAIA pods and wait for their replacements to become ready:
 
-3. Verify Splunk Web is now HTTP:
+```bash
+kubectl delete pod -n "$NS" -l "app=$AISERVICE"
+kubectl wait -n "$NS" --for=condition=Ready pod \
+  -l "app=$AISERVICE" --timeout=10m
+```
 
-    ```bash
-    curl -sv http://<public-ip>:8000 2>&1 | grep -E "< HTTP|Location"
-    # Expected: HTTP/1.1 303 or 200
+Record this ConfigMap override. Reapply it if the `AIService` or ConfigMap is
+deleted and recreated.
 
-    # Confirm HTTPS is no longer serving (port 8000 still open but speaks HTTP,
-    # so TLS negotiation fails — not "connection refused"):
-    curl -sv https://<public-ip>:8000 2>&1 | grep -E "SSL|TLS|handshake|wrong version|unknown protocol"
-    # Expected: one of the above TLS error strings
-    ```
+## Step 3 - Publish SAIA over HTTPS
 
-> **Remember to revert this** once the SAIA backend is served over HTTPS or
-> you are done testing. See [Cleanup](#cleanup).
+### What the customer should do
 
----
+Place a TLS-terminating load balancer, ingress, or reverse proxy in front of the
+SAIA service. The certificate must be trusted by customer browsers. Preserve
+streaming responses and allow the exact `SPLUNK_WEB_ORIGIN` when CORS
+configuration is required.
 
-### Option B — TLS Termination via Load Balancer or Ingress (production fix)
+### File or resource to edit
 
-The correct long-term fix is to front the SAIA backend with a TLS-terminating
-load balancer or ingress controller, so both Splunk Web and SAIA are served
-over HTTPS. Mixed-content is then eliminated at the root — no browser config
-changes needed.
+- For a k0s installer deployment, edit
+  [`k0s-cluster-config.yaml`](k0s-cluster-config.yaml).
+- Edit the customer's load-balancer, ingress, reverse-proxy, DNS, certificate,
+  firewall, and CORS configuration as required.
 
-#### How it resolves the mixed-content problem
+To expose SAIA through a NodePort, use:
 
+```yaml
+aiPlatform:
+  serviceTemplate:
+    type: NodePort
+    nodePort: 30080
 ```
-External Splunk Web (HTTPS :8000)
-  │   issues HTTPS page
-  │
-  │  XHR/EventSource to https://<host>:<port>   ← same scheme ✓
-  ▼
-Load balancer / ingress controller
-  │   TLS terminated here; presents a trusted or imported certificate
-  ▼
-SAIA service (in-cluster HTTP)
-```
-
-#### What to configure
-
-1. **Place a TLS terminator in front of SAIA.** This can be any of:
-   - A cloud load balancer (ALB, NLB, GCP HTTPS LB) with an ACM/managed cert
-   - An ingress controller (nginx, HAProxy, or similar) with a cert-manager certificate
-   - An API gateway or reverse proxy (nginx, Envoy) on the same host
 
-   The terminator listens on HTTPS (e.g. `:8443`) and proxies to the SAIA service
-   on its internal HTTP port.
+Apply the configuration through the normal installer workflow, for example:
 
-2. **Ensure the certificate is trusted by the browser.** Options:
-   - Use a publicly-trusted cert (ACM, Let's Encrypt, corporate PKI)
-   - Use a self-signed CA cert and import it once into the OS/browser trust store
-
-3. **Open firewall / security-group rules** from the client/VPN CIDR to the HTTPS port
-   on the load balancer or node.
-
-4. **Update the SAIA URL** in Splunk AI Assistant onboarding to the HTTPS address:
-
-    ```ini
-    # splunkaiassistant.conf
-    [saia_sok_configurations]
-    saia_sok_enabled = true
-    saia_sok_url = https://<host>:<port>
-    ```
-
-5. **If the SAIA backend's TLS cert is self-signed**, the load balancer's backend
-   health check and the SAIA service itself may need to be configured to accept
-   that cert (e.g. `proxy_ssl_verify off` in nginx, or importing the backend CA
-   into the terminator's trust store).
+```bash
+CONFIG_FILE=./k0s-cluster-config.yaml ./k0s_cluster_with_stack.sh install
+```
 
-#### Access pattern after TLS termination
+### Values to provide
 
-| Before | After |
+| Value | Requirement |
 |---|---|
-| `http://<host>:30080` (plain HTTP NodePort) | `https://<host>:<port>` |
-| Browser blocks XHR from HTTPS Splunk page | Same scheme — no block |
-| Requires keeping Splunk Web on HTTP | Splunk Web can stay on HTTPS |
-
----
+| SAIA service | `${AISERVICE}-saia-service`, normally port `8080` |
+| HTTPS URL | Stable `SAIA_URL`, normally on port `443` |
+| Certificate | Covers the SAIA hostname and is trusted by the browser |
+| Allowed origin | Exact `SPLUNK_WEB_ORIGIN` |
 
-## Step 4 — Fix "Issuer Not Allowed" from SAIA Backend
+Verify the service and HTTPS endpoint:
 
-**Symptom:** SAIA returns:
-
-```json
-{"detail": "Issuer 'https://127.0.0.1:8089' is not allowed"}
+```bash
+kubectl get service "${AISERVICE}-saia-service" -n "$NS" -o wide
+curl --fail --show-error "${SAIA_URL%/}/health"
 ```
 
-(or whatever the old issuer was)
+## Step 4 - Configure Splunk AI Assistant
 
-**Root cause:** `SPLUNK_ISSUERS` is a key in the SAIA config `ConfigMap`. The
-operator sets it to the hardcoded default (`https://splunk-splunk-standalone-standalone-service:8089`)
-when the key is absent or empty. Importantly:
+### What the customer should do
 
-- `AIService.spec.splunkConfiguration.endpoint` is the **HEC telemetry endpoint**
-  (used by the log-forwarding sidecar as `<endpoint>/services/collector`) — it is
-  **not** used to populate `SPLUNK_ISSUERS`. Patching `AIPlatform.spec.splunkConfiguration`
-  will not update the issuer allowlist and will redirect telemetry to the wrong endpoint.
-- The reconciler only fills **missing or empty** ConfigMap keys — once `SPLUNK_ISSUERS`
-  is set, the operator will not overwrite it. Editing the ConfigMap directly is safe
-  and is the correct fix.
+Open **Splunk AI Assistant > Configuration**, enter the SAIA URL, and save it.
 
-**Fix — edit the SAIA ConfigMap directly:**
+### File or resource to edit
 
-1. Find the SAIA config ConfigMap:
+Use the Splunk AI Assistant setup page. No application file is edited directly.
 
-    ```bash
-    kubectl get configmap -n <namespace> | grep saia-config
-    ```
+### Value to provide
 
-2. Patch `SPLUNK_ISSUERS` with the **exact** `issuer_uri` value from
-   `authentication.conf` — this must be a character-for-character match (IP or
-   FQDN, same scheme and port) because SAIA compares it literally against the
-   `iss` claim in each JWT:
-
-    ```bash
-    # Use the exact issuer_uri value from Step 1/2, e.g.:
-    #   https://43.203.164.228:8089   (if configured as IP)
-    #   https://splunk.example.com:8089  (if configured as FQDN)
-    kubectl patch configmap <name>-saia-config -n <namespace> --type merge \
-      -p '{"data":{"SPLUNK_ISSUERS":"https://<EXACT_ISSUER_URI>:8089"}}'
-    ```
-
-    To allow **both** the in-cluster Splunk and the external Splunk simultaneously,
-    separate the URLs with a space:
-
-    ```bash
-    kubectl patch configmap <name>-saia-config -n <namespace> --type merge \
-      -p '{"data":{"SPLUNK_ISSUERS":"https://splunk-splunk-standalone-standalone-service:8089 https://<EXACT_ISSUER_URI>:8089"}}'
-    ```
-
-3. Confirm the value is set:
-
-    ```bash
-    kubectl get configmap <name>-saia-config -n <namespace> \
-      -o jsonpath='{.data.SPLUNK_ISSUERS}'
-    ```
-
-4. Force SAIA pods to restart so they pick up the updated `ConfigMap`.
-
-    > **Why not `kubectl rollout restart`?** The operator's reconcile loop can
-    > race with a rollout restart. Deleting pods directly is more reliable — the
-    > operator recreates them with the current ConfigMap value.
-
-    ```bash
-    # Find the SAIA v1 and v2 pods
-    kubectl get pods -n <namespace> | grep saia
-
-    # Delete them — operator recreates with updated env
-    kubectl delete pod <saia-v1-pod> <saia-v2-pod> -n <namespace>
-    ```
-
-5. Wait for pods to reach `1/1 Running`, then re-test.
-
----
-
-## Step 5 — Restart Splunk Correctly
-
-**Symptom:** `sudo /opt/splunk/bin/splunk restart` appears to succeed (or
-silently exits 1) but the old config is still active — new tokens still carry
-the stale `issuer_uri`, and `splunkd` keeps the same PID.
-
-**Root cause:** Splunk is owned by a non-root user (e.g. `ec2-user`). Running
-`sudo splunk restart` switches to root, which cannot stop/start the process
-owned by another user. The command exits without touching the running process.
-
-**Correct procedure:**
-
-1. Check who owns the Splunk install:
-
-    ```bash
-    stat -c '%U' $SPLUNK_HOME      # Linux
-    ```
-
-2. Run as the owning user. If you are already logged in as the owner:
-
-    ```bash
-    $SPLUNK_HOME/bin/splunk stop
-    $SPLUNK_HOME/bin/splunk start --answer-yes --accept-license
-    ```
-
-    If the SSH user is different from the owner (e.g. logged in as `ec2-user`
-    but Splunk is owned by `splunk`), use `sudo -H -u`:
-
-    ```bash
-    SPLUNK_OWNER=$(stat -c '%U' $SPLUNK_HOME)
-    sudo -H -u "${SPLUNK_OWNER}" $SPLUNK_HOME/bin/splunk stop
-    sudo -H -u "${SPLUNK_OWNER}" $SPLUNK_HOME/bin/splunk start --answer-yes --accept-license
-    ```
-
-3. Confirm the PID actually changed (a matching old/new PID means it didn't restart):
-
-    ```bash
-    $SPLUNK_HOME/bin/splunk status
-    ```
-
-4. Confirm the config took effect:
-
-    ```bash
-    $SPLUNK_HOME/bin/splunk btool authentication list oauth2_settings
-    # issuer_uri must show your public IP, not 127.0.0.1
-    ```
-
----
-
-## Step 6 — Final Verification
-
-1. **Get a fresh token.** Old tokens signed before the real restart still carry
-   the stale issuer and will fail even after the fix. Log out and back in to
-   the Splunk AI Assistant to force a new token.
+```text
+https://saia.example.com
+```
 
-2. **Test end-to-end:**
+If Splunk Web uses HTTPS, the SAIA URL must also use HTTPS.
 
-    ```bash
-    # From the browser, send a prompt in Splunk AI Assistant
-    # Expected: response returned without error
-    ```
+## Step 5 - Test the integration
 
-3. **Confirm SAIA accepted the token** (check SAIA v1 pod logs):
+1. Sign out of Splunk Web and sign back in to obtain a fresh JWT.
+2. Open `${SAIA_URL}/health` from the same browser and network used for Splunk
+   Web. It must load without a certificate warning.
+3. Open Splunk AI Assistant and submit a test prompt.
+4. Confirm that the request returns successfully and that the browser shows no
+   mixed-content or CORS error.
+5. Check the SAIA logs for authentication errors:
 
     ```bash
-    kubectl logs -n <namespace> <saia-v1-pod> --tail=20 | grep -E "200|401|issuer|token"
-    ```
-
----
-
-## Cleanup
-
-After testing is complete, revert the temporary workaround from Step 3:
-
-1. Edit `$SPLUNK_HOME/etc/system/local/web.conf`:
-
-    ```ini
-    [settings]
-    enableSplunkWebSSL = 1
+    kubectl logs -n "$NS" -l "app=$AISERVICE" \
+      --all-containers --prefix --tail=100
     ```
 
-2. Restart Splunk as the owning user (see [Step 5](#step-5--restart-splunk-correctly)).
-
-3. Update the SAIA URL in the Splunk AI Assistant app config to use `https://`
-   once Splunk Web is back on HTTPS.
+## If testing fails
 
-**Check for side-effects on the k0s cluster:**
+| Error | Likely cause | What the customer should check |
+|---|---|---|
+| `Unable to load keys for signing interactive JWT` | Missing stanza, unreadable PEM, or wrong password | Check `authentication.conf`, PEM permissions, and the effective settings with `btool --debug`; then restart Splunk |
+| JWKS request times out or is refused | DNS, routing, firewall, or port `8089` is blocked | Test from the SAIA pod and allow the cluster to reach the Splunk management endpoint |
+| JWKS certificate verification fails | Untrusted CA, expired certificate, or hostname mismatch | Install the correct certificate chain and use a matching FQDN |
+| `Issuer '...' is not allowed` | `SPLUNK_ISSUERS` differs from the JWT `iss` claim | Make `authentication.conf` and the SAIA ConfigMap match exactly, then recreate the SAIA pods |
+| `401 Unauthorized` | Issuer/JWKS failure, stale token, clock skew, or old pod environment | Check SAIA logs and system clocks, then sign out and back in |
+| Config change has no effect | Splunk was restarted as the wrong OS user | Restart with the account that runs `splunkd` and confirm the process restarted |
+| Browser reports `blocked:mixed-content` | Splunk Web is HTTPS but SAIA is HTTP | Publish SAIA over HTTPS and update the app URL |
+| Browser reports a CORS error | The proxy does not allow `SPLUNK_WEB_ORIGIN` | Allow the exact Splunk Web origin and preserve CORS headers |
+| Proxy returns `502` or `504` | Wrong backend service/port, unhealthy SAIA, or short proxy timeouts | Route to `${AISERVICE}-saia-service:8080` and enable suitable streaming timeouts |
 
-If the k0s cluster previously had its own bundled Splunk standalone, the
-ConfigMap patch in Step 4 may have replaced the in-cluster issuer with the
-external one. Verify nothing else on the cluster depended on the bundled
-instance:
+Useful diagnostic commands:
 
 ```bash
-# Check if the in-cluster Splunk standalone still exists and is healthy
-kubectl get standalone -n ai-platform
-kubectl get pods -n ai-platform | grep splunk
+sudo -H -u "$SPLUNK_RUN_USER" \
+  "$SPLUNK_HOME/bin/splunk" btool authentication list oauth2_settings --debug
 
-# Check current SPLUNK_ISSUERS value in the ConfigMap
-kubectl get configmap -n ai-platform -o json | jq -r '.items[].data | select(has("SPLUNK_ISSUERS")) | .SPLUNK_ISSUERS'
-```
-
-If the in-cluster Splunk is still deployed and needs to be trusted alongside
-the external one, patch `SPLUNK_ISSUERS` with both URLs as shown in Step 4:
+grep -E "Unable to load keys|oauth2|JWT|signing" \
+  $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -50
 
-```bash
-kubectl patch configmap <name>-saia-config -n <namespace> --type merge \
-  -p '{"data":{"SPLUNK_ISSUERS":"https://splunk-splunk-standalone-standalone-service:8089 https://<PUBLIC_IP>:8089"}}'
+kubectl get configmap,pod,service -n "$NS" | grep "$AISERVICE"
+kubectl describe pods -n "$NS" -l "app=$AISERVICE"
 ```
-
-Otherwise, decommission whichever Splunk instance is no longer the source of
-truth and leave only its issuer in `SPLUNK_ISSUERS`.
-
----
-
-## Troubleshooting Quick Reference
-
-| Symptom | Most likely cause | Section |
-|---------|-------------------|---------|
-| `Unable to load keys for signing interactive JWT` | Missing `[oauth2_settings]` in `authentication.conf` | [Step 1](#step-1--fix-jwt-signing-key-error) |
-| `401 Unauthorized` from SAIA, JWKS fetch fails | `issuer_uri = https://127.0.0.1:8089` | [Step 2](#step-2--fix-401-unauthorized-from-saia-backend) |
-| Browser `blocked:mixed-content`, request never sent | Splunk HTTPS + SAIA HTTP | [Step 3](#step-3--fix-browser-mixed-content-block) |
-| `{"detail":"Issuer '...' is not allowed"}` | External issuer not in `SPLUNK_ISSUERS` allowlist | [Step 4](#step-4--fix-issuer-not-allowed-from-saia-backend) |
-| Config change has no effect after restart | Restarted with `sudo` but Splunk owned by another user | [Step 5](#step-5--restart-splunk-correctly) |
-| Fresh fix works but old browser session still fails | Stale JWT from before the restart — log out and back in | [Step 6](#step-6--final-verification) |
-| Patching `AIPlatform.splunkConfiguration.endpoint` doesn't fix issuer | That field is the HEC endpoint, not the issuer — patch `SPLUNK_ISSUERS` in the ConfigMap directly | [Step 4](#step-4--fix-issuer-not-allowed-from-saia-backend) |

From 56001ff4511f0102d1069cc6240114688ede24bf Mon Sep 17 00:00:00 2001
From: Kumar Pratyush <kpratyush@splunk.com>
Date: Sun, 9 Aug 2026 22:01:15 +0530
Subject: [PATCH 2/3] docs: preserve content and reorder integration flow

---
 .../EXTERNAL_SPLUNK_INTEGRATION.md            | 606 ++++++++++++------
 1 file changed, 423 insertions(+), 183 deletions(-)

diff --git a/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md b/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
index 201c9114..9236c051 100644
--- a/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
+++ b/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
@@ -1,272 +1,512 @@
-# On-Premises Splunk Enterprise Integration with Splunk AI Platform
+# External Splunk Integration with Splunk AI Platform
 
-This runbook connects an on-premises Splunk Enterprise search head to the Splunk
-AI Platform backend (SAIA). It covers a single search head running outside the
-k0s cluster. Search-head clusters require a separate deployer and configuration
-bundle procedure.
+Runbook for connecting an **externally-hosted Splunk Enterprise instance** (outside the
+k0s cluster) to the Splunk AI Platform backend (SAIA). The required actions,
+files, and values are presented first. Failure modes encountered in practice are
+covered after testing.
 
-The request flow is:
+Use this when:
+- Splunk Enterprise runs on a separate host (bare-metal, EC2, VM) — not the bundled
+  in-cluster Splunk standalone deployed by the installer.
+- The SAIA backend (`AIService`) must validate JWT tokens issued by that external Splunk.
+
+---
+
+## Table of Contents
+
+- [1. Problem Statement](#1-problem-statement)
+- [2. What the Customer Should Do](#2-what-the-customer-should-do)
+- [3. Files and Resources to Edit](#3-files-and-resources-to-edit)
+- [4. Values to Provide](#4-values-to-provide)
+- [5. Testing](#5-testing)
+- [6. If Testing Fails](#6-if-testing-fails)
+
+---
+
+## 1. Problem Statement
+
+### Architecture Overview
 
-```text
-Browser -> Splunk Web -> issues JWT
-Browser -> SAIA API -> validates JWT
-SAIA API -> Splunk management port 8089 -> downloads JWKS signing keys
+```
+Browser (Splunk AI Assistant)
+  │
+  ├─── HTTP/HTTPS :8000 ──────────────────────────────────────────────────────►
+  │                                                                Splunk Enterprise
+  │                                                          (external host, e.g. 43.203.164.228)
+  │◄── JWT token (issuer = https://<issuer_uri>:8089) ─────────────────────────
+  │
+  │  XHR / EventSource — SAIA API calls with Bearer token
+  │  (browser calls SAIA directly — NOT through Splunk)
+  ▼
+SAIA Backend (k0s cluster, e.g. 15.164.171.171)
+  │  1. Validates JWT: issuer must be in SPLUNK_ISSUERS (ConfigMap <name>-saia-config)
+  │  2. Fetches JWKS from https://<issuer_uri>:8089/.well-known/oauth2_keys
+  │     → port 8089 must be reachable from k0s cluster nodes
+  ▼
+Ray inference / LLM
 ```
 
-## Before you begin
+**Important:** The browser calls SAIA directly using the URL in `saia_sok_url`. Splunk
+only issues the JWT token — it is not a proxy for SAIA requests. Firewall/SG rules must
+allow access from the **browser's network** (not just from the Splunk host) to the SAIA
+endpoint.
 
-The customer needs:
+Key constraint: the SAIA backend fetches the public signing keys from
+`<issuer_uri>/.well-known/oauth2_keys` at JWT validation time. The `issuer_uri`
+in the token must exactly match an entry in the SAIA backend's `SPLUNK_ISSUERS`
+ConfigMap key — patched directly as described in
+[SAIA ConfigMap Values](#saia-configmap-values).
 
-- Administrator and shell access to the Splunk Enterprise host.
-- `kubectl` access to the namespace containing the SAIA `AIService`.
-- Splunk Enterprise and Splunk AI Assistant versions validated to issue
-  interactive JWTs and expose `/.well-known/oauth2_keys`. Static or ephemeral
-  token support alone is not sufficient.
-- A Splunk management FQDN that resolves from the SAIA pods and has a certificate
-  trusted by those pods.
-- Port `8089` open from the SAIA cluster to the Splunk host.
-- An HTTPS URL for SAIA that is reachable from the customer's browser.
+---
 
-Use the following values throughout the procedure:
+## 2. What the Customer Should Do
 
-| Value | Description | Example |
-|---|---|---|
-| `ISSUER_URI` | Splunk management URL used as the JWT issuer | `https://splunk.example.com:8089` |
-| `NS` | Namespace containing SAIA | `ai-platform` |
-| `AISERVICE` | SAIA `AIService` name | `production-ai-platform-saia` |
-| `SAIA_URL` | Browser-reachable SAIA URL | `https://saia.example.com` |
-| `SPLUNK_WEB_ORIGIN` | Scheme, host, and port used for Splunk Web | `https://splunk.example.com:8000` |
-| `SPLUNK_HOME` | Splunk Enterprise installation directory | `/opt/splunk` |
-| `SPLUNK_RUN_USER` | Actual operating-system account that runs `splunkd` | `splunk` |
+### Prerequisites
 
-Set them once in each administration shell:
+- SSH access to the external Splunk host
+- `kubectl` access to the k0s cluster running SAIA
+- The external Splunk host's public IP or FQDN (used as `issuer_uri`)
+- Port **8089** (Splunk management) open from the k0s cluster nodes to the Splunk host
 
-Use the service unit or live `splunkd` process to identify `SPLUNK_RUN_USER`; do
-not assume that the example account is correct.
+Complete the integration in this order:
 
-```bash
-NS=ai-platform
-AISERVICE=production-ai-platform-saia
-ISSUER_URI=https://splunk.example.com:8089
-SAIA_URL=https://saia.example.com
-SPLUNK_WEB_ORIGIN=https://splunk.example.com:8000
-SPLUNK_HOME=/opt/splunk
-SPLUNK_RUN_USER=splunk
-```
+1. SSH into the Splunk host.
+2. Configure JWT signing in `authentication.conf` with a routable issuer.
+3. Restart Splunk as the user that owns the installation.
+4. Verify that the k0s cluster can reach the Splunk management port.
+5. Place a TLS-terminating load balancer or ingress in front of SAIA.
+6. Update the SAIA URL in Splunk AI Assistant.
+7. Add the exact issuer to the SAIA `SPLUNK_ISSUERS` ConfigMap value.
+8. Restart the SAIA pods and complete the end-to-end test.
 
-Find the SAIA `AIService` name if it is not already known:
+### Restart Splunk Correctly
 
-```bash
-kubectl get aiservice -n "$NS" -l feature=saia
-```
+**Correct procedure:**
 
-## Step 1 - Configure JWT signing in Splunk Enterprise
+1. Check who owns the Splunk install:
 
-### What the customer should do
+    ```bash
+    stat -c '%U' $SPLUNK_HOME      # Linux
+    ```
 
-Configure Splunk Enterprise to issue JWTs with a routable issuer URI and signing
-certificate. Restart Splunk as the operating-system user that runs `splunkd`.
+2. Run as the owning user. If you are already logged in as the owner:
 
-### File to edit
+    ```bash
+    $SPLUNK_HOME/bin/splunk stop
+    $SPLUNK_HOME/bin/splunk start --answer-yes --accept-license
+    ```
 
-```text
-$SPLUNK_HOME/etc/system/local/authentication.conf
-```
+    If the SSH user is different from the owner (e.g. logged in as `ec2-user`
+    but Splunk is owned by `splunk`), use `sudo -H -u`:
+
+    ```bash
+    SPLUNK_OWNER=$(stat -c '%U' $SPLUNK_HOME)
+    sudo -H -u "${SPLUNK_OWNER}" $SPLUNK_HOME/bin/splunk stop
+    sudo -H -u "${SPLUNK_OWNER}" $SPLUNK_HOME/bin/splunk start --answer-yes --accept-license
+    ```
+
+3. Confirm the PID actually changed (a matching old/new PID means it didn't restart):
+
+    ```bash
+    $SPLUNK_HOME/bin/splunk status
+    ```
+
+4. Confirm the config took effect:
+
+    ```bash
+    $SPLUNK_HOME/bin/splunk btool authentication list oauth2_settings
+    # issuer_uri must show your public IP, not 127.0.0.1
+    ```
+
+5. Verify the key is now loaded:
+
+    ```bash
+    grep -E "oauth2|JWT|signing" $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -10
+    ```
+
+---
 
-Back up the file and preserve unrelated stanzas.
+## 3. Files and Resources to Edit
 
-### Values to provide
+| File or resource | Customer change |
+|---|---|
+| `$SPLUNK_HOME/etc/system/local/authentication.conf` | Add or update `[oauth2_settings]` |
+| `server.conf` `[sslConfig]` | Read the encrypted certificate passphrase; do not edit it for this procedure |
+| Load balancer, ingress, firewall, security-group, DNS, and certificate resources | Publish SAIA over HTTPS and allow the required network paths |
+| Splunk AI Assistant app-local `splunkaiassistant.conf` | Set the SAIA URL used by Splunk AI Assistant; the app-local path depends on the installed app version |
+| SAIA ConfigMap `<name>-saia-config` | Set `data.SPLUNK_ISSUERS` |
+| `$SPLUNK_HOME/etc/system/local/web.conf` | Edit only for the temporary mixed-content workaround under [If Testing Fails](#browser-mixed-content-block) |
+
+---
+
+## 4. Values to Provide
+
+### Splunk JWT Values
+
+Edit `$SPLUNK_HOME/etc/system/local/authentication.conf` and add:
 
 ```ini
 [oauth2_settings]
-issuer_uri = https://splunk.example.com:8089
-certFile = /opt/splunk/etc/auth/server.pem
-sslPassword = <certificate-private-key-password>
+issuer_uri = https://<PUBLIC_IP_OR_FQDN>:8089
+certFile = $SPLUNK_HOME/etc/auth/server.pem
+sslPassword = <passphrase>
 ```
 
-- `issuer_uri` must be the exact `ISSUER_URI`; do not use `127.0.0.1` or
-  `localhost`.
-- `certFile` must be an absolute path to a supported PEM containing the signing
-  certificate and accessible private-key material.
-- `sslPassword` must be the actual PEM password. Do not store it in source
-  control or shell history.
+To find the passphrase, check `server.conf`'s `[sslConfig]` stanza, or decrypt it:
 
-If the management certificate uses a private CA, add that CA to the SAIA pod's
-trust store before testing.
+```bash
+$SPLUNK_HOME/bin/splunk show-decrypted --value '<encrypted value from server.conf>'
+```
 
-Use the configured service manager when one exists. For a CLI-managed install,
-apply and verify the change as follows:
+A typical default passphrase for a fresh Splunk install is `password`.
 
-```bash
-sudo -H -u "$SPLUNK_RUN_USER" \
-  "$SPLUNK_HOME/bin/splunk" btool authentication list oauth2_settings --debug
+Restart Splunk (see [Restart Splunk Correctly](#restart-splunk-correctly) for the correct procedure).
 
-BEFORE_PID="$(pgrep -u "$SPLUNK_RUN_USER" -x splunkd || true)"
-sudo -H -u "$SPLUNK_RUN_USER" "$SPLUNK_HOME/bin/splunk" restart
-AFTER_PID="$(pgrep -u "$SPLUNK_RUN_USER" -x splunkd || true)"
-printf 'before=%s after=%s\n' "$BEFORE_PID" "$AFTER_PID"
-```
+### Option B — TLS Termination via Load Balancer or Ingress (production fix)
 
-The restart must succeed, and the before/after PIDs must differ.
+The correct long-term fix is to front the SAIA backend with a TLS-terminating
+load balancer or ingress controller, so both Splunk Web and SAIA are served
+over HTTPS. Mixed-content is then eliminated at the root — no browser config
+changes needed.
 
-From the SAIA runtime, confirm that the JWKS endpoint is reachable and contains
-at least one key:
+#### How it resolves the mixed-content problem
 
-```bash
-kubectl exec -n "$NS" deployment/"${AISERVICE}-saia-deployment" -- \
-  python -c 'import json,sys,urllib.request; print(len(json.load(urllib.request.urlopen(sys.argv[1]))["keys"]))' \
-  "${ISSUER_URI%/}/.well-known/oauth2_keys"
+```
+External Splunk Web (HTTPS :8000)
+  │   issues HTTPS page
+  │
+  │  XHR/EventSource to https://<host>:<port>   ← same scheme ✓
+  ▼
+Load balancer / ingress controller
+  │   TLS terminated here; presents a trusted or imported certificate
+  ▼
+SAIA service (in-cluster HTTP)
 ```
 
-The command must return a number greater than zero.
+#### What to configure
 
-## Step 2 - Configure SAIA to trust the issuer
+1. **Place a TLS terminator in front of SAIA.** This can be any of:
+   - A cloud load balancer (ALB, NLB, GCP HTTPS LB) with an ACM/managed cert
+   - An ingress controller (nginx, HAProxy, or similar) with a cert-manager certificate
+   - An API gateway or reverse proxy (nginx, Envoy) on the same host
 
-### What the customer should do
+   The terminator listens on HTTPS (e.g. `:8443`) and proxies to the SAIA service
+   on its internal HTTP port.
 
-Set `SPLUNK_ISSUERS` to the same value used for `issuer_uri`, then recreate the
-SAIA pods so they load the updated value.
+2. **Ensure the certificate is trusted by the browser.** Options:
+   - Use a publicly-trusted cert (ACM, Let's Encrypt, corporate PKI)
+   - Use a self-signed CA cert and import it once into the OS/browser trust store
 
-### Resource to edit
+3. **Open firewall / security-group rules** from the client/VPN CIDR to the HTTPS port
+   on the load balancer or node.
 
-```text
-ConfigMap/${AISERVICE}-saia-config
-data.SPLUNK_ISSUERS
-```
+4. **Update the SAIA URL** in Splunk AI Assistant onboarding to the HTTPS address:
+
+    ```ini
+    # splunkaiassistant.conf
+    [saia_sok_configurations]
+    saia_sok_enabled = true
+    saia_sok_url = https://<host>:<port>
+    ```
+
+5. **If the SAIA backend's TLS cert is self-signed**, the load balancer's backend
+   health check and the SAIA service itself may need to be configured to accept
+   that cert (e.g. `proxy_ssl_verify off` in nginx, or importing the backend CA
+   into the terminator's trust store).
+
+#### Access pattern after TLS termination
 
-Do not change `AIPlatform.spec.splunkConfiguration.endpoint`; that field is the
-HEC telemetry endpoint, not the JWT issuer.
+| Before | After |
+|---|---|
+| `http://<host>:30080` (plain HTTP NodePort) | `https://<host>:<port>` |
+| Browser blocks XHR from HTTPS Splunk page | Same scheme — no block |
+| Requires keeping Splunk Web on HTTP | Splunk Web can stay on HTTPS |
+
+### SAIA ConfigMap Values
+
+The following details explain why this resource is edited:
+
+- `AIService.spec.splunkConfiguration.endpoint` is the **HEC telemetry endpoint**
+  (used by the log-forwarding sidecar as `<endpoint>/services/collector`) — it is
+  **not** used to populate `SPLUNK_ISSUERS`. Patching `AIPlatform.spec.splunkConfiguration`
+  will not update the issuer allowlist and will redirect telemetry to the wrong endpoint.
+- The reconciler only fills **missing or empty** ConfigMap keys — once `SPLUNK_ISSUERS`
+  is set, the operator will not overwrite it. Editing the ConfigMap directly is safe
+  and is the correct fix.
+
+**Fix — edit the SAIA ConfigMap directly:**
+
+1. Find the SAIA config ConfigMap:
+
+    ```bash
+    kubectl get configmap -n <namespace> | grep saia-config
+    ```
+
+2. Patch `SPLUNK_ISSUERS` with the **exact** `issuer_uri` value from
+   `authentication.conf` — this must be a character-for-character match (IP or
+   FQDN, same scheme and port) because SAIA compares it literally against the
+   `iss` claim in each JWT:
+
+    ```bash
+    # Use the exact issuer_uri value from authentication.conf, e.g.:
+    #   https://43.203.164.228:8089   (if configured as IP)
+    #   https://splunk.example.com:8089  (if configured as FQDN)
+    kubectl patch configmap <name>-saia-config -n <namespace> --type merge \
+      -p '{"data":{"SPLUNK_ISSUERS":"<EXACT_ISSUER_URI>"}}'
+    ```
 
-### Value to provide
+    To allow **both** the in-cluster Splunk and the external Splunk simultaneously,
+    separate the URLs with a space:
 
-The value must match the JWT `iss` claim exactly, including scheme, hostname,
-port, case, and any trailing slash.
+    ```bash
+    kubectl patch configmap <name>-saia-config -n <namespace> --type merge \
+      -p '{"data":{"SPLUNK_ISSUERS":"https://splunk-splunk-standalone-standalone-service:8089 <EXACT_ISSUER_URI>"}}'
+    ```
 
-This sets one on-premises issuer. If another issuer must remain trusted, use the
-multi-issuer format supported by the deployed SAIA version.
+3. Confirm the value is set:
+
+    ```bash
+    kubectl get configmap <name>-saia-config -n <namespace> \
+      -o jsonpath='{.data.SPLUNK_ISSUERS}'
+    ```
+
+4. Force SAIA pods to restart so they pick up the updated `ConfigMap`.
+
+    > **Why not `kubectl rollout restart`?** The operator's reconcile loop can
+    > race with a rollout restart. Deleting pods directly is more reliable — the
+    > operator recreates them with the current ConfigMap value.
+
+    ```bash
+    # Find the SAIA v1 and v2 pods
+    kubectl get pods -n <namespace> | grep saia
+
+    # Delete them — operator recreates with updated env
+    kubectl delete pod <saia-v1-pod> <saia-v2-pod> -n <namespace>
+    ```
+
+5. Wait for pods to reach `1/1 Running`, then re-test.
+
+**Check for side-effects on the k0s cluster:**
+
+If the k0s cluster previously had its own bundled Splunk standalone, the
+ConfigMap patch above may have replaced the in-cluster issuer with the
+external one. Verify nothing else on the cluster depended on the bundled
+instance:
 
 ```bash
-kubectl patch configmap "${AISERVICE}-saia-config" -n "$NS" --type merge \
-  -p "{\"data\":{\"SPLUNK_ISSUERS\":\"${ISSUER_URI}\"}}"
+# Check if the in-cluster Splunk standalone still exists and is healthy
+kubectl get standalone -n ai-platform
+kubectl get pods -n ai-platform | grep splunk
 
-kubectl get configmap "${AISERVICE}-saia-config" -n "$NS" \
-  -o jsonpath='{.data.SPLUNK_ISSUERS}{"\n"}'
+# Check current SPLUNK_ISSUERS value in the ConfigMap
+kubectl get configmap -n ai-platform -o json | jq -r '.items[].data | select(has("SPLUNK_ISSUERS")) | .SPLUNK_ISSUERS'
 ```
 
-Recreate the SAIA pods and wait for their replacements to become ready:
+If the in-cluster Splunk is still deployed and needs to be trusted alongside
+the external one, patch `SPLUNK_ISSUERS` with both URLs as shown above:
 
 ```bash
-kubectl delete pod -n "$NS" -l "app=$AISERVICE"
-kubectl wait -n "$NS" --for=condition=Ready pod \
-  -l "app=$AISERVICE" --timeout=10m
+kubectl patch configmap <name>-saia-config -n <namespace> --type merge \
+  -p '{"data":{"SPLUNK_ISSUERS":"https://splunk-splunk-standalone-standalone-service:8089 https://<PUBLIC_IP>:8089"}}'
 ```
 
-Record this ConfigMap override. Reapply it if the `AIService` or ConfigMap is
-deleted and recreated.
+Otherwise, decommission whichever Splunk instance is no longer the source of
+truth and leave only its issuer in `SPLUNK_ISSUERS`.
+
+---
 
-## Step 3 - Publish SAIA over HTTPS
+## 5. Testing
 
-### What the customer should do
+### Verify Issuer Reachability
 
-Place a TLS-terminating load balancer, ingress, or reverse proxy in front of the
-SAIA service. The certificate must be trusted by customer browsers. Preserve
-streaming responses and allow the exact `SPLUNK_WEB_ORIGIN` when CORS
-configuration is required.
+Verify port 8089 is reachable **from the k0s cluster** (this is the path that
+actually performs JWT validation — a check from your laptop can pass while
+SAIA pods are still blocked by a security group or firewall):
 
-### File or resource to edit
+```bash
+# Run from any k0s cluster node (e.g. the installer or controller)
+nc -zv <public-ip> 8089
+# or
+curl -sk https://<public-ip>:8089/services/server/info | grep -c "<title>"
+```
 
-- For a k0s installer deployment, edit
-  [`k0s-cluster-config.yaml`](k0s-cluster-config.yaml).
-- Edit the customer's load-balancer, ingress, reverse-proxy, DNS, certificate,
-  firewall, and CORS configuration as required.
+### Confirm a Fresh Token Uses the Correct Issuer
 
-To expose SAIA through a NodePort, use:
+Confirm a new token carries the correct issuer:
 
-```yaml
-aiPlatform:
-  serviceTemplate:
-    type: NodePort
-    nodePort: 30080
+```bash
+# Grab a fresh token from Splunk AI Assistant log
+grep "Successfully retrieved interactive token" \
+  $SPLUNK_HOME/var/log/splunk/splunk_ai_assistant.log | tail -3
 ```
 
-Apply the configuration through the normal installer workflow, for example:
+### End-to-End Verification
 
-```bash
-CONFIG_FILE=./k0s-cluster-config.yaml ./k0s_cluster_with_stack.sh install
+1. **Get a fresh token.** Old tokens signed before the real restart still carry
+   the stale issuer and will fail even after the fix. Log out and back in to
+   the Splunk AI Assistant to force a new token.
+
+2. **Test end-to-end:**
+
+    ```bash
+    # From the browser, send a prompt in Splunk AI Assistant
+    # Expected: response returned without error
+    ```
+
+3. **Confirm SAIA accepted the token** (check SAIA v1 pod logs):
+
+    ```bash
+    kubectl logs -n <namespace> <saia-v1-pod> --tail=20 | grep -E "200|401|issuer|token"
+    ```
+
+---
+
+## 6. If Testing Fails
+
+### JWT Signing Key Error
+
+**Symptom:** Splunk log shows:
+
+```
+Unable to load keys for signing interactive JWT
 ```
 
-### Values to provide
+**Root cause:** The `[oauth2_settings]` stanza in `authentication.conf` is missing
+or empty — `AuthenticationRSAKeysManager` has no certificate to sign tokens with.
 
-| Value | Requirement |
-|---|---|
-| SAIA service | `${AISERVICE}-saia-service`, normally port `8080` |
-| HTTPS URL | Stable `SAIA_URL`, normally on port `443` |
-| Certificate | Covers the SAIA hostname and is trusted by the browser |
-| Allowed origin | Exact `SPLUNK_WEB_ORIGIN` |
 
-Verify the service and HTTPS endpoint:
+Confirm the error:
 
 ```bash
-kubectl get service "${AISERVICE}-saia-service" -n "$NS" -o wide
-curl --fail --show-error "${SAIA_URL%/}/health"
+grep "Unable to load keys for signing interactive JWT" \
+  $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -5
 ```
 
-## Step 4 - Configure Splunk AI Assistant
+Apply the values in [Splunk JWT Values](#splunk-jwt-values), restart Splunk, and repeat the JWT signing test.
 
-### What the customer should do
+### 401 Unauthorized from SAIA Backend
 
-Open **Splunk AI Assistant > Configuration**, enter the SAIA URL, and save it.
+**Symptom:** SAIA returns `401 Unauthorized`. Splunk AI Assistant log shows
+token fetch succeeds but SAIA rejects it.
 
-### File or resource to edit
+**Root cause:** `issuer_uri` is set to `https://127.0.0.1:8089`. The SAIA
+backend can't reach `127.0.0.1` on the Splunk host — it resolves to its own
+loopback. The JWT validation (JWKS fetch) fails.
 
-Use the Splunk AI Assistant setup page. No application file is edited directly.
+**Fix:**
 
-### Value to provide
+1. Change `issuer_uri` in `authentication.conf` to the public IP or FQDN:
 
-```text
-https://saia.example.com
-```
+    ```ini
+    [oauth2_settings]
+    issuer_uri = https://<YOUR_PUBLIC_IP>:8089
+    certFile = $SPLUNK_HOME/etc/auth/server.pem
+    sslPassword = <passphrase>
+    ```
 
-If Splunk Web uses HTTPS, the SAIA URL must also use HTTPS.
+2. If the reachability check passes from your laptop but fails from the cluster,
+   update the Splunk host's security-group inbound rules to allow port 8089 from
+   the cluster nodes' IP range.
 
-## Step 5 - Test the integration
+3. Restart Splunk (see [Restart Splunk Correctly](#restart-splunk-correctly)).
 
-1. Sign out of Splunk Web and sign back in to obtain a fresh JWT.
-2. Open `${SAIA_URL}/health` from the same browser and network used for Splunk
-   Web. It must load without a certificate warning.
-3. Open Splunk AI Assistant and submit a test prompt.
-4. Confirm that the request returns successfully and that the browser shows no
-   mixed-content or CORS error.
-5. Check the SAIA logs for authentication errors:
+Confirm the issuer and network values in [Splunk JWT Values](#splunk-jwt-values), then repeat the issuer reachability test.
 
-    ```bash
-    kubectl logs -n "$NS" -l "app=$AISERVICE" \
-      --all-containers --prefix --tail=100
-    ```
+### Browser Mixed-Content Block
+
+**Symptom:** Browser dev tools → Network tab shows `blocked:mixed-content`.
+The SAIA API call is blocked before it even leaves the browser.
+
+**Root cause:** Splunk Web is served over **HTTPS** (port 8000), but the SAIA
+backend URL configured in the AI Assistant app uses plain **HTTP**
+(e.g. `http://15.164.171.171:30080`). Browsers enforce mixed-content policy
+and block HTTPS pages from making HTTP sub-requests.
+
+**Options (choose one):**
+
+| Option | When to use |
+|--------|-------------|
+| [Option A — Disable Splunk Web SSL](#option-a--disable-splunk-web-ssl-temporary-workaround) | Testing / short-term debugging only |
+| [Option B — TLS Termination via Load Balancer or Ingress](#option-b--tls-termination-via-load-balancer-or-ingress-production-fix) | Production — eliminates root cause |
+
+#### Option A — Disable Splunk Web SSL (temporary workaround)
 
-## If testing fails
+1. Edit `$SPLUNK_HOME/etc/system/local/web.conf`:
 
-| Error | Likely cause | What the customer should check |
-|---|---|---|
-| `Unable to load keys for signing interactive JWT` | Missing stanza, unreadable PEM, or wrong password | Check `authentication.conf`, PEM permissions, and the effective settings with `btool --debug`; then restart Splunk |
-| JWKS request times out or is refused | DNS, routing, firewall, or port `8089` is blocked | Test from the SAIA pod and allow the cluster to reach the Splunk management endpoint |
-| JWKS certificate verification fails | Untrusted CA, expired certificate, or hostname mismatch | Install the correct certificate chain and use a matching FQDN |
-| `Issuer '...' is not allowed` | `SPLUNK_ISSUERS` differs from the JWT `iss` claim | Make `authentication.conf` and the SAIA ConfigMap match exactly, then recreate the SAIA pods |
-| `401 Unauthorized` | Issuer/JWKS failure, stale token, clock skew, or old pod environment | Check SAIA logs and system clocks, then sign out and back in |
-| Config change has no effect | Splunk was restarted as the wrong OS user | Restart with the account that runs `splunkd` and confirm the process restarted |
-| Browser reports `blocked:mixed-content` | Splunk Web is HTTPS but SAIA is HTTP | Publish SAIA over HTTPS and update the app URL |
-| Browser reports a CORS error | The proxy does not allow `SPLUNK_WEB_ORIGIN` | Allow the exact Splunk Web origin and preserve CORS headers |
-| Proxy returns `502` or `504` | Wrong backend service/port, unhealthy SAIA, or short proxy timeouts | Route to `${AISERVICE}-saia-service:8080` and enable suitable streaming timeouts |
+```ini
+[settings]
+enableSplunkWebSSL = 0
+```
 
-Useful diagnostic commands:
+2. Restart Splunk (see [Restart Splunk Correctly](#restart-splunk-correctly)).
+
+3. Verify Splunk Web is now HTTP:
 
 ```bash
-sudo -H -u "$SPLUNK_RUN_USER" \
-  "$SPLUNK_HOME/bin/splunk" btool authentication list oauth2_settings --debug
+curl -sv http://<public-ip>:8000 2>&1 | grep -E "< HTTP|Location"
+# Expected: HTTP/1.1 303 or 200
+
+# Confirm HTTPS is no longer serving (port 8000 still open but speaks HTTP,
+# so TLS negotiation fails — not "connection refused"):
+curl -sv https://<public-ip>:8000 2>&1 | grep -E "SSL|TLS|handshake|wrong version|unknown protocol"
+# Expected: one of the above TLS error strings
+```
+
+> **Remember to revert this** once the SAIA backend is served over HTTPS or
+> you are done testing. See [Cleanup](#cleanup).
+
+#### Cleanup
+
+After testing is complete, revert the temporary workaround above:
 
-grep -E "Unable to load keys|oauth2|JWT|signing" \
-  $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -50
+1. Edit `$SPLUNK_HOME/etc/system/local/web.conf`:
 
-kubectl get configmap,pod,service -n "$NS" | grep "$AISERVICE"
-kubectl describe pods -n "$NS" -l "app=$AISERVICE"
+    ```ini
+    [settings]
+    enableSplunkWebSSL = 1
+    ```
+
+2. Restart Splunk as the owning user (see [Restart Splunk Correctly](#restart-splunk-correctly)).
+
+3. Update the SAIA URL in the Splunk AI Assistant app config to use `https://`
+   once Splunk Web is back on HTTPS.
+
+### Issuer Not Allowed from SAIA Backend
+
+**Symptom:** SAIA returns:
+
+```json
+{"detail": "Issuer 'https://127.0.0.1:8089' is not allowed"}
 ```
+
+(or whatever the old issuer was)
+
+**Root cause:** `SPLUNK_ISSUERS` is a key in the SAIA config `ConfigMap`. The
+operator sets it to the hardcoded default (`https://splunk-splunk-standalone-standalone-service:8089`)
+when the key is absent or empty.
+
+Apply the exact issuer value in [SAIA ConfigMap Values](#saia-configmap-values), restart the SAIA pods, and re-test.
+
+### Splunk Restart Did Not Apply
+
+**Symptom:** `sudo /opt/splunk/bin/splunk restart` appears to succeed (or
+silently exits 1) but the old config is still active — new tokens still carry
+the stale `issuer_uri`, and `splunkd` keeps the same PID.
+
+**Root cause:** Splunk is owned by a non-root user (e.g. `ec2-user`). Running
+`sudo splunk restart` switches to root, which cannot stop/start the process
+owned by another user. The command exits without touching the running process.
+
+Follow [Restart Splunk Correctly](#restart-splunk-correctly), then repeat the restart and effective-configuration checks.
+
+### Troubleshooting Quick Reference
+
+| Symptom | Most likely cause | Section |
+|---------|-------------------|---------|
+| `Unable to load keys for signing interactive JWT` | Missing `[oauth2_settings]` in `authentication.conf` | [JWT signing error](#jwt-signing-key-error) |
+| `401 Unauthorized` from SAIA, JWKS fetch fails | `issuer_uri = https://127.0.0.1:8089` | [401 / JWKS failure](#401-unauthorized-from-saia-backend) |
+| Browser `blocked:mixed-content`, request never sent | Splunk HTTPS + SAIA HTTP | [Mixed content](#browser-mixed-content-block) |
+| `{"detail":"Issuer '...' is not allowed"}` | External issuer not in `SPLUNK_ISSUERS` allowlist | [Issuer not allowed](#issuer-not-allowed-from-saia-backend) |
+| Config change has no effect after restart | Restarted with `sudo` but Splunk owned by another user | [Restart did not apply](#splunk-restart-did-not-apply) |
+| Fresh fix works but old browser session still fails | Stale JWT from before the restart — log out and back in | [End-to-end verification](#end-to-end-verification) |
+| Patching `AIPlatform.splunkConfiguration.endpoint` doesn't fix issuer | That field is the HEC endpoint, not the issuer — patch `SPLUNK_ISSUERS` in the ConfigMap directly | [Issuer not allowed](#issuer-not-allowed-from-saia-backend) |

From 4bacceeb617d4a6c062a2f45af66a2ff1571c789 Mon Sep 17 00:00:00 2001
From: Kumar Pratyush <kpratyush@splunk.com>
Date: Mon, 10 Aug 2026 09:59:15 +0530
Subject: [PATCH 3/3] docs: simplify external Splunk integration flow to 4
 sections

Drop the standalone Problem Statement and Testing sections; fold
verification steps into "What the Customer Should Do" so the doc reads
as: what to do, which files to edit, what values to provide, then
what to do if testing fails.

Co-Authored-By: Claude <noreply@anthropic.com>
---
 .../EXTERNAL_SPLUNK_INTEGRATION.md            | 137 +++++++++---------
 1 file changed, 67 insertions(+), 70 deletions(-)

diff --git a/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md b/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
index 9236c051..5a105446 100644
--- a/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
+++ b/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md
@@ -1,30 +1,15 @@
 # External Splunk Integration with Splunk AI Platform
 
 Runbook for connecting an **externally-hosted Splunk Enterprise instance** (outside the
-k0s cluster) to the Splunk AI Platform backend (SAIA). The required actions,
-files, and values are presented first. Failure modes encountered in practice are
-covered after testing.
+k0s cluster) to the Splunk AI Platform backend (SAIA). What the customer should do,
+which files to edit, and what values to provide are covered first. Failure modes
+encountered in practice are covered at the end.
 
 Use this when:
 - Splunk Enterprise runs on a separate host (bare-metal, EC2, VM) — not the bundled
   in-cluster Splunk standalone deployed by the installer.
 - The SAIA backend (`AIService`) must validate JWT tokens issued by that external Splunk.
 
----
-
-## Table of Contents
-
-- [1. Problem Statement](#1-problem-statement)
-- [2. What the Customer Should Do](#2-what-the-customer-should-do)
-- [3. Files and Resources to Edit](#3-files-and-resources-to-edit)
-- [4. Values to Provide](#4-values-to-provide)
-- [5. Testing](#5-testing)
-- [6. If Testing Fails](#6-if-testing-fails)
-
----
-
-## 1. Problem Statement
-
 ### Architecture Overview
 
 ```
@@ -59,7 +44,16 @@ ConfigMap key — patched directly as described in
 
 ---
 
-## 2. What the Customer Should Do
+## Table of Contents
+
+- [1. What the Customer Should Do](#1-what-the-customer-should-do)
+- [2. Files and Resources to Edit](#2-files-and-resources-to-edit)
+- [3. Values to Provide](#3-values-to-provide)
+- [4. If Testing Fails](#4-if-testing-fails)
+
+---
+
+## 1. What the Customer Should Do
 
 ### Prerequisites
 
@@ -124,9 +118,48 @@ Complete the integration in this order:
     grep -E "oauth2|JWT|signing" $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -10
     ```
 
+### Verify the Fix
+
+1. **Verify issuer reachability.** Confirm port 8089 is reachable **from the
+   k0s cluster** (this is the path that actually performs JWT validation — a
+   check from your laptop can pass while SAIA pods are still blocked by a
+   security group or firewall):
+
+    ```bash
+    # Run from any k0s cluster node (e.g. the installer or controller)
+    nc -zv <public-ip> 8089
+    # or
+    curl -sk https://<public-ip>:8089/services/server/info | grep -c "<title>"
+    ```
+
+2. **Confirm a fresh token uses the correct issuer:**
+
+    ```bash
+    # Grab a fresh token from Splunk AI Assistant log
+    grep "Successfully retrieved interactive token" \
+      $SPLUNK_HOME/var/log/splunk/splunk_ai_assistant.log | tail -3
+    ```
+
+3. **Get a fresh browser token.** Old tokens signed before the real restart
+   still carry the stale issuer and will fail even after the fix. Log out and
+   back in to the Splunk AI Assistant to force a new token.
+
+4. **Test end-to-end:**
+
+    ```bash
+    # From the browser, send a prompt in Splunk AI Assistant
+    # Expected: response returned without error
+    ```
+
+5. **Confirm SAIA accepted the token** (check SAIA v1 pod logs):
+
+    ```bash
+    kubectl logs -n <namespace> <saia-v1-pod> --tail=20 | grep -E "200|401|issuer|token"
+    ```
+
 ---
 
-## 3. Files and Resources to Edit
+## 2. Files and Resources to Edit
 
 | File or resource | Customer change |
 |---|---|
@@ -139,7 +172,7 @@ Complete the integration in this order:
 
 ---
 
-## 4. Values to Provide
+## 3. Values to Provide
 
 ### Splunk JWT Values
 
@@ -315,53 +348,7 @@ truth and leave only its issuer in `SPLUNK_ISSUERS`.
 
 ---
 
-## 5. Testing
-
-### Verify Issuer Reachability
-
-Verify port 8089 is reachable **from the k0s cluster** (this is the path that
-actually performs JWT validation — a check from your laptop can pass while
-SAIA pods are still blocked by a security group or firewall):
-
-```bash
-# Run from any k0s cluster node (e.g. the installer or controller)
-nc -zv <public-ip> 8089
-# or
-curl -sk https://<public-ip>:8089/services/server/info | grep -c "<title>"
-```
-
-### Confirm a Fresh Token Uses the Correct Issuer
-
-Confirm a new token carries the correct issuer:
-
-```bash
-# Grab a fresh token from Splunk AI Assistant log
-grep "Successfully retrieved interactive token" \
-  $SPLUNK_HOME/var/log/splunk/splunk_ai_assistant.log | tail -3
-```
-
-### End-to-End Verification
-
-1. **Get a fresh token.** Old tokens signed before the real restart still carry
-   the stale issuer and will fail even after the fix. Log out and back in to
-   the Splunk AI Assistant to force a new token.
-
-2. **Test end-to-end:**
-
-    ```bash
-    # From the browser, send a prompt in Splunk AI Assistant
-    # Expected: response returned without error
-    ```
-
-3. **Confirm SAIA accepted the token** (check SAIA v1 pod logs):
-
-    ```bash
-    kubectl logs -n <namespace> <saia-v1-pod> --tail=20 | grep -E "200|401|issuer|token"
-    ```
-
----
-
-## 6. If Testing Fails
+## 4. If Testing Fails
 
 ### JWT Signing Key Error
 
@@ -374,7 +361,6 @@ Unable to load keys for signing interactive JWT
 **Root cause:** The `[oauth2_settings]` stanza in `authentication.conf` is missing
 or empty — `AuthenticationRSAKeysManager` has no certificate to sign tokens with.
 
-
 Confirm the error:
 
 ```bash
@@ -499,6 +485,17 @@ owned by another user. The command exits without touching the running process.
 
 Follow [Restart Splunk Correctly](#restart-splunk-correctly), then repeat the restart and effective-configuration checks.
 
+### Fresh Fix Works but Old Browser Session Still Fails
+
+**Symptom:** The values above are all correct and verified, but the browser
+still gets a `401` or an "issuer not allowed" error.
+
+**Root cause:** A stale JWT issued before the fix (or before the real restart)
+is still cached in the browser session.
+
+Log out and back in to the Splunk AI Assistant to force a new token, then
+repeat [Verify the Fix](#verify-the-fix).
+
 ### Troubleshooting Quick Reference
 
 | Symptom | Most likely cause | Section |
@@ -508,5 +505,5 @@ Follow [Restart Splunk Correctly](#restart-splunk-correctly), then repeat the re
 | Browser `blocked:mixed-content`, request never sent | Splunk HTTPS + SAIA HTTP | [Mixed content](#browser-mixed-content-block) |
 | `{"detail":"Issuer '...' is not allowed"}` | External issuer not in `SPLUNK_ISSUERS` allowlist | [Issuer not allowed](#issuer-not-allowed-from-saia-backend) |
 | Config change has no effect after restart | Restarted with `sudo` but Splunk owned by another user | [Restart did not apply](#splunk-restart-did-not-apply) |
-| Fresh fix works but old browser session still fails | Stale JWT from before the restart — log out and back in | [End-to-end verification](#end-to-end-verification) |
+| Fresh fix works but old browser session still fails | Stale JWT from before the restart — log out and back in | [Stale browser session](#fresh-fix-works-but-old-browser-session-still-fails) |
 | Patching `AIPlatform.splunkConfiguration.endpoint` doesn't fix issuer | That field is the HEC endpoint, not the issuer — patch `SPLUNK_ISSUERS` in the ConfigMap directly | [Issuer not allowed](#issuer-not-allowed-from-saia-backend) |