diff --git a/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md b/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md index ecc45a0d..5a105446 100644 --- a/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md +++ b/tools/cluster_setup/EXTERNAL_SPLUNK_INTEGRATION.md @@ -1,34 +1,16 @@ # External Splunk Integration with Splunk AI Platform Runbook for connecting an **externally-hosted Splunk Enterprise instance** (outside the -k0s cluster) to the Splunk AI Platform backend (SAIA). Covers every failure mode -encountered in practice, in the order you are likely to hit them. +k0s cluster) to the Splunk AI Platform backend (SAIA). What the customer should do, +which files to edit, and what values to provide are covered first. Failure modes +encountered in practice are covered at the end. Use this when: - Splunk Enterprise runs on a separate host (bare-metal, EC2, VM) — not the bundled in-cluster Splunk standalone deployed by the installer. - The SAIA backend (`AIService`) must validate JWT tokens issued by that external Splunk. ---- - -## Table of Contents - -- [Architecture Overview](#architecture-overview) -- [Prerequisites](#prerequisites) -- [Step 1 — Fix JWT Signing Key Error](#step-1--fix-jwt-signing-key-error) -- [Step 2 — Fix 401 Unauthorized from SAIA Backend](#step-2--fix-401-unauthorized-from-saia-backend) -- [Step 3 — Fix Browser Mixed-Content Block](#step-3--fix-browser-mixed-content-block) - - [Option A — Disable Splunk Web SSL (temporary workaround)](#option-a--disable-splunk-web-ssl-temporary-workaround) - - [Option B — TLS Termination via Load Balancer or Ingress (production fix)](#option-b--tls-termination-via-load-balancer-or-ingress-production-fix) -- [Step 4 — Fix "Issuer Not Allowed" from SAIA Backend](#step-4--fix-issuer-not-allowed-from-saia-backend) -- [Step 5 — Restart Splunk Correctly](#step-5--restart-splunk-correctly) -- [Step 6 — Final Verification](#step-6--final-verification) -- [Cleanup](#cleanup) -- [Troubleshooting Quick Reference](#troubleshooting-quick-reference) - ---- - -## Architecture Overview +### Architecture Overview ``` Browser (Splunk AI Assistant) @@ -57,59 +39,78 @@ endpoint. Key constraint: the SAIA backend fetches the public signing keys from `/.well-known/oauth2_keys` at JWT validation time. The `issuer_uri` in the token must exactly match an entry in the SAIA backend's `SPLUNK_ISSUERS` -ConfigMap key — patched directly as described in Step 4. +ConfigMap key — patched directly as described in +[SAIA ConfigMap Values](#saia-configmap-values). --- -## Prerequisites +## Table of Contents + +- [1. What the Customer Should Do](#1-what-the-customer-should-do) +- [2. Files and Resources to Edit](#2-files-and-resources-to-edit) +- [3. Values to Provide](#3-values-to-provide) +- [4. If Testing Fails](#4-if-testing-fails) + +--- + +## 1. What the Customer Should Do + +### Prerequisites - SSH access to the external Splunk host - `kubectl` access to the k0s cluster running SAIA - The external Splunk host's public IP or FQDN (used as `issuer_uri`) - Port **8089** (Splunk management) open from the k0s cluster nodes to the Splunk host ---- - -## Step 1 — Fix JWT Signing Key Error +Complete the integration in this order: -**Symptom:** Splunk log shows: +1. SSH into the Splunk host. +2. Configure JWT signing in `authentication.conf` with a routable issuer. +3. Restart Splunk as the user that owns the installation. +4. Verify that the k0s cluster can reach the Splunk management port. +5. Place a TLS-terminating load balancer or ingress in front of SAIA. +6. Update the SAIA URL in Splunk AI Assistant. +7. Add the exact issuer to the SAIA `SPLUNK_ISSUERS` ConfigMap value. +8. Restart the SAIA pods and complete the end-to-end test. -``` -Unable to load keys for signing interactive JWT -``` +### Restart Splunk Correctly -**Root cause:** The `[oauth2_settings]` stanza in `authentication.conf` is missing -or empty — `AuthenticationRSAKeysManager` has no certificate to sign tokens with. +**Correct procedure:** -**Fix:** +1. Check who owns the Splunk install: -1. SSH into the Splunk host. + ```bash + stat -c '%U' $SPLUNK_HOME # Linux + ``` -2. Confirm the error: +2. Run as the owning user. If you are already logged in as the owner: ```bash - grep "Unable to load keys for signing interactive JWT" \ - $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -5 + $SPLUNK_HOME/bin/splunk stop + $SPLUNK_HOME/bin/splunk start --answer-yes --accept-license ``` -3. Edit `$SPLUNK_HOME/etc/system/local/authentication.conf` and add: + If the SSH user is different from the owner (e.g. logged in as `ec2-user` + but Splunk is owned by `splunk`), use `sudo -H -u`: - ```ini - [oauth2_settings] - issuer_uri = https://:8089 - certFile = $SPLUNK_HOME/etc/auth/server.pem - sslPassword = + ```bash + SPLUNK_OWNER=$(stat -c '%U' $SPLUNK_HOME) + sudo -H -u "${SPLUNK_OWNER}" $SPLUNK_HOME/bin/splunk stop + sudo -H -u "${SPLUNK_OWNER}" $SPLUNK_HOME/bin/splunk start --answer-yes --accept-license ``` - To find the passphrase, check `server.conf`'s `[sslConfig]` stanza, or decrypt it: +3. Confirm the PID actually changed (a matching old/new PID means it didn't restart): ```bash - $SPLUNK_HOME/bin/splunk show-decrypted --value '' + $SPLUNK_HOME/bin/splunk status ``` - A typical default passphrase for a fresh Splunk install is `password`. +4. Confirm the config took effect: -4. Restart Splunk (see [Step 5](#step-5--restart-splunk-correctly) for the correct procedure). + ```bash + $SPLUNK_HOME/bin/splunk btool authentication list oauth2_settings + # issuer_uri must show your public IP, not 127.0.0.1 + ``` 5. Verify the key is now loaded: @@ -117,31 +118,12 @@ or empty — `AuthenticationRSAKeysManager` has no certificate to sign tokens wi grep -E "oauth2|JWT|signing" $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -10 ``` ---- - -## Step 2 — Fix 401 Unauthorized from SAIA Backend +### Verify the Fix -**Symptom:** SAIA returns `401 Unauthorized`. Splunk AI Assistant log shows -token fetch succeeds but SAIA rejects it. - -**Root cause:** `issuer_uri` is set to `https://127.0.0.1:8089`. The SAIA -backend can't reach `127.0.0.1` on the Splunk host — it resolves to its own -loopback. The JWT validation (JWKS fetch) fails. - -**Fix:** - -1. Change `issuer_uri` in `authentication.conf` to the public IP or FQDN: - - ```ini - [oauth2_settings] - issuer_uri = https://:8089 - certFile = $SPLUNK_HOME/etc/auth/server.pem - sslPassword = - ``` - -2. Verify port 8089 is reachable **from the k0s cluster** (this is the path that - actually performs JWT validation — a check from your laptop can pass while - SAIA pods are still blocked by a security group or firewall): +1. **Verify issuer reachability.** Confirm port 8089 is reachable **from the + k0s cluster** (this is the path that actually performs JWT validation — a + check from your laptop can pass while SAIA pods are still blocked by a + security group or firewall): ```bash # Run from any k0s cluster node (e.g. the installer or controller) @@ -150,13 +132,7 @@ loopback. The JWT validation (JWKS fetch) fails. curl -sk https://:8089/services/server/info | grep -c "" ``` - If the check passes from your laptop but fails from the cluster, update the - Splunk host's security-group inbound rules to allow port 8089 from the - cluster nodes' IP range. - -3. Restart Splunk (see [Step 5](#step-5--restart-splunk-correctly)). - -4. Confirm a new token carries the correct issuer: +2. **Confirm a fresh token uses the correct issuer:** ```bash # Grab a fresh token from Splunk AI Assistant log @@ -164,54 +140,60 @@ loopback. The JWT validation (JWKS fetch) fails. $SPLUNK_HOME/var/log/splunk/splunk_ai_assistant.log | tail -3 ``` ---- +3. **Get a fresh browser token.** Old tokens signed before the real restart + still carry the stale issuer and will fail even after the fix. Log out and + back in to the Splunk AI Assistant to force a new token. -## Step 3 — Fix Browser Mixed-Content Block +4. **Test end-to-end:** -**Symptom:** Browser dev tools → Network tab shows `blocked:mixed-content`. -The SAIA API call is blocked before it even leaves the browser. - -**Root cause:** Splunk Web is served over **HTTPS** (port 8000), but the SAIA -backend URL configured in the AI Assistant app uses plain **HTTP** -(e.g. `http://15.164.171.171:30080`). Browsers enforce mixed-content policy -and block HTTPS pages from making HTTP sub-requests. + ```bash + # From the browser, send a prompt in Splunk AI Assistant + # Expected: response returned without error + ``` -**Options (choose one):** +5. **Confirm SAIA accepted the token** (check SAIA v1 pod logs): -| Option | When to use | -|--------|-------------| -| [Option A — Disable Splunk Web SSL](#option-a--disable-splunk-web-ssl-temporary-workaround) | Testing / short-term debugging only | -| [Option B — TLS Termination via Load Balancer or Ingress](#option-b--tls-termination-via-load-balancer-or-ingress-production-fix) | Production — eliminates root cause | + ```bash + kubectl logs -n <namespace> <saia-v1-pod> --tail=20 | grep -E "200|401|issuer|token" + ``` --- -### Option A — Disable Splunk Web SSL (temporary workaround) +## 2. Files and Resources to Edit -1. Edit `$SPLUNK_HOME/etc/system/local/web.conf`: +| File or resource | Customer change | +|---|---| +| `$SPLUNK_HOME/etc/system/local/authentication.conf` | Add or update `[oauth2_settings]` | +| `server.conf` `[sslConfig]` | Read the encrypted certificate passphrase; do not edit it for this procedure | +| Load balancer, ingress, firewall, security-group, DNS, and certificate resources | Publish SAIA over HTTPS and allow the required network paths | +| Splunk AI Assistant app-local `splunkaiassistant.conf` | Set the SAIA URL used by Splunk AI Assistant; the app-local path depends on the installed app version | +| SAIA ConfigMap `<name>-saia-config` | Set `data.SPLUNK_ISSUERS` | +| `$SPLUNK_HOME/etc/system/local/web.conf` | Edit only for the temporary mixed-content workaround under [If Testing Fails](#browser-mixed-content-block) | - ```ini - [settings] - enableSplunkWebSSL = 0 - ``` +--- -2. Restart Splunk (see [Step 5](#step-5--restart-splunk-correctly)). +## 3. Values to Provide -3. Verify Splunk Web is now HTTP: +### Splunk JWT Values - ```bash - curl -sv http://<public-ip>:8000 2>&1 | grep -E "< HTTP|Location" - # Expected: HTTP/1.1 303 or 200 +Edit `$SPLUNK_HOME/etc/system/local/authentication.conf` and add: - # Confirm HTTPS is no longer serving (port 8000 still open but speaks HTTP, - # so TLS negotiation fails — not "connection refused"): - curl -sv https://<public-ip>:8000 2>&1 | grep -E "SSL|TLS|handshake|wrong version|unknown protocol" - # Expected: one of the above TLS error strings - ``` +```ini +[oauth2_settings] +issuer_uri = https://<PUBLIC_IP_OR_FQDN>:8089 +certFile = $SPLUNK_HOME/etc/auth/server.pem +sslPassword = <passphrase> +``` -> **Remember to revert this** once the SAIA backend is served over HTTPS or -> you are done testing. See [Cleanup](#cleanup). +To find the passphrase, check `server.conf`'s `[sslConfig]` stanza, or decrypt it: ---- +```bash +$SPLUNK_HOME/bin/splunk show-decrypted --value '<encrypted value from server.conf>' +``` + +A typical default passphrase for a fresh Splunk install is `password`. + +Restart Splunk (see [Restart Splunk Correctly](#restart-splunk-correctly) for the correct procedure). ### Option B — TLS Termination via Load Balancer or Ingress (production fix) @@ -273,21 +255,9 @@ SAIA service (in-cluster HTTP) | Browser blocks XHR from HTTPS Splunk page | Same scheme — no block | | Requires keeping Splunk Web on HTTP | Splunk Web can stay on HTTPS | ---- - -## Step 4 — Fix "Issuer Not Allowed" from SAIA Backend +### SAIA ConfigMap Values -**Symptom:** SAIA returns: - -```json -{"detail": "Issuer 'https://127.0.0.1:8089' is not allowed"} -``` - -(or whatever the old issuer was) - -**Root cause:** `SPLUNK_ISSUERS` is a key in the SAIA config `ConfigMap`. The -operator sets it to the hardcoded default (`https://splunk-splunk-standalone-standalone-service:8089`) -when the key is absent or empty. Importantly: +The following details explain why this resource is edited: - `AIService.spec.splunkConfiguration.endpoint` is the **HEC telemetry endpoint** (used by the log-forwarding sidecar as `<endpoint>/services/collector`) — it is @@ -311,11 +281,11 @@ when the key is absent or empty. Importantly: `iss` claim in each JWT: ```bash - # Use the exact issuer_uri value from Step 1/2, e.g.: + # Use the exact issuer_uri value from authentication.conf, e.g.: # https://43.203.164.228:8089 (if configured as IP) # https://splunk.example.com:8089 (if configured as FQDN) kubectl patch configmap <name>-saia-config -n <namespace> --type merge \ - -p '{"data":{"SPLUNK_ISSUERS":"https://<EXACT_ISSUER_URI>:8089"}}' + -p '{"data":{"SPLUNK_ISSUERS":"<EXACT_ISSUER_URI>"}}' ``` To allow **both** the in-cluster Splunk and the external Splunk simultaneously, @@ -323,7 +293,7 @@ when the key is absent or empty. Importantly: ```bash kubectl patch configmap <name>-saia-config -n <namespace> --type merge \ - -p '{"data":{"SPLUNK_ISSUERS":"https://splunk-splunk-standalone-standalone-service:8089 https://<EXACT_ISSUER_URI>:8089"}}' + -p '{"data":{"SPLUNK_ISSUERS":"https://splunk-splunk-standalone-standalone-service:8089 <EXACT_ISSUER_URI>"}}' ``` 3. Confirm the value is set: @@ -349,81 +319,131 @@ when the key is absent or empty. Importantly: 5. Wait for pods to reach `1/1 Running`, then re-test. +**Check for side-effects on the k0s cluster:** + +If the k0s cluster previously had its own bundled Splunk standalone, the +ConfigMap patch above may have replaced the in-cluster issuer with the +external one. Verify nothing else on the cluster depended on the bundled +instance: + +```bash +# Check if the in-cluster Splunk standalone still exists and is healthy +kubectl get standalone -n ai-platform +kubectl get pods -n ai-platform | grep splunk + +# Check current SPLUNK_ISSUERS value in the ConfigMap +kubectl get configmap -n ai-platform -o json | jq -r '.items[].data | select(has("SPLUNK_ISSUERS")) | .SPLUNK_ISSUERS' +``` + +If the in-cluster Splunk is still deployed and needs to be trusted alongside +the external one, patch `SPLUNK_ISSUERS` with both URLs as shown above: + +```bash +kubectl patch configmap <name>-saia-config -n <namespace> --type merge \ + -p '{"data":{"SPLUNK_ISSUERS":"https://splunk-splunk-standalone-standalone-service:8089 https://<PUBLIC_IP>:8089"}}' +``` + +Otherwise, decommission whichever Splunk instance is no longer the source of +truth and leave only its issuer in `SPLUNK_ISSUERS`. + --- -## Step 5 — Restart Splunk Correctly +## 4. If Testing Fails -**Symptom:** `sudo /opt/splunk/bin/splunk restart` appears to succeed (or -silently exits 1) but the old config is still active — new tokens still carry -the stale `issuer_uri`, and `splunkd` keeps the same PID. +### JWT Signing Key Error -**Root cause:** Splunk is owned by a non-root user (e.g. `ec2-user`). Running -`sudo splunk restart` switches to root, which cannot stop/start the process -owned by another user. The command exits without touching the running process. +**Symptom:** Splunk log shows: -**Correct procedure:** +``` +Unable to load keys for signing interactive JWT +``` -1. Check who owns the Splunk install: +**Root cause:** The `[oauth2_settings]` stanza in `authentication.conf` is missing +or empty — `AuthenticationRSAKeysManager` has no certificate to sign tokens with. - ```bash - stat -c '%U' $SPLUNK_HOME # Linux - ``` +Confirm the error: -2. Run as the owning user. If you are already logged in as the owner: +```bash +grep "Unable to load keys for signing interactive JWT" \ + $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -5 +``` - ```bash - $SPLUNK_HOME/bin/splunk stop - $SPLUNK_HOME/bin/splunk start --answer-yes --accept-license - ``` +Apply the values in [Splunk JWT Values](#splunk-jwt-values), restart Splunk, and repeat the JWT signing test. - If the SSH user is different from the owner (e.g. logged in as `ec2-user` - but Splunk is owned by `splunk`), use `sudo -H -u`: +### 401 Unauthorized from SAIA Backend - ```bash - SPLUNK_OWNER=$(stat -c '%U' $SPLUNK_HOME) - sudo -H -u "${SPLUNK_OWNER}" $SPLUNK_HOME/bin/splunk stop - sudo -H -u "${SPLUNK_OWNER}" $SPLUNK_HOME/bin/splunk start --answer-yes --accept-license - ``` +**Symptom:** SAIA returns `401 Unauthorized`. Splunk AI Assistant log shows +token fetch succeeds but SAIA rejects it. -3. Confirm the PID actually changed (a matching old/new PID means it didn't restart): +**Root cause:** `issuer_uri` is set to `https://127.0.0.1:8089`. The SAIA +backend can't reach `127.0.0.1` on the Splunk host — it resolves to its own +loopback. The JWT validation (JWKS fetch) fails. - ```bash - $SPLUNK_HOME/bin/splunk status - ``` +**Fix:** -4. Confirm the config took effect: +1. Change `issuer_uri` in `authentication.conf` to the public IP or FQDN: - ```bash - $SPLUNK_HOME/bin/splunk btool authentication list oauth2_settings - # issuer_uri must show your public IP, not 127.0.0.1 + ```ini + [oauth2_settings] + issuer_uri = https://<YOUR_PUBLIC_IP>:8089 + certFile = $SPLUNK_HOME/etc/auth/server.pem + sslPassword = <passphrase> ``` ---- +2. If the reachability check passes from your laptop but fails from the cluster, + update the Splunk host's security-group inbound rules to allow port 8089 from + the cluster nodes' IP range. -## Step 6 — Final Verification +3. Restart Splunk (see [Restart Splunk Correctly](#restart-splunk-correctly)). -1. **Get a fresh token.** Old tokens signed before the real restart still carry - the stale issuer and will fail even after the fix. Log out and back in to - the Splunk AI Assistant to force a new token. +Confirm the issuer and network values in [Splunk JWT Values](#splunk-jwt-values), then repeat the issuer reachability test. -2. **Test end-to-end:** +### Browser Mixed-Content Block - ```bash - # From the browser, send a prompt in Splunk AI Assistant - # Expected: response returned without error - ``` +**Symptom:** Browser dev tools → Network tab shows `blocked:mixed-content`. +The SAIA API call is blocked before it even leaves the browser. -3. **Confirm SAIA accepted the token** (check SAIA v1 pod logs): +**Root cause:** Splunk Web is served over **HTTPS** (port 8000), but the SAIA +backend URL configured in the AI Assistant app uses plain **HTTP** +(e.g. `http://15.164.171.171:30080`). Browsers enforce mixed-content policy +and block HTTPS pages from making HTTP sub-requests. - ```bash - kubectl logs -n <namespace> <saia-v1-pod> --tail=20 | grep -E "200|401|issuer|token" - ``` +**Options (choose one):** ---- +| Option | When to use | +|--------|-------------| +| [Option A — Disable Splunk Web SSL](#option-a--disable-splunk-web-ssl-temporary-workaround) | Testing / short-term debugging only | +| [Option B — TLS Termination via Load Balancer or Ingress](#option-b--tls-termination-via-load-balancer-or-ingress-production-fix) | Production — eliminates root cause | + +#### Option A — Disable Splunk Web SSL (temporary workaround) + +1. Edit `$SPLUNK_HOME/etc/system/local/web.conf`: -## Cleanup +```ini +[settings] +enableSplunkWebSSL = 0 +``` + +2. Restart Splunk (see [Restart Splunk Correctly](#restart-splunk-correctly)). -After testing is complete, revert the temporary workaround from Step 3: +3. Verify Splunk Web is now HTTP: + +```bash +curl -sv http://<public-ip>:8000 2>&1 | grep -E "< HTTP|Location" +# Expected: HTTP/1.1 303 or 200 + +# Confirm HTTPS is no longer serving (port 8000 still open but speaks HTTP, +# so TLS negotiation fails — not "connection refused"): +curl -sv https://<public-ip>:8000 2>&1 | grep -E "SSL|TLS|handshake|wrong version|unknown protocol" +# Expected: one of the above TLS error strings +``` + +> **Remember to revert this** once the SAIA backend is served over HTTPS or +> you are done testing. See [Cleanup](#cleanup). + +#### Cleanup + +After testing is complete, revert the temporary workaround above: 1. Edit `$SPLUNK_HOME/etc/system/local/web.conf`: @@ -432,48 +452,58 @@ After testing is complete, revert the temporary workaround from Step 3: enableSplunkWebSSL = 1 ``` -2. Restart Splunk as the owning user (see [Step 5](#step-5--restart-splunk-correctly)). +2. Restart Splunk as the owning user (see [Restart Splunk Correctly](#restart-splunk-correctly)). 3. Update the SAIA URL in the Splunk AI Assistant app config to use `https://` once Splunk Web is back on HTTPS. -**Check for side-effects on the k0s cluster:** - -If the k0s cluster previously had its own bundled Splunk standalone, the -ConfigMap patch in Step 4 may have replaced the in-cluster issuer with the -external one. Verify nothing else on the cluster depended on the bundled -instance: +### Issuer Not Allowed from SAIA Backend -```bash -# Check if the in-cluster Splunk standalone still exists and is healthy -kubectl get standalone -n ai-platform -kubectl get pods -n ai-platform | grep splunk +**Symptom:** SAIA returns: -# Check current SPLUNK_ISSUERS value in the ConfigMap -kubectl get configmap -n ai-platform -o json | jq -r '.items[].data | select(has("SPLUNK_ISSUERS")) | .SPLUNK_ISSUERS' +```json +{"detail": "Issuer 'https://127.0.0.1:8089' is not allowed"} ``` -If the in-cluster Splunk is still deployed and needs to be trusted alongside -the external one, patch `SPLUNK_ISSUERS` with both URLs as shown in Step 4: +(or whatever the old issuer was) -```bash -kubectl patch configmap <name>-saia-config -n <namespace> --type merge \ - -p '{"data":{"SPLUNK_ISSUERS":"https://splunk-splunk-standalone-standalone-service:8089 https://<PUBLIC_IP>:8089"}}' -``` +**Root cause:** `SPLUNK_ISSUERS` is a key in the SAIA config `ConfigMap`. The +operator sets it to the hardcoded default (`https://splunk-splunk-standalone-standalone-service:8089`) +when the key is absent or empty. -Otherwise, decommission whichever Splunk instance is no longer the source of -truth and leave only its issuer in `SPLUNK_ISSUERS`. +Apply the exact issuer value in [SAIA ConfigMap Values](#saia-configmap-values), restart the SAIA pods, and re-test. ---- +### Splunk Restart Did Not Apply + +**Symptom:** `sudo /opt/splunk/bin/splunk restart` appears to succeed (or +silently exits 1) but the old config is still active — new tokens still carry +the stale `issuer_uri`, and `splunkd` keeps the same PID. + +**Root cause:** Splunk is owned by a non-root user (e.g. `ec2-user`). Running +`sudo splunk restart` switches to root, which cannot stop/start the process +owned by another user. The command exits without touching the running process. + +Follow [Restart Splunk Correctly](#restart-splunk-correctly), then repeat the restart and effective-configuration checks. + +### Fresh Fix Works but Old Browser Session Still Fails + +**Symptom:** The values above are all correct and verified, but the browser +still gets a `401` or an "issuer not allowed" error. + +**Root cause:** A stale JWT issued before the fix (or before the real restart) +is still cached in the browser session. + +Log out and back in to the Splunk AI Assistant to force a new token, then +repeat [Verify the Fix](#verify-the-fix). -## Troubleshooting Quick Reference +### Troubleshooting Quick Reference | Symptom | Most likely cause | Section | |---------|-------------------|---------| -| `Unable to load keys for signing interactive JWT` | Missing `[oauth2_settings]` in `authentication.conf` | [Step 1](#step-1--fix-jwt-signing-key-error) | -| `401 Unauthorized` from SAIA, JWKS fetch fails | `issuer_uri = https://127.0.0.1:8089` | [Step 2](#step-2--fix-401-unauthorized-from-saia-backend) | -| Browser `blocked:mixed-content`, request never sent | Splunk HTTPS + SAIA HTTP | [Step 3](#step-3--fix-browser-mixed-content-block) | -| `{"detail":"Issuer '...' is not allowed"}` | External issuer not in `SPLUNK_ISSUERS` allowlist | [Step 4](#step-4--fix-issuer-not-allowed-from-saia-backend) | -| Config change has no effect after restart | Restarted with `sudo` but Splunk owned by another user | [Step 5](#step-5--restart-splunk-correctly) | -| Fresh fix works but old browser session still fails | Stale JWT from before the restart — log out and back in | [Step 6](#step-6--final-verification) | -| Patching `AIPlatform.splunkConfiguration.endpoint` doesn't fix issuer | That field is the HEC endpoint, not the issuer — patch `SPLUNK_ISSUERS` in the ConfigMap directly | [Step 4](#step-4--fix-issuer-not-allowed-from-saia-backend) | +| `Unable to load keys for signing interactive JWT` | Missing `[oauth2_settings]` in `authentication.conf` | [JWT signing error](#jwt-signing-key-error) | +| `401 Unauthorized` from SAIA, JWKS fetch fails | `issuer_uri = https://127.0.0.1:8089` | [401 / JWKS failure](#401-unauthorized-from-saia-backend) | +| Browser `blocked:mixed-content`, request never sent | Splunk HTTPS + SAIA HTTP | [Mixed content](#browser-mixed-content-block) | +| `{"detail":"Issuer '...' is not allowed"}` | External issuer not in `SPLUNK_ISSUERS` allowlist | [Issuer not allowed](#issuer-not-allowed-from-saia-backend) | +| Config change has no effect after restart | Restarted with `sudo` but Splunk owned by another user | [Restart did not apply](#splunk-restart-did-not-apply) | +| Fresh fix works but old browser session still fails | Stale JWT from before the restart — log out and back in | [Stale browser session](#fresh-fix-works-but-old-browser-session-still-fails) | +| Patching `AIPlatform.splunkConfiguration.endpoint` doesn't fix issuer | That field is the HEC endpoint, not the issuer — patch `SPLUNK_ISSUERS` in the ConfigMap directly | [Issuer not allowed](#issuer-not-allowed-from-saia-backend) |