Skip to content

Ai tier ga internal - #182

Merged
kupratyu-splunk merged 204 commits into
mainfrom
ai-tier-ga-internal
Aug 27, 2026
Merged

Ai tier ga internal#182
kupratyu-splunk merged 204 commits into
mainfrom
ai-tier-ga-internal

Conversation

@kupratyu-splunk

Copy link
Copy Markdown
Collaborator

Description

Related Issues

  • Related to #

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test improvement
  • CI/CD improvement
  • Chore (dependency updates, etc.)

Changes Made

Testing Performed

  • Unit tests pass (make test)
  • Linting passes (make lint)
  • Integration tests pass (if applicable)
  • E2E tests pass (if applicable)
  • Manual testing performed

Test Environment

  • Kubernetes Version:
  • Cloud Provider:
  • Deployment Method:

Test Steps

Documentation

  • Updated inline code comments
  • Updated README.md (if adding features)
  • Updated API documentation
  • Updated deployment guides
  • Updated CHANGELOG.md
  • No documentation needed

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published
  • I have updated the Helm chart version (if applicable)
  • I have updated CRD schemas (if applicable)

Breaking Changes

Impact:

Migration Path:

Screenshots/Recordings

Additional Notes

Reviewer Notes

Please pay special attention to:


Commit Message Convention: This PR follows Conventional Commits

kbhos-splunk and others added 30 commits June 15, 2026 12:32
feat(AIP-3938): AI tier openshift support
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
feat(AIP-4163): Ai pod script enhancements
feat(AIP-4161): Openshift air gap install script
Bring ai-tier-ga in sync with main
… script issues (#126)

* feat(openshift): route GPU/CPU workers via ai-tier-node label

* resolve codex comments
chore: Merge ai-pod-feature-branch to ai-tier-ga
Creates EC2 infrastructure consumed by k0s_cluster_with_stack.sh:
- VPC + public subnet + IGW in a single AZ
- Security group with self-referencing rule (private-IP k0s comms)
- RHEL 9 instances: controller(s), CPU workers, GPU workers, installer
- Installer gets an EIP; k0s nodes stay on private IPs only
- Separate EBS volumes for /var/lib/k0s (GPU) and /data/minio (installer)
- Auto-creates or reuses AWS key pair, downloads .pem locally
- Optional MinIO install via install_minio_ec2.sh after provisioning
- Generates my-k0s-config.yaml on the installer with private IPs pre-filled
- Commands: provision / output / status / destroy / validate / dry-run
- Test config (t3.medium, ~$1-2) for Level 2 validation
- Design + test plan in K0S_AWS_PROVISION.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…mount fix

Rewrites CloudFormation-based provisioner to direct AWS CLI calls,
bypassing SCP p-m68tib3s which blocks ec2:RunInstances via CloudFormation.

Key changes:
- IMDSv2 required (HttpTokens=required) on all instances — required by SCP
- EBS encryption on all volumes — required by SCP
- Existing VPC only (vpc-09b191e89c83d588e) — no new VPC creation
- Separate subnets: k0s nodes in private subnet, installer in public subnet with EIP
- SSH ProxyCommand instead of -J flag (fixes key forwarding on macOS)
- mount_disk_via_ssh: mount EBS data disks via SSH after attach, not UserData
  (UserData runs before EBS volumes are attached by the provisioner)
- destroy --yes / -y flag for non-interactive teardown
- Fix _INSTANCE grep pattern to include INSTALLER_INSTANCE (was missing)
- Fix ${var,,} bash 3.2 incompatibility with tr [:upper:] [:lower:]

Validated end-to-end: provision → status → output → destroy all pass.
All 4 instances, EBS volumes, EIP, SG, key pair created and destroyed cleanly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on steps, failure history

Updates the provisioner doc to reflect the rewrite from CloudFormation to
direct AWS CLI calls. Adds:
- SCP compliance table (IMDSv2, EBS encryption, existing VPC requirement)
- Corrected architecture diagram (public vs private subnet split)
- Full k0s integration guide (Option A from installer, Option B from laptop + ProxyJump)
- Development history: 9 documented failures with root causes and fixes
  (SCP block, VPC, set -e false negative, private subnet EIP, SSH -J key,
   EBS mount timing, INSTALLER_INSTANCE grep, destroy TTY, bash 3.2 compat)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s-west-2 VPC

When network.vpcId is empty (or omitted in non-us-west-2 regions), the
provisioner now creates the full network stack automatically:
  VPC → IGW → public subnet + route (0.0.0.0/0 → IGW)
  → NAT GW EIP → NAT GW → private subnet + route (0.0.0.0/0 → NAT GW)

All six resources are tagged and tracked in the state file so destroy
tears them down cleanly in reverse order (private RT → private subnet
→ NAT GW → NAT EIP → public RT → public subnet → IGW → VPC).

For us-west-2, the default vpcId remains vpc-09b191e89c83d588e (SCP in
splunkcloud-ai-dev blocks new-VPC creation; leaving the default avoids
breaking existing deployments). Any other region defaults to auto-create.

Config additions:
  network.vpcCidr          (default 10.0.0.0/16)
  network.publicSubnetCidr (default 10.0.1.0/24)
  network.privateSubnetCidr (default 10.0.2.0/24)

Docs updated with Network Modes table, mandatory-fields list, and
full config reference with inline comments for each network field.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… to not create AWS resources

- Add WARNING comments to ensure_network and destroy_network (untested code path)
- Emit runtime warn() when auto-create mode activates
- dry-run: skip pick_subnet in auto-create mode (no VPC exists yet)
- dry-run: remove ensure_key_pair call (was creating real key pairs in AWS)

Dry-run now creates zero AWS resources in both existing-VPC and auto-create modes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…writing

Old behaviour: cat > my-k0s-config.yaml (full overwrite with hardcoded defaults)
New behaviour:
  1. If my-k0s-config.yaml exists on installer → back it up as
     my-k0s-config.bak-<timestamp>.yaml, then yq-patch in-place.
  2. If it doesn't exist → copy k0s-cluster-config.yaml (already
     uploaded by setup_installer) as the base, then yq-patch.

Only infrastructure fields are written:
  cluster.{name,region,sshKeyPath,sshUser}
  nodes.existingIPs.{controllers,workers}
  storage.objectStore.{type,bucket,endpoint,auth.*}  (when minio.enabled)

All other fields (images, operators, aiPlatform, metallb, ecr, etc.)
are preserved from the user's own file.

IPs are serialised via jq to a proper JSON array before yq injection,
avoiding quoting issues with multi-node configs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kupratyu-splunk and others added 12 commits August 21, 2026 13:39
Ubuntu admins had no path to install Docker or crane despite the mandatory
image-mirroring steps requiring one of them; document crane (cross-platform,
no daemon/root needed) as the default and Docker as an alternative on both
Ubuntu and RHEL 9. Also parameterize the mirroring loops with a TAG variable
instead of hardcoding "preview".
Admins whose browser only has SSH access to a bastion/installer machine had
no documented way to reach Splunk Web or the SAIA API. Add a SOCKS-tunnel
walkthrough to K0S_README.md and link it from the quick-reference guide.
Derive the Splunk port-forward target from NAMESPACE/STANDALONE_NAME
instead of hardcoding the default service name, and add a Linux
(google-chrome) command alongside the macOS-only open -na example.

Co-Authored-By: Claude <noreply@anthropic.com>
docs: document SSH SOCKS tunnel for accessing internal Splunk/SAIA
fix(saia): roll v2 workloads on issuer changes
fix(ai-tier): pin Ray 2.56 and disable BiEncoder
@kupratyu-splunk

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 810547a238

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/ai/sidecars/builder.go
Comment thread pkg/ai/features/slim/impl.go
Comment thread pkg/ai/features/slim/impl.go
Comment thread tools/cluster_setup/k0s_cluster_with_stack.sh
@kupratyu-splunk
kupratyu-splunk merged commit 7af17ec into main Aug 27, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants