Cluster: lab.ocp.local | OCP 4.19.42 | Proxmox (3 masters + 2 workers)
This repository is a living record of issues encountered on the OCP lab cluster. Each issue has its own folder containing the RCA, fix steps, and any scripts used.
Reading commands in this repo: anything wrapped in angle brackets —
<new-username>, <generated-password>, <target-namespace>, <date>,
etc. — is a placeholder, not literal text. Replace the entire token,
including the < and > characters, with your actual value before running
the command. Typing a placeholder literally will either fail outright or,
worse, silently create a resource named e.g. <new-username>. This applies
throughout every issues/ and checklists/ document.
| # | Title | Date | Severity | Status |
|---|---|---|---|---|
| 01 | Web Console Unreachable After Cluster Restart | 2026-06-30 | High | Resolved |
| 02 | Minor Version Upgrade 4.15.59 → 4.16.55 | 2026-06-30 | Medium | Completed |
| 03 | OVN-Kubernetes Crash Loop on Rebooted Nodes (Web Console Down) | 2026-07-01 | High | Resolved |
| 04 | oc Client/Server Version Skew After Cluster Upgrade |
2026-07-01 | Low | Resolved |
| 05 | MTV VM Migration Readiness (ESXi/vCenter → OpenShift Virtualization) | 2026-07-01 | N/A (Planned Migration) | Precheck complete — Blocked on capacity |
| 06 | master-2 Transient NotReady / <unknown> Metrics After Node Reboot |
2026-07-20 | Low | Resolved (self-healed) |
| 07 | Recurring Kubelet Cert Expiry After Extended Shutdown; Cron Automation Blind Spot Found & Fixed | 2026-08-04 | Medium | Resolved |
| 08 | Upgrade 4.19.41 → 4.19.42: Worker Image Pull Stall (IPv6 DNS) + Post-Upgrade Channel Drift | 2026-08-20 | Low / Medium | Resolved |
| 09 | Upgrade 4.18.50 → 4.19.41: Master Node Stuck on Extensions Image Pull | 2026-08-18 | Medium | Resolved (self-recovered) |
| 10 | Onboard babus as Named Cluster-Admin (Patching & Upgrade Duties) |
2026-08-20 | N/A (Administration) | Completed |
| 11 | Cluster Patch Readiness Review: 4.19.42 → 4.19.43 Security Z-Stream | 2026-08-20 | N/A (Change Readiness Review) | Review complete — Blocked on target availability |
| 12 | Uninstall Idle OpenShift Virtualization (CNV) to Reclaim Resources | 2026-08-20 | N/A (Resource Reclamation) | Completed |
OCP_Issue-Fix_RCA/
├── README.md # This file — issue index
│
├── issues/ # One folder per issue
│ └── 01-web-console-unreachable/
│ ├── README.md # Issue summary & quick fix
│ ├── RCA.md # Full root cause analysis
│ └── scripts/
│ └── approve-csrs.sh # Automated recovery script
│
├── checklists/ # Operational checklists
│ ├── cluster-startup.md # Run on every cluster restart
│ ├── admin-user-onboarding.md # Add a traceable named cluster-admin user
│ └── z-stream-patch-procedure.md # Z-stream patch upgrade: pre/post checks, downtime, backup/restore plan
│
└── scripts/ # Shared/reusable scripts
└── approve-csrs.sh # (symlink to latest version)
| Resource | Value |
|---|---|
| OCP Version | 4.19.42 |
| Console | https://console-openshift-console.apps.lab.ocp.local |
| API | https://api.lab.ocp.local:6443 |
| HAProxy (Load Balancer) | svc-infra.ocp.local — 192.168.29.10 |
| Masters | 192.168.29.21 / .22 / .23 |
| Workers | 192.168.29.31 / .32 |
| SSH Key | ~/.ssh/ocp4-key (user: core) |
| kubeconfig | /home/centos/ocp/install/auth/kubeconfig |
- Create a folder:
issues/NN-short-description/ - Add
README.md(summary + quick fix) - Add
RCA.md(full root cause analysis) - Add
scripts/(any fix scripts used) - Add a row to the Issue Index table above