Skip to content

Update memory and add cpu requests to deployment configs#1920

Draft
juliawu wants to merge 5 commits into
datacommonsorg:masterfrom
juliawu:mixer-deployment-testing
Draft

Update memory and add cpu requests to deployment configs#1920
juliawu wants to merge 5 commits into
datacommonsorg:masterfrom
juliawu:mixer-deployment-testing

Conversation

@juliawu

@juliawu juliawu commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Updates the memory and CPU limits and requests in our deployment config files.

In order to address the mixer restart issues we've been seeing in production, this PR:

  • Adds CPU requests to each mixer pods to better right-size and bin-pack during deployment
  • Increases the memory request and limits for observation and node pods, as a band-aid for OOMKills seen in production.

More specifically, the relevant memory upgrades are:

observation pods:
  esp container: 512Gi request, 2Gi limit
  mixer container: 12Gi request, 20Gi limit

node pods:
  mixer container: 12Gi request, 12Gi limit

These values were selected based on both scaling recommendations via GKE's autoscaler recommender tool, and looking at historical memory usage over the month of May 2026.

Note that this does not fix the mixer restart issues, but because we cannot prioritize hunting for the root cause at this time, we are masking this issue until we can come back to a fix later in the year.

Testing strategy:
These changes have already been deployed successfully via a manual deployment. This PR just checks in the code so that these changes are included in all future deployments.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the mixer Helm chart to support dynamic configuration of ESP container resources (CPU and memory) globally or per service group, standardizes memory units to "Gi", and adds CPU requests across several service groups. Feedback suggests simplifying the ESP resource assignment logic in deployment.yaml by removing redundant conditional checks, as $.Values.esp is already guaranteed to be defined.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread deploy/helm_charts/mixer/templates/deployment.yaml
@juliawu juliawu requested review from ajaits and dhotchkiss June 30, 2026 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant