PagesController#dashboard renders all analyses unpaginated — 55-70s TTFB at ~950 analyses

[anchapin]

Problem

The dashboard page (GET /) renders every analysis in the database unpaginated. On a production cluster with ~950 analyses, each page load takes 55-70 seconds (TTFB), measured via curl inside the web container.

Root cause: PagesController#dashboard runs multiple aggregate queries against the os_docker.data_points collection (706k documents, avg 268 KB each) without any filter, limit, or cursor. Each aggregate is a COLLSCAN of the full collection — roughly 185 GB of logical I/O per call. There are ~14 such aggregate calls per page render.

Reproduction

1. Deploy openstudio-server chart with ~950+ analyses and ~700k data points
2. curl -w '%{time_starttransfer}' http://<web-service>/
3. TTFB: 55-70 seconds (sequential), 64-67 seconds (burst of 5 parallel)

Control route (/__kube_probe__ — intentionally returns 404): 2-4 ms.

Impact

• UI unusable: Every page load ties up a Passenger process for 55-70 seconds
• Cascading load: 590 Passenger processes × concurrent requests = massive MongoDB connection count
• Probe unusable: The old readiness probe used GET /, so failure detection took ~35 minutes
• Workaround applied: Helm chart changes (PR Bump OpenStudio version to OS.1.13.3.44ac130fa7 #124 on openstudio-server-helm) reduce pool size and fix probes, but the underlying Rails issue remains

Proposed fixes

1. Add pagination to the dashboard data_points aggregate — limit to N most recent analyses, with offset/cursor for scrolling
2. Add a materialized count or cached summary to avoid full-collection scans on every page load
3. Consider adding an index on os_docker.data_points.analysis_id if one doesn't already exist
4. Add server-side caching (e.g., Rails fragment cache or HTTP Cache-Control headers) for the dashboard response

Environment

• openstudio-server chart 0.7.0, app version 3.8.0-1
• ~950 analyses, 706k data points, 285 GB MongoDB on 32 GB RAM
• 1 web pod (192Gi memory), 590 Passenger processes (old formula)

Related

• openstudio-server-helm PR Bump OpenStudio version to OS.1.13.3.44ac130fa7 #124 (chart-level workarounds)
• Cluster profiling data available on request