Skip to content

Commit 11ef472

Browse files
feat(web): add deployment system stats to service ping (#1424)
* feat(web): add deployment system stats to service ping Collects a best-effort snapshot of the deployment's host/container resources (CPU cores + cgroup CPU quota, host + cgroup memory, disk for the data volume, load average, platform) and includes it as an optional `systemInfo` object on the service ping payload. This surfaces resource facts in the `lh_ping` PostHog event so we can quickly diagnose resource issues (e.g. a customer running out of RAM) and point them towards scaling. Container-aware cgroup limits/usage are read because os.totalmem()/os.cpus() report the host, not the container's limits. The `systemInfo` field is optional for back-compat with Lighthouse deployments that predate it, and every collected value is best-effort (reported as null when unreadable) so it can never break the ping. Fixes SOU-1488 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add CHANGELOG entry for service ping system stats [#1424] Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * remove uneeded comment * refactor(web): report service ping memory/disk stats in MiB Report the systemInfo memory and disk fields in whole MiB (binary, 1024-based units) instead of raw bytes, so the values are directly human-readable when diagnosing resource issues (e.g. a 16Gi container limit reads as 16384). Converts at the collection boundary via a bytesToMiB helper and updates the mirrored docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(web): trim service ping systemInfo to container-scoped fields Drops the host/node-level readings from the systemInfo snapshot (cpuCores, loadAverage1m, totalMemoryMiB, freeMemoryMiB). These reported the underlying machine rather than the deployment's own allocation, which leaked node sizing (especially in k8s) and added noise without aiding the "is this deployment resource-constrained?" diagnosis. What remains is purely container/volume-scoped: cpuQuota (cgroup CPU limit), memoryLimitMiB/memoryUsedMiB (cgroup), and diskTotalMiB/ diskFreeMiB (statfs on the data volume). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 9c4780d commit 11ef472

5 files changed

Lines changed: 201 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2020
- [EE] Added DPoP sender-constrained OAuth tokens for MCP clients. [#1395](https://github.com/sourcebot-dev/sourcebot/pull/1395)
2121
- [EE] Added text file attachments to Ask Sourcebot, letting users attach text/code/config files to a chat message via the paperclip button, drag-and-drop, or paste, with large pastes auto-converted to attachments. [#1374](https://github.com/sourcebot-dev/sourcebot/pull/1374)
2222
- [EE] Added image attachments to Ask Sourcebot, letting users attach images to a chat message when the selected model supports image input. [#1375](https://github.com/sourcebot-dev/sourcebot/pull/1375)
23+
- Added deployment system resource stats (CPU cores + cgroup quota, host + container memory, disk, load average) to the service ping, so resource issues can be diagnosed more quickly. [#1424](https://github.com/sourcebot-dev/sourcebot/pull/1424)
2324

2425
### Fixed
2526
- Send anonymous server-side PostHog events as personless so unauthenticated requests don't inflate person counts. [#1367](https://github.com/sourcebot-dev/sourcebot/pull/1367)

docs/docs/misc/service-ping.mdx

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,19 @@ The data contained within the Service Ping is limited to:
2525
| `isTelemetryEnabled` | `boolean` | Whether anonymous product telemetry is enabled. |
2626
| `isLanguageModelConfigured` | `boolean` | Whether at least one language model is configured. |
2727
| `activationCode` | `string` | Activation code, if your instance has one bound. |
28+
| `systemInfo` | `object` | Resource facts about the deployment (CPU, memory, disk). See below. |
29+
30+
The `systemInfo` object contains a best-effort snapshot of the resources the deployment is running with. It is used to diagnose resource issues (e.g. insufficient RAM). Any field that cannot be read is reported as `null`.
31+
32+
| Field | Type | Description |
33+
| --- | --- | --- |
34+
| `platform` | `string` | Operating system platform (e.g. `linux`). |
35+
| `arch` | `string` | CPU architecture (e.g. `x64`, `arm64`). |
36+
| `cpuQuota` | `number \| null` | Effective cgroup CPU limit in cores, or `null` if unlimited. |
37+
| `memoryLimitMiB` | `number \| null` | cgroup memory limit in MiB, or `null` if unlimited. |
38+
| `memoryUsedMiB` | `number \| null` | Current cgroup memory usage, in MiB. |
39+
| `diskTotalMiB` | `number \| null` | Total disk space for the data volume, in MiB. |
40+
| `diskFreeMiB` | `number \| null` | Free disk space for the data volume, in MiB. |
2841

2942

3043
You can fetch the schema for the Service Ping by submitting a GET request to `https://deployments.sourcebot.dev/schema`
@@ -59,6 +72,15 @@ body: {
5972
"deploymentType": "other",
6073
"isTelemetryEnabled": true,
6174
"isLanguageModelConfigured": true,
75+
"systemInfo": {
76+
"platform": "linux",
77+
"arch": "x64",
78+
"cpuQuota": 4,
79+
"memoryLimitMiB": 8192,
80+
"memoryUsedMiB": 3072,
81+
"diskTotalMiB": 102400,
82+
"diskFreeMiB": 54968
83+
},
6284
"activationCode": "sb_act_3a925f51...."
6385
}
6486
```

packages/web/src/features/billing/servicePing.ts

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import { ServicePingRequest } from "./types";
1414
import { ServiceErrorException } from "@/lib/serviceError";
1515
import { getConfiguredLanguageModels } from "@/features/chat/utils.server";
1616
import { activeMembershipWhere } from "@/features/membership/utils";
17+
import { getSystemInfo } from "./systemInfo";
1718

1819
const logger = createLogger('service-ping');
1920

@@ -79,6 +80,12 @@ export const syncWithLighthouse = async (orgId: number) => {
7980

8081
const isLanguageModelConfigured = (await getConfiguredLanguageModels()).length > 0;
8182

83+
// Best-effort — a failure to collect system info must never prevent the ping.
84+
const systemInfo = await getSystemInfo().catch((error) => {
85+
logger.warn(`Failed to collect system info for service ping: ${error}`);
86+
return undefined;
87+
});
88+
8289
const payload: ServicePingRequest = {
8390
installId: env.SOURCEBOT_INSTALL_ID,
8491
version: SOURCEBOT_VERSION,
@@ -91,6 +98,7 @@ export const syncWithLighthouse = async (orgId: number) => {
9198
deploymentType: inferDeploymentType(),
9299
isTelemetryEnabled: env.SOURCEBOT_TELEMETRY_DISABLED === 'false',
93100
isLanguageModelConfigured,
101+
...(systemInfo && { systemInfo }),
94102
...(activationCode && { activationCode }),
95103
};
96104

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
import os from "node:os";
2+
import fs from "node:fs/promises";
3+
import { env, createLogger } from "@sourcebot/shared";
4+
import { SystemInfo } from "./types";
5+
6+
const logger = createLogger('system-info');
7+
8+
// cgroup v2 (unified hierarchy).
9+
const CGROUP_V2_MEMORY_MAX = '/sys/fs/cgroup/memory.max';
10+
const CGROUP_V2_MEMORY_CURRENT = '/sys/fs/cgroup/memory.current';
11+
const CGROUP_V2_CPU_MAX = '/sys/fs/cgroup/cpu.max';
12+
13+
// cgroup v1 (legacy hierarchy).
14+
const CGROUP_V1_MEMORY_LIMIT = '/sys/fs/cgroup/memory/memory.limit_in_bytes';
15+
const CGROUP_V1_MEMORY_USAGE = '/sys/fs/cgroup/memory/memory.usage_in_bytes';
16+
const CGROUP_V1_CPU_QUOTA = '/sys/fs/cgroup/cpu/cpu.cfs_quota_us';
17+
const CGROUP_V1_CPU_PERIOD = '/sys/fs/cgroup/cpu/cpu.cfs_period_us';
18+
19+
/**
20+
* Collects a snapshot of the resources this deployment is running with, so we
21+
* can quickly diagnose resource issues (e.g. "the customer doesn't have enough
22+
* RAM") from the service ping.
23+
*
24+
* Everything is best-effort: any value we can't read is reported as `null` so a
25+
* partial snapshot still goes out. Resource figures come from the container's
26+
* cgroup (and `statfs` for disk) rather than host-level readings, which in a
27+
* Dockerized or Kubernetes deployment report the underlying machine, not the
28+
* container's actual limits.
29+
*/
30+
export const getSystemInfo = async (): Promise<SystemInfo> => {
31+
const [memoryLimitBytes, memoryUsedBytes, cpuQuota, disk] = await Promise.all([
32+
readCgroupMemoryLimit(),
33+
readCgroupMemoryUsage(),
34+
readCgroupCpuQuota(),
35+
readDiskUsage(env.DATA_CACHE_DIR),
36+
]);
37+
38+
return {
39+
platform: os.platform(),
40+
arch: os.arch(),
41+
cpuQuota,
42+
memoryLimitMiB: memoryLimitBytes === null ? null : bytesToMiB(memoryLimitBytes),
43+
memoryUsedMiB: memoryUsedBytes === null ? null : bytesToMiB(memoryUsedBytes),
44+
diskTotalMiB: disk.totalBytes === null ? null : bytesToMiB(disk.totalBytes),
45+
diskFreeMiB: disk.freeBytes === null ? null : bytesToMiB(disk.freeBytes),
46+
};
47+
};
48+
49+
const BYTES_PER_MIB = 1024 * 1024;
50+
51+
// Memory and disk are both reported in whole MiB (binary, 1024-based units),
52+
// matching how RAM and container/k8s volumes are sized.
53+
const bytesToMiB = (bytes: number): number => Math.round(bytes / BYTES_PER_MIB);
54+
55+
const readFileTrimmed = async (path: string): Promise<string | null> => {
56+
try {
57+
const contents = await fs.readFile(path, 'utf8');
58+
return contents.trim();
59+
} catch {
60+
// The file won't exist outside of a cgroup-managed environment (e.g.
61+
// local dev on macOS), which is expected — treat as "unknown".
62+
return null;
63+
}
64+
};
65+
66+
/**
67+
* Parses a byte count from a cgroup file, returning `null` for "unlimited":
68+
* the literal `max` in cgroup v2, or a sentinel larger than JS can safely
69+
* represent in cgroup v1 (e.g. 9223372036854771712).
70+
*/
71+
const parseCgroupBytes = (raw: string | null): number | null => {
72+
if (raw === null || raw === 'max') {
73+
return null;
74+
}
75+
const value = Number(raw);
76+
if (!Number.isSafeInteger(value) || value < 0) {
77+
return null;
78+
}
79+
return value;
80+
};
81+
82+
const readCgroupMemoryLimit = async (): Promise<number | null> => {
83+
const v2 = parseCgroupBytes(await readFileTrimmed(CGROUP_V2_MEMORY_MAX));
84+
if (v2 !== null) {
85+
return v2;
86+
}
87+
return parseCgroupBytes(await readFileTrimmed(CGROUP_V1_MEMORY_LIMIT));
88+
};
89+
90+
const readCgroupMemoryUsage = async (): Promise<number | null> => {
91+
const v2 = parseCgroupBytes(await readFileTrimmed(CGROUP_V2_MEMORY_CURRENT));
92+
if (v2 !== null) {
93+
return v2;
94+
}
95+
return parseCgroupBytes(await readFileTrimmed(CGROUP_V1_MEMORY_USAGE));
96+
};
97+
98+
/**
99+
* Returns the effective CPU limit in cores (e.g. 0.5, 2), or `null` when there
100+
* is no quota (unlimited) or it can't be read.
101+
*/
102+
const readCgroupCpuQuota = async (): Promise<number | null> => {
103+
// cgroup v2: `cpu.max` is "<quota> <period>" (in microseconds), or
104+
// "max <period>" when unlimited.
105+
const v2 = await readFileTrimmed(CGROUP_V2_CPU_MAX);
106+
if (v2 !== null) {
107+
const [quotaRaw, periodRaw] = v2.split(/\s+/);
108+
if (quotaRaw === 'max') {
109+
return null;
110+
}
111+
const quota = Number(quotaRaw);
112+
const period = Number(periodRaw);
113+
if (Number.isFinite(quota) && quota > 0 && Number.isFinite(period) && period > 0) {
114+
return roundToTwo(quota / period);
115+
}
116+
return null;
117+
}
118+
119+
// cgroup v1: cpu.cfs_quota_us / cpu.cfs_period_us; a quota of -1 is unlimited.
120+
const quota = Number(await readFileTrimmed(CGROUP_V1_CPU_QUOTA));
121+
const period = Number(await readFileTrimmed(CGROUP_V1_CPU_PERIOD));
122+
if (!Number.isFinite(quota) || quota <= 0 || !Number.isFinite(period) || period <= 0) {
123+
return null;
124+
}
125+
return roundToTwo(quota / period);
126+
};
127+
128+
const readDiskUsage = async (path: string): Promise<{ totalBytes: number | null; freeBytes: number | null }> => {
129+
try {
130+
const stats = await fs.statfs(path);
131+
return {
132+
totalBytes: stats.blocks * stats.bsize,
133+
// bavail is blocks available to unprivileged users, which best
134+
// reflects what the deployment can actually write.
135+
freeBytes: stats.bavail * stats.bsize,
136+
};
137+
} catch (error) {
138+
logger.debug(`Failed to read disk usage for ${path}: ${error}`);
139+
return { totalBytes: null, freeBytes: null };
140+
}
141+
};
142+
143+
const roundToTwo = (n: number): number => Math.round(n * 100) / 100;

packages/web/src/features/billing/types.ts

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,30 @@
11
import { z } from "zod";
22

3+
/**
4+
* A best-effort snapshot of the host / container resources the deployment is
5+
* running with. Used to quickly diagnose resource issues (e.g. insufficient
6+
* RAM) from the service ping. Every field is best-effort; anything we can't
7+
* read is reported as `null`.
8+
*/
9+
export const systemInfoSchema = z.object({
10+
platform: z.string(),
11+
arch: z.string(),
12+
13+
// CPU. `cpuQuota` is the effective cgroup CPU limit in cores (null when
14+
// unset or unreadable), which is what a container is actually allowed to use.
15+
cpuQuota: z.number().nonnegative().nullable(),
16+
17+
// Memory, in MiB, from the cgroup: the container's actual RAM limit and
18+
// current usage (null when unset or unreadable).
19+
memoryLimitMiB: z.number().nonnegative().nullable(),
20+
memoryUsedMiB: z.number().nonnegative().nullable(),
21+
22+
// Disk, in MiB, for the DATA_CACHE_DIR volume (where repos are indexed).
23+
diskTotalMiB: z.number().nonnegative().nullable(),
24+
diskFreeMiB: z.number().nonnegative().nullable(),
25+
});
26+
export type SystemInfo = z.infer<typeof systemInfoSchema>;
27+
328
export const servicePingRequestSchema = z.object({
429
installId: z.string(),
530
version: z.string(),
@@ -18,6 +43,8 @@ export const servicePingRequestSchema = z.object({
1843
isTelemetryEnabled: z.boolean(),
1944
isLanguageModelConfigured: z.boolean(),
2045
activationCode: z.string().optional(),
46+
// optional for back-compat with Lighthouse deployments that predate it.
47+
systemInfo: systemInfoSchema.optional(),
2148
});
2249
export type ServicePingRequest = z.infer<typeof servicePingRequestSchema>;
2350

0 commit comments

Comments
 (0)