Skip to content

fix(migration): include id in compress_orderby for hypertables#180

Merged
TerrifiedBug merged 1 commit intomainfrom
fix/timescaledb-compression-pk-orderby
Apr 26, 2026
Merged

fix(migration): include id in compress_orderby for hypertables#180
TerrifiedBug merged 1 commit intomainfrom
fix/timescaledb-compression-pk-orderby

Conversation

@TerrifiedBug
Copy link
Copy Markdown
Owner

@TerrifiedBug TerrifiedBug commented Apr 26, 2026

Summary

Follow-up to #179. Bringing up the demo on TimescaleDB surfaced two further bugs in 20260329000001_timescaledb_compression. Both are in this PR.

Bug 1 — id missing from segmentby/orderby

ERROR: column "id" must be used for segmenting or ordering

The previous migration changed each hypertable's primary key to (id, timestamp). TimescaleDB then refuses to enable compression unless every column of the hypertable's unique indexes appears in either compress_segmentby or compress_orderby. The id column was in neither.

Fix: add id as a secondary compress_orderby column for all four hypertables. Choosing orderby (not segmentby) because id is a high-cardinality cuid — segmenting by it would produce one segment per row and destroy the compression ratio.

Bug 2 — option-string parser folds identifiers to lowercase

ERROR: column "pipelineid" does not exist
HINT: The timescaledb.compress_segmentby option must reference a valid column.

Same case-sensitivity bug class as #179, but on the segmentby/orderby option strings instead of the regclass argument. Prisma's columns are camelCase (pipelineId, nodeId), and the option-string parser folds unquoted identifiers to lowercase — so 'pipelineId' was being looked up as pipelineid.

Fix: wrap every column name in double quotes inside the option strings. Same pattern as the table-name fix in #179.

Final state of each ALTER TABLE

ALTER TABLE "PipelineMetric" SET (
  timescaledb.compress,
  timescaledb.compress_segmentby = '"pipelineId"',
  timescaledb.compress_orderby = '"timestamp" DESC, "id"'
);

…and analogous for NodeMetric ("nodeId"), PipelineLog ("pipelineId"), NodeStatusEvent ("nodeId").

Why this didn't surface earlier

Same as #179: every prior deployment ran on plain Postgres, where the IF EXISTS (... timescaledb) guard makes this whole migration a no-op. The hosted demo on timescale/timescaledb:2.16.0-pg16 is the first environment in the wild with the extension installed.

Recovery instructions

For anyone whose DB is currently in the failed state on this migration:

Option A — wipe and reapply (cleanest if no real data, e.g. demo):

docker compose down -v
docker volume ls | grep vf-demo-pg   # verify gone, manual rm if needed
docker compose up

Option B — preserve data:

npx prisma migrate resolve --rolled-back 20260329000001_timescaledb_compression
npx prisma migrate deploy

Drift warning for plain-Postgres users

Same caveat as #179: the migration's checksum changes. On next prisma migrate deploy you may see a "migration was modified after applied" warning. Resolve with:

npx prisma migrate resolve --applied 20260329000001_timescaledb_compression

One-time fix; the no-op behaviour on plain Postgres is unchanged.

Test plan

  • Visual inspection — every camelCase column name in segmentby/orderby is now wrapped in escaped quotes; every orderby ends with "id"
  • Pull this branch into the demo deployment, wipe the volume, restart — both 20260329000000 and 20260329000001 apply cleanly
  • Verify compression policies exist: SELECT hypertable_name, attname, segmentby_column_index, orderby_column_index FROM timescaledb_information.compression_settings; — should show pipelineId/nodeId as a segmentby column and timestamp + id as orderby columns

The hypertables migration changed each hypertable's primary key to
(id, timestamp). TimescaleDB rejects ALTER TABLE ... SET (compress, ...)
unless every column of the hypertable's unique indexes is present in
either compress_segmentby or compress_orderby — so the next migration
errored with 'column id must be used for segmenting or ordering' on
every TimescaleDB-enabled deployment.

Adding id to compress_orderby satisfies the constraint without harming
compression: id is a high-cardinality cuid, so segmenting by it would
produce one segment per row and destroy the ratio.
@github-actions github-actions Bot added the fix label Apr 26, 2026
@TerrifiedBug TerrifiedBug merged commit a2bdd2b into main Apr 26, 2026
7 checks passed
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Apr 26, 2026

Greptile Summary

Adds id as a secondary compress_orderby column for all four TimescaleDB hypertables (PipelineMetric, NodeMetric, PipelineLog, NodeStatusEvent), resolving the ERROR: column "id" must be used for segmenting or ordering failure introduced when the previous migration changed each table's primary key to (id, timestamp). The fix is applied consistently across all four tables, uses orderby (not segmentby) to avoid the high-cardinality-CUID segment explosion, and the IF EXISTS (timescaledb) guard keeps the migration a no-op on plain PostgreSQL.

Confidence Score: 5/5

Safe to merge — targeted, correct fix with no regressions on plain PostgreSQL.

The change is a minimal, well-reasoned fix: id is added as a secondary compress_orderby column on all four affected hypertables, satisfying TimescaleDB's requirement that every column in a unique index appear in segmentby or orderby. The placement in orderby (rather than segmentby) is correct to avoid the CUID cardinality problem. The fix is applied symmetrically across all four tables, the extension guard is preserved, and plain-Postgres behaviour is unchanged. No logic, security, or correctness issues found.

No files require special attention.

Important Files Changed

Filename Overview
prisma/migrations/20260329000001_timescaledb_compression/migration.sql Adds id to compress_orderby for all four hypertables — correct placement, consistent across tables, no-op guard preserved.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Migration starts] --> B{timescaledb extension present?}
    B -- No --> C[RAISE NOTICE: skipping\nno-op on plain Postgres]
    B -- Yes --> D[PipelineMetric\ncompress_segmentby=pipelineId\ncompress_orderby=timestamp DESC, id]
    D --> E[add_compression_policy\ncompress_after=24h]
    E --> F[NodeMetric\ncompress_segmentby=nodeId\ncompress_orderby=timestamp DESC, id]
    F --> G[add_compression_policy\ncompress_after=24h]
    G --> H[PipelineLog\ncompress_segmentby=pipelineId\ncompress_orderby=timestamp DESC, id]
    H --> I[add_compression_policy\ncompress_after=24h]
    I --> J[NodeStatusEvent\ncompress_segmentby=nodeId\ncompress_orderby=timestamp DESC, id]
    J --> K[add_compression_policy\ncompress_after=24h]
    K --> L[RAISE NOTICE: compression policies enabled]
Loading

Reviews (1): Last reviewed commit: "fix(migration): include id in compress_o..." | Re-trigger Greptile

@github-actions github-actions Bot added fix and removed fix labels Apr 26, 2026
@TerrifiedBug TerrifiedBug deleted the fix/timescaledb-compression-pk-orderby branch April 26, 2026 18:32
TerrifiedBug added a commit that referenced this pull request Apr 26, 2026
#182)

Bare SELECT inside a PL/pgSQL DO block is invalid when the result is
discarded — Postgres raises 42601 "query has no destination for result
data". Switch the four add_compression_policy calls to PERFORM so the
migration applies cleanly on TimescaleDB.

Follows #179, #180, #181 — same file, same DO block.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant