feat(app): search across multiple log and trace sources at once - #2862
feat(app): search across multiple log and trace sources at once#2862teeohhem wants to merge 3 commits into
Conversation
The search page's source selector can expand into a multi-select (up to 5 log/trace sources). Each selected source runs its own query pipeline (own connection, Lucene serializer, windowed pagination) with its SELECT rewritten to canonical aliases; a client-side k-way merge interleaves the streams by timestamp behind a "safe frontier" so the timeline never shows a gap a lagging source could still fill. Results render in one table with per-row source badges, a histogram stacked by source, summed totals, and an add-column picker over the union of the sources' columns. Multi mode is Lucene-only and URL-shareable (?sources=). Saved searches, alerts, filters, and delta/pattern modes stay single-source and are gated off with explanations; single-source behavior is unchanged. Also fixes chSqlToAliasMap dropping NULL-literal projections, which would have made row-WHERE clauses reference nonexistent columns.
🦋 Changeset detectedLatest commit: 0ab39a2 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThe PR adds concurrent search across multiple log and trace sources, normalizes and merges their rows, and renders combined counts and histograms.
Confidence Score: 4/5The PR is not yet safe to merge because source-defined ascending or non-timestamp-first ordering can still violate the merged timeline's descending-stream assumption. Explicit source order expressions are passed through unchanged while the multi-source merge always applies descending frontier semantics, leaving the previously reported ordering failure reachable. Files Needing Attention: packages/app/src/DBSearchPage.tsx and packages/app/src/components/MultiSourceRowTable.tsx
|
| Filename | Overview |
|---|---|
| packages/app/src/DBSearchPage.tsx | Adds multi-source URL/form state, source-specific query specifications, and conditional multi-source result rendering. |
| packages/app/src/components/MultiSourceRowTable.tsx | Orchestrates per-source paginated queries, merges normalized rows, and integrates row details and source status. |
| packages/app/src/utils/multiSourceMerge.ts | Implements timestamp-based stream coverage, frontier calculation, row interleaving, and lagging-source selection. |
| packages/app/src/components/MultiSourceTimeChart.tsx | Executes and combines per-source histogram and count queries for the multi-source presentation. |
| packages/common-utils/src/core/searchChartConfig.ts | Adds canonical multi-source search projections while preserving each source's query semantics. |
| packages/common-utils/src/clickhouse/index.ts | Preserves NULL literal projections in SQL alias recovery for downstream row predicates. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
URL[URL source selection] --> Config[Per-source chart configs]
Config --> Q1[Source query 1]
Config --> Q2[Source query 2]
Config --> QN[Source query N]
Q1 --> Merge[Safe-frontier merge]
Q2 --> Merge
QN --> Merge
Merge --> Table[Normalized result timeline]
Config --> Hist[Per-source histograms]
Hist --> Chart[Stacked chart and summed count]
Reviews (3): Last reviewed commit: "feat(app): cap multi-source search at 3 ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| {streams.map((stream, i) => { | ||
| if (stream.spec == null) return null; | ||
| const name = stream.spec.source.name; | ||
| return ( |
There was a problem hiding this comment.
This new 472-line component combines query orchestration, pagination, merge state, side-panel state, and rendering; MultiSourceTimeChart.tsx is also 320 lines. Split these concerns into smaller modules to satisfy the repository's 300-line component limit and make the behavior independently testable.
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
E2E Test Results✅ All tests passed • 276 passed • 1 skipped • 906s
Tests ran across 4 shards in parallel. |
Three components were each hand-unrolling slot0..slot4 hook calls to keep a constant hook count across a variable source selection. Rules of hooks allow calling a use*-named function parameter from a custom hook, so one generic useMultiSourceSlots(items, useSlot, opts) now owns the unrolling and the MAX_SEARCH_SOURCES pin; consumers are a single call. Slot hooks return memoized values so the returned array is dependency-list safe.
Query cost scales linearly with the selection (~3 ClickHouse queries per source per refresh, re-fired every live-tail tick, plus one-time metadata), and past 3 sources the interleaved timeline stops being legible. 3 covers the common "app logs + infra logs + traces" case.
1 participant
Summary
Searching has meant picking exactly one source, so a question that spans sources (several log sources, or logs plus traces) takes multiple tabs and manual correlation. The search page's source selector can now expand into a multi-select (up to 3 log/trace sources): every selected source runs its own query pipeline, and the results interleave into one timestamp-ordered timeline with per-row source badges, normalized columns (Timestamp, Source, Service, Level, Message, and Duration when traces are included), a histogram stacked by source, and summed totals.
Schemas are reconciled by projecting each source's semantic expressions (
timestampValueExpression,severityTextExpression/statusCodeExpression,bodyExpression/spanNameExpression, ...) under shared column aliases, and an add-column picker offers the union of the selected sources' columns (blank cells where a source lacks one). There is noUNION ALL: sources can live on different ClickHouse connections, each source keeps its per-table machinery (Lucene serializer, text-index detection, materialized-column rewrites, query settings), and a failing source degrades to a status chip instead of failing the whole search. A client-side k-way merge orders rows behind a "safe frontier" (the timestamp every source has covered), so the timeline never shows a gap a slower source could still fill, and "fetch more" advances only the lagging source.The selection is shareable via a
?sources=URL param. Multi mode is Lucene-only for now; saved searches, alerts, sidebar filters, and delta/pattern modes remain single-source and are gated off with explanations. Single-source search is unchanged. Also fixeschSqlToAliasMapdroppingNULLliteral projections, which made row-WHERE clauses reference nonexistent columns.Screenshots or video
Performance
Each source’s stream arrives already timestamp-sorted (its SQL ORDER BY), and the browser does a k-way merge: parse timestamps (cached per distinct raw value), filter to the “safe frontier,” one stable sort. Row volume is bounded — 200 rows/source/page, the table stops auto-fetching at 1,000 rendered rows, and live mode caps retention at 5 pages/source — so the merge sorts a few thousand rows at most, single-digit milliseconds against network round-trips of hundreds. Memory is a few MB worst case. Client-side merging is also what buys the properties we wanted: sources on different ClickHouse connections work, and one failing source degrades to a chip instead of killing the query. The trade-off it does have: total rows read from ClickHouse can exceed what’s displayed, because each source fetches its own LIMIT while the frontier holds some rows back — that’s the conservative-correctness cost, bounded by the same caps.
How to test on Vercel preview
Preview routes: /search
Steps: