An Iceberg-first SQL server that scales. Run SQE embedded as one binary on your laptop, or distributed across a cluster of stateless workers behind an Arrow Flight SQL, Trino HTTP, or MCP endpoint. Same SQL surface, same Iceberg semantics, same identity model.
SQE is a Rust-based SQL query engine for Apache Iceberg tables, built on DataFusion 54 and iceberg-rust. Every query runs as the authenticated user. No service account. No shared root.
# Point SQE at AWS S3 Tables (managed Iceberg) and run it as a SQL server.
cat > sqe.toml <<'EOF'
[catalog]
type = "s3tables"
table_bucket_arn = "arn:aws:s3tables:us-east-1:ACCOUNT:bucket/sales"
EOF
cargo run --release --bin sqe-server -- --config sqe.toml &
cargo run --bin sqe-cli -- --host localhost --port 50051
# Iceberg time travel + manifest-derived stats + per-query identity.
sqe> SELECT customer_id, sum(amount)
...> FROM s3tables.sales.orders FOR TIMESTAMP AS OF '2026-04-01'
...> WHERE region = 'EU' GROUP BY customer_id;
# Snapshot history straight from the metadata.
sqe> SELECT snapshot_id, committed_at FROM s3tables.sales."orders$snapshots";Top-five on the public Iceberg matrix. 167/189. 88.4%. Only non-Spark engine in the top five. Per-cell breakdown and a side-by-side against 20 other engines tracked internally.
Wins all seven benchmark suites against Trino 465 at SF1 (measured on 2026-08-15). TPC-H, SSB, TPC-DS, TPC-C, TPC-E, TPC-BB, ClickBench. 222 of 222 queries pass, and the results are differentially validated: every query runs against both engines and the rows are diffed, with DuckDB's official dsdgen as an independent data oracle. Tables and method below.
One binary scales from CLI to cluster. sqe-cli --embedded is a DuckDB-class single-process engine with the same SQL surface as the distributed coordinator. Persistent SQLite-backed Iceberg catalogs at ~/.sqe/warehouse/ survive restarts, served by a pure-Rust SQLite (the vendored Lab271 sqlite-rs): no C library in the default binary, and the file stays readable by stock sqlite3. Cross-catalog joins across multiple --catalog NAME=PATH mounts, plus runtime mounts via SQL ATTACH against any of the six supported backends (REST, Glue, S3 Tables, HMS, JDBC, SQLite).
Multi-catalog and multi-cloud, in one engine. Apache Polaris, Project Nessie, Unity Catalog OSS, AWS Glue (native SDK), AWS S3 Tables (native SDK), Hive Metastore, JDBC (Postgres, MySQL, SQLite), and Hadoop storage-only. Object stores: S3 (with endpoint override for Ceph, R2, Garage, MinIO), Azure ADLS, GCS, local filesystem, HuggingFace hf://.
Hive external tables, no metastore required (cycles 1, 2, 5 done; cycle 4 writes started). CREATE EXTERNAL TABLE with Athena and Glue syntax, against a warehouse root attached via SQL ATTACH ... TYPE hive_external, [[hive.roots]] config, or --hive-root NAME=URL. CSV, line-delimited JSON, and Parquet, with Hive-style k=v/ partitions, a five-class SerDe allow-list, and a sidecar JSON manifest shaped like Glue's TableInput. Cycle 2 persists sqe.partition_index in that manifest, maintained by MSCK REPAIR TABLE and ALTER TABLE ... ADD/DROP PARTITION, so a read plans from the index instead of listing the table prefix. MSCK REPAIR TABLE skips junk paths like Hive and Athena, and is a no-op on projection-enabled tables. SHOW PARTITIONS, Athena projection.* properties, and ANALYZE TABLE are in the same cycle. A stale index omits partitions rather than inventing them; MSCK REPAIR is the recovery path. Reads run on engine credentials or a named secret rather than the caller's own, bounded by the per-user [storage.tvf] allowlist and the policy layer, and a Hive table's grants key under <root>.<database> so they can never collide with a same-named Iceberg namespace. Full page: Hive External Tables; runnable demo with its own data generator in quickstart/hive-external-s3/. Cycle 5 moves Hive scans onto workers behind query.distributed_hive_scans (default false): one ScanTask per surviving partition, worker-side readers for all three formats, and a HiveLocalScanExec marker so EXPLAIN says why a scan did not distribute. A differential suite compares coordinator-local against worker-side rows per format. Cycle 4 adds writes: INSERT INTO appends into k=v/ partition directories and CTAS creates a partitioned table and loads it, both refreshing the partition index so the rows are visible to the next query. The visibility contract is stated rather than implied (per-file atomic, no multi-file atomicity, use Iceberg for snapshot isolation on write). INSERT OVERWRITE, DELETE, UPDATE and MERGE are refused rather than approximated, all for the same reason: there is no atomic multi-file commit to build them on. Planning a Hive table costs one metadata round trip per [hive] metadata_recheck_secs (default 10) rather than a manifest HEAD plus up to two data-prefix listings per lookup: the codec probe and the size-statistics listing are kept on the manifest's cache entry, and every DDL and write in the coordinator drops that entry, so the window only delays another writer's change (#663). Open: tag-based masks and row filters on Hive scans (cycle 3). Open within cycle 5: the multi-worker measurement, predicate and projection pushdown into a distributed task, and a streaming CSV/JSON decode.
Identity flows end to end. OIDC password grant, or a per-connection service principal that presents its own client_id/client_secret (the client_credentials_passthrough provider) over Flight SQL, the Trino HTTP path, or dbt (OAuth profile fields); demo in quickstart/polaris-ranger-service-principal/. The user's (or service principal's) bearer token is passed through to Polaris and S3 on every query. Row filters and column masks at the LogicalPlan layer are now enforced when [policy] engine = "ranger" is set: SQE downloads the shared frontend-query Ranger service policy set (query in the quickstarts, servicedef type hive) and rewrites the plan before DataFusion optimization (row filters above the scan, column masks that block predicate pushdown). Phase 1 covers MASK_NULL and row-filter expressions. Phase 2A delivers the full mask vocabulary: hash, partial show-first/last, date truncation to year/month/day, full redact, and custom expressions, all with type preservation through the physical planner. Phase 2B delivers session-context SQL functions (is_role_in_session, current_user, current_database, current_schema) that const-fold to literals before plan distribution. Phase 3a delivers tag-based masking: TagSource reads Iceberg sqe.column-tags properties, PolicyStore::resolve_tags maps tags to MaskType via Ranger tagPolicies, and the rewriter joins them with configurable precedence (policy.mask-precedence, default tag to match Spark/Kyuubi), unmappable-tag fail-closed, and full multi-level namespace identity. Validated end to end against a live Polaris + Ranger + Keycloak stack by the access_control_e2e suite (make test-access-control): grant/revoke, Ranger deny precedence, column masks with exact expected values, keyed HMAC hash masks, row filters, tag masks and tag row filters driven by SET TAG DDL, and tag fail-closed. That run found and fixed a real defect: Ranger tag policies name mask types with the owning component's prefix (hive:MASK_SHOW_LAST_4), which SQE did not accept, so every tag mask restricted the column instead of masking it. The end-to-end demo is in quickstart/polaris-ranger-keycloak/. GRANT on a table writes all three policies grant-profile.json v4 specifies for it (catalog namespace-list, namespace namespace-properties-read, then the table), because a table-level policy is inert on its own: SQE resolves a table only through a namespace its visibility probe could load, so the grant used to report success while the grantee still got "table not found". The catalog level is part of that plan rather than a separate statement, which means one table grant does widen namespace-name visibility across the catalog; separate catalogs are the answer when namespace names are themselves sensitive. WITH GRANT OPTION becomes usable with [access_control] grant_authority = "ranger-delegate", which hands the GRANT/REVOKE decision to Ranger's per-resource delegateAdmin so a table owner needs no engine-wide admin role. Getting there needed one more thing than removing the role check: Ranger's delegateAdmin does not cascade upward (measured on 2.8), so SQE skips a traversal level the grantee already holds, which is the only call a delegated grantor is not authorized to make. The default stays admin-role, and DENY is always admin-only. Apache Spark is now subject to the same object-level gates, and it took no engine code: Polaris already runs its own Ranger plugin keyed on the OIDC identity, so handing Spark's Iceberg REST catalog a per-user Keycloak token (with token-refresh-enabled=false, or Iceberg swaps it back for a service-account token) makes Polaris authorize the end user instead of root. make test-access-control-spark writes each grant through SQE's GRANT statement and asserts it through Spark, covering read, write, revoke and role-versus-user grants. Kyuubi checks its own privilege before Polaris is consulted and default-denies, so the shared service carries one deliberate blanket allow that makes it defer; a guard test proves that allow grants no data access of its own. Two divergences are documented rather than papered over: Spark's object tier verifies a JWT while its fine-grained tier trusts an asserted HADOOP_USER_NAME, so a mismatched pair gets one user's object rights with another's masks, and an unauthorized INSERT is refused at the snapshot commit, leaving staged files behind even though the table is untouched. Cross-engine parity is now asserted directly rather than per-engine: 21 statements run through both engines with the output compared cell by cell, and once a SET TAG association is projected into Ranger's tag store, tag-based masking renders byte-identically in both. Four divergences are written down (a named mask type is not byte-portable, an unprojected tag is invisible to Spark, Kyuubi can refuse before Polaris is consulted, and a row filter that reads a tag-masked column matches nothing under Kyuubi, which puts its masking projection below the row filter where SQE evaluates the filter against stored values). A fifth was closed rather than documented: policy.mask-precedence now defaults to tag, matching the Ranger plugin order Kyuubi implements, so a column carrying both a resource mask and a tag mask renders the same value in both engines (resource restores the earlier most-specific-rule-wins behaviour). scripts/access-control-parity-demo.sh consolidates the whole story into 34 cross-engine comparisons over a two-table EU retail bank fixture (a 12-row customer register and a 24-row payment ledger), and each divergence is asserted with both engines' expected values rather than skipped. Two personas beyond the analyst/engineer pair carry the shapes those two cannot express: a fraud desk that sees every jurisdiction with no customer identity, and an auditor that reads the register unmasked but the ledger only inside a retention window. The row worth acting on first is one where the engines AGREE and both are wrong: sqe.column-tags is keyed by column NAME, so RENAME COLUMN silently unmasks a governed column in both engines. SHOW MASKING POLICIES lists what is configured (admin-only, and disabled rather than empty without a Ranger backend, since an empty listing reads as "nothing is masked"), and ALTER TABLE ... MODIFY COLUMN ... SET MASKING POLICY attaches a named tag policy by resolving it to its tag; attaching a RESOURCE policy is refused, because Ranger evaluates a policy's database/table/column lists as a cross product and appending a column would widen the mask to tables nobody named. Two findings first filed as access-control DDL defects turned out to be one defect in the small-file scan path, which resolved projections against each data file's parquet column names instead of Iceberg field ids: after ADD COLUMN or RENAME COLUMN a projected column was silently dropped, and when no projected name matched the file at all the reader returned a DIFFERENT column's values under the projected name. The control that settled it was running the same DDL with no policy at all.
An MCP server that never holds a token. SQE speaks Model Context Protocol over Streamable HTTP (revision 2025-11-25) from the same coordinator binary, off by default behind [mcp]. It is an OAuth resource server, not an authorization server and not a credential proxy: the caller's bearer runs through SQE's existing auth chain, becomes the same per-user Session Flight SQL and Trino HTTP use, and is forwarded to the source catalog and storage, so user bearer -> MCP -> Session -> catalog/storage holds with no MCP service account anywhere. RFC 9728 protected-resource metadata, WWW-Authenticate challenges carrying the missing scope, and RFC 8707 audience-bound tokens from any OIDC provider. Nine read tools unlock on one scope; execute_sql_readonly accepts exactly one statement and is gated by SQE's SQL parser and statement classifier, not a keyword regex. execute_sql_change is absent from tools/list unless allow_writes = true, and then needs three independent authorizations (OAuth write scope, an SQE role in write_roles, and source authorization for that user and statement) because scopes bound the MCP surface and grant no data access of their own. The runnable stack in quickstart/mcp/ proves it with 14 checks against live Keycloak, Polaris and RustFS, including the one a service-account server structurally cannot pass: a reader holding the OAuth write scope is still denied by the source. Codex and Claude Code templates included. For catalogs reached with engine credentials instead of user tokens (Glue, S3 Tables, an attached warehouse), [mcp.auth] mode adds static principals with env or base64 bearer tokens and an anonymous mode that is refused whenever an IdP is configured, under production_mode, or with writes on. Full page: Model Context Protocol.
Lineage shipped. Coordinator emits OpenLineage 2-0-2 events with column-level lineage on writes. File and HTTP sinks. Disk-spool fallback for collector outages. Off by default. docs/site/book/src/operations/openlineage.md.
| SQE | Trino | DuckDB | |
|---|---|---|---|
| Embedded mode (one binary, no cluster) | yes | no | yes |
| Distributed mode (coordinator + workers) | yes | yes | no |
| Iceberg V2 + V3 read + write | native | V2 + partial V3 | extension, read-only |
| Per-query OIDC bearer passthrough | yes | service account only | n/a (single-tenant) |
| Ranger row filters + column masks at LogicalPlan | yes (incl. tag-based) | no | no |
| GRANT/REVOKE to Polaris or Apache Ranger | yes | no | no |
| Multi-catalog in one engine | 7 backends | one at a time | per-extension |
| Wire protocols | Arrow Flight SQL + Trino HTTP + MCP (Streamable HTTP) | Trino HTTP | extension |
| Runtime | Rust binary, no JVM | JVM | C++ binary |
| Cold start | sub-second | tens of seconds | sub-second |
| OpenLineage emitter | native, column-level | plugin | no |
Two longer comparison docs trace the lineage of these positions:
- vs Trino:
docs/site/compare/trino-compatibility.md. SQL function and feature parity by category. ~96% coverage. - vs DuckDB:
docs/site/compare/duckdb-comparision.md. What SQE has that DuckDB does not, and vice versa, with the V8 to V12 work that closed the embedded-mode gap.
Measured on 2026-08-15. Both engines against the same Iceberg catalog. Compare reports in benchmarks/results/compare-*-sf1-2026-08-15*.json; SQE-only Flight runs in *-sf1-flight-2026-08-15*.json. Totals below are the compare sqe_total_ms / trino_total_ms.
| Suite | SQE | Trino | Speedup | Pass |
|---|---|---|---|---|
| TPC-H (22) | 27.5s | 55.9s | 2.0x | 22/22 |
| SSB (13) | 11.1s | 12.1s | 1.1x | 13/13 |
| TPC-DS (99) | 74.9s | 140.3s | 1.9x | 99/99 |
| TPC-C (8 read) | 3.31s | 6.53s | 2.0x | 8/8 |
| TPC-E (11) | 5.57s | 13.7s | 2.5x | 11/11 |
| TPC-BB (10) | 56.3s | 290.3s | 5.2x | 10/10 |
| ClickBench (43) | 2.26s | 4.58s | 2.0x | 43/43 |
SQE wins all seven suites on that rig. TPC-DS compare is 99/99 match; the earlier ROLLUP / GROUPING() gaps are closed. TPC-BB is 10/10, but the standalone Flight run the same day is 63.1s with q01 at 35.5s; do not quote the suite total without naming q01 (issue #417). SSB is slightly faster than Trino here (11.1s vs 12.1s), not the suite we trail. The DynamicPredicate bridge that first collapsed TPC-DS (q82 1787ms to 113ms, q80 1398ms to 103ms, q13 1317ms to 220ms) is still the two-tier pushdown in the scan path: iceberg-rust samples once per file scan task for row-group / page-index pruning, and a per-batch wrapper catches filters that resolve after the task opened. The dynamic-filter type-coercion fix that flipped q72 from 10.7s to 0.77s is in docs/site/blog/2026-05-16-q72-the-nemesis.md. The earlier "Our Nemesis" investigation is preserved as docs/site/blog/2026-04-16-our-nemesis-q72.md.
June 2026, after the scan-parallelism work. Both engines run containerized in the same Docker network, read the same Iceberg tables from the same S3 store, and get the same envelope: 8 CPUs, bounded heaps, 5GB per query. Totals per suite; full per-query compare reports live in benchmarks/results/.
| Suite | SQE single-node | SQE distributed (2 workers) | Trino 481 | Verdict |
|---|---|---|---|---|
| TPC-H (22) | 130.5s | 95.5s | 106.4s - 138.6s | SQE distributed wins |
| SSB (13) | 42.0s | 53.6s | 28.0s - 41.1s | Trino, gap closing |
| TPC-DS (99) | 543.9s | 338.3s | 328.4s - 468.0s | even |
Trino shows a range because every compare run re-measures it; the high end is the run where two idle SQE worker containers shared the VM. Three changes carried SF10 from "3 to 5x slower than Trino" in the morning to this table in the evening:
- Parallel parquet decode. The iceberg-rust reader overlapped I/O but decoded every file on the one thread polling the merged stream. Scan-bound queries pinned one core while ten sat idle. Now every 128MB byte-range subtask decodes on its own runtime task. TPC-H q06 went from 6.4s to 1.65s distributed.
- A fair benchmark rig. The old rig ran SQE on the host, reading the dockerized S3 store through the port-forward at ~160 MB/s aggregate, while Trino read container-to-container at ~320 MB/s. Half the reported gap was the pipe, not the engine. Measure your rig before you profile your engine.
- A greedy memory pool. FairSpillPool statically split the pool across every registered spillable consumer; wide TPC-DS plans register ~90 of them, capping each at ~95MB of an 8GB pool and failing queries Trino finished in 5GB. The pool is now greedy with tracked consumers;
coordinator.memory_pool = "fair"restores the old behavior.
Distribution is a per-shape decision, not a default: big fact-to-fact joins (TPC-H, TPC-DS) gain 25 to 40 percent from two workers, while star schemas (SSB) pay shuffle costs that single-node avoids. Known open items at SF10: four TPC-DS inventory queries (q23, q37, q72, q82) fail distributed when the worker's scan buffer outruns Flight shipment and exhausts the 4GB worker pool, and SSB still trails Trino on raw star-join throughput.
A later SF10 pass found a separate blow-up: TPC-H q12, q17, and q10 ran 160 to 300 seconds against Trino's 2 to 8. The cause was not partition layout or join distribution. On a partitioned hash join the build-side runtime filter is a CASE over per-partition key sets (q12: eleven branches of ~28K keys, ~300K expression nodes), and the probe scan re-snapshotted it once per batch. Each snapshot rebuilt the whole tree (~10ms), so a 14,600-batch scan spent ~150s reconstructing a filter it then evaluated in under a second. We now cache the first sealed snapshot per scan: q12 161s to 2.7s, q17 176s to 7.1s, q10 from a 300s failure to 3.3s, result rows unchanged, no threshold touched. SSB improved too (q4.1 11.6s to 6.8s), which retired the IN-list-threshold tradeoff we thought we had. The walk-through is in The Filter That Rebuilt Itself; the EXPLAIN comparison is in docs/evidence/perf/sf10-slow-queries.md.
The table above came off a shared VM. These numbers come off a dedicated 8-core box, both engines containerized against the same Iceberg store, query cache off, on DataFusion 54. One run per query, so the cache cannot inflate a sweep. Single-node correctness held: TPC-H 21/22, SSB 13/13, TPC-DS 95/99, the same generator-boundary vacuous rows as SF1, zero dialect diffs, no OOM, and q72 completes.
| Suite | SQE single-node | SQE distributed (2w) | Trino 465 | Verdict |
|---|---|---|---|---|
| TPC-H SF10 | 89.0s | 85.8s | 105.7s | SQE 1.2x |
| SSB SF10 | 31.7s | 29.3s | 14.5s | Trino 2.2x |
| TPC-DS SF10 | 234.0s | 276.0s | 447.8s | SQE 1.9x |
The June-16 read on this rig called SF10 an "honest crossover" where SQE lost (TPC-H 126.4s/0.86x, SSB 31.8s/0.53x, TPC-DS 374s/1.22x on DataFusion 53). That reversed. The parallel Tier-2 scan filter and the move to DataFusion 54 flipped TPC-H to a 1.2x win and lifted TPC-DS to 1.9x. SSB is the one suite SQE still trails: lineorder's uniform foreign-key distribution defeats row-group pruning, so the runtime filter only helps at row level and Trino's vectorized decoder wins. These are single-run numbers on one box; read the ratios as directional, not certified.
Distribution earns little on a single host. Two workers sit co-tenant with the coordinator and Trino on the same eight cores: TPC-H gains four percent, SSB a little, and TPC-DS regresses to 276s while three inventory queries (q23, q37, q72, q82) exhaust the 4GB worker pool and drop the pass count from 95/99 to 92/99. Distribution is a per-shape decision, and its real payoff needs separate worker hosts, which this rig does not have. A true multi-node verdict is still missing, and it is the SF100 question below.
Loading SF10 surfaced a separate write-path gap worth naming: a partitioned CREATE TABLE AS SELECT with a sort-on-write clustering hint fans the sort into one merge buffer per output partition, and that merge phase cannot spill. At SF10 the monthly-partitioned TPC-H lineitem (60M rows across ~84 partitions) exhausted the pool where the unpartitioned SSB lineorder of the same size sorted fine. The bench loader now skips the redundant sort on already-partitioned tables, since partition pruning already delivers the clustering; the engine-level fix (bounded or spillable partition writers) is still open.
SF100 is the next frontier, and it inverts the SF1 and SF10 playbook. Broadcasting the build side, building hash tables in memory, and emitting one scan stream all win at SF10. Each becomes the bottleneck at SF100. Getting there needs three things: memory-pool discipline under concurrency (cap concurrent sort consumers, bound per-consumer reservations, proactive spill), a proven multi-node distributed path on separate worker hosts rather than one shared box, and a data generator that streams row groups to disk instead of buffering a whole table in memory. The predicted failure modes, each grounded in something we actually observed at SF1 or SF10, are written up in docs/evidence/perf/sf100-scaling-risks.md.
A benchmark row that says "Match" can still validate nothing: if the generated data contains no rows a query can select, both engines agree on empty and the diff passes. We learned this the hard way. Since June 2026 the harness reports those cases as Vacuous, and the generators are validated against DuckDB's official dsdgen output as an engine-free oracle (scripts/validate-generator-tpcds.py). That oracle caught a TPC-C generator bug that zeroed every warehouse join at fractional scales and a set of TPC-DS vocabulary gaps that had silently blanked 16 query results. Details in the validation blog post.
The same validation pass found one query where the engines disagree and SQE is right: TPC-DS q75 returns 57 rows on SQE and 55 on Trino, because Trino's DECIMAL(17,2) division rounds two sales ratios of 0.8983 and 0.8984 up to 0.90 and drops them from the < 0.9 filter. DuckDB returns SQE's exact 57 rows on the same parquet files.
Distributed mode gets the same scrutiny on a forced-distribution rig (single worker, distribution threshold zero, every fact scan shipped over Arrow Flight). That rig exposed that dynamic join filters never reached the workers; fixing the pushdown took TPC-DS SF1 under that worst case from 4.4x slower than Trino to 1.7x faster. SSB under the rig still trails: its star-join selectivity lives in the hash-set membership of the runtime filter, which a serialized predicate cannot carry. That loss is now instrumented and measured rather than assumed. On the forced-distribution rig at SF1 the shipped 65536-value cap keeps every SSB dim build in InList form and nothing is dropped; force the builds above the cap and the same 13 queries drop 62 hash_lookup conjuncts and the suite goes 7.4s to 10.0s. Shipping build-side key sets (bloom filters) to workers is the open follow-up, and it needs a key iterator DataFusion does not expose yet.
Run your own:
BENCH_SCALE=1 ./scripts/benchmark-test.sh --compare-trino tpch tpcds ssb clickbench
# differential compare against a live Trino on the same Iceberg catalog
sqe-bench compare tpcds --scale 1 --trino-url http://localhost:38080
# validate generated TPC-DS data against DuckDB's official dsdgen
duckdb /tmp/dsdgen.db -c "INSTALL tpcds; LOAD tpcds; CALL dsdgen(sf=1)"
scripts/validate-generator-tpcds.py --ours data/tpcds/sf1 --dsdgen-db /tmp/dsdgen.dbReruns of four read-only suites (TPC-H, SSB, TPC-DS, ClickBench) do not need a fresh load every time. The unified harness scripts/benchmark.sh orchestrates the setup and runs the suites, with profiles in benchmarks/profiles/<name>.toml. For example: BENCH_PROFILE=local BENCH_SCALE=1 scripts/benchmark.sh tpch ssb tpcds clickbench. Credentials come from environment variables or AWS profiles, never stored in the TOML file. Alternatively, scripts/benchmark-publish-iceberg.sh publishes Iceberg tables once into a persistent Polaris; BENCH_DATA_SOURCE=attach ./scripts/benchmark-test.sh tpch then attaches that catalog read-only and skips generate and load entirely. TPC-C, TPC-E, bank, and TPC-BB still generate and load normally in attach mode: TPC-C/TPC-E are write suites, bank is published to golden but attach mode does not yet query it from there, and TPC-BB's own tables (it shares TPC-DS's namespace) are not published. Bank/TPC-BB via attach, and a shallow-clone path for TPC-C/TPC-E, are the deferred follow-up. Full usage in Benchmark Suite.
Client (JDBC / Flight SQL / Trino HTTP / MCP)
|
v
+-----------+ OIDC Provider
|Coordinator|<-- (Keycloak, Auth0,
| | Okta, or any IdP)
| DataFusion|
| + Policy |---> Apache Ranger
+-----+-----+
| Bearer token passthrough
+----+----+
v v v
+---++---++---+ Stateless scan workers
| W1|| W2|| W3| (distributed mode: workers scan
+---++---++---+ and stream Arrow back)
| | |
v v v
+-----------+
| Polaris |---> S3-compatible storage
|REST Catalog|
+-----------+
Distributed mode is a distributed scan: the coordinator hands each worker a secured ScanTask (file list, projection, predicate, limit) with the user's bearer token, workers scan and stream Arrow batches back, and every join, aggregate and sort runs on the coordinator. There is no stage-based shuffle on any query path; the DoExchange intake exists only behind the worker-shuffle cargo feature. Size the coordinator for the post-scan work. Worker heartbeats carry sqe.proto.version and the DataFusion version, and a mismatched worker is refused at registration (a legacy versionless heartbeat is refused only under production_mode).
Detailed Mermaid diagrams (query pipeline, crate dependencies, caching layers, distributed scan, write path) in docs/site/book/src/architecture/overview.md.
A read-only ops dashboard ships in the binary, on the coordinator's health port (metrics_port + 1). Queries with per-fragment timing, cluster nodes, and live engine metrics (stat cards, a query-activity histogram, memory and concurrency gauges), all from the coordinator's in-memory state. No login (network-gated), no build step, no external assets. Toggle with [metrics] web_ui. Full reference: docs/site/book/src/operations/web-ui.md.
Five-minute walkthrough covering all seven catalog backends with sample TOML and verification queries: QUICKSTART.md.
cargo install --path crates/sqe-cli
sqe-cli --embedded # persistent warehouse at ~/.sqe/warehouse/
sqe> SELECT * FROM '/data/sales.parquet' LIMIT 5;
sqe> SELECT * FROM read_csv('s3://bucket/orders.tsv.gz');
sqe> SELECT * FROM 'hf://datasets/squad/plain_text/train-00000-of-00001.parquet' LIMIT 5;
sqe> SELECT * FROM read_delta('/data/delta/sales', version => '5');
sqe> CREATE SECRET partner (TYPE bearer, TOKEN 'eyJ...');
sqe> ATTACH 'http://catalog.example.com/api/catalog' AS partner_cat
(TYPE iceberg_rest, WAREHOUSE 'analytics', SECRET partner);
sqe> SELECT * FROM partner_cat.sales.orders LIMIT 10;Full embedded reference: docs/site/book/src/getting-started/cli.md. Runtime ATTACH / SECRET reference: docs/site/book/src/getting-started/catalogs.md.
docker compose -f docker-compose.test.yml up -d
./scripts/bootstrap-test.sh
cargo run --release --bin sqe-server -- --config tests/sqe-test.toml
# Connect with the CLI
cargo run --bin sqe-cli -- --host localhost --port 50051 --username root --protocol flight
sqe> SHOW CATALOGS;
sqe> SELECT * FROM test_warehouse.default.my_table LIMIT 10;Same binary against external infrastructure (Glue, S3 Tables, HMS, JDBC, Hadoop): see QUICKSTART.md and docs/site/book/src/getting-started/catalogs.md.
Docker, Kubernetes, TLS, and auth provider setup: docs/site/book/src/deployment/configuration.md.
Production operator guide (memory, observability, scaling): operations/production-guide.md.
The reference docs:
| Doc | What |
|---|---|
| Architecture | Mermaid diagrams across the engine |
| Deployment | Docker Compose, K8s, TLS, auth providers, monitoring |
| Model Context Protocol | The MCP endpoint: OAuth resource-server model, scopes, tools, the three write gates |
| Operational Runbook | On-call triage: crashloops, catalog/OIDC outages, OOM, registry flap |
| Access control tutorial | Both gates, worked end to end: the Polaris catalog gate (GRANT/REVOKE/DENY, views) and SQE's row filters, column masks and tags |
| Access control: support matrix | What is supported, what is proven, and by which test |
| Iceberg Matrix | Per-cell SQE coverage on the public scoreboard |
| Iceberg Matrix Comparison | V2/V3 side-by-side against 20 engines |
| Trino Compatibility | SQL function and feature matrix vs Trino |
| DuckDB Comparison | Symmetry between SQE and DuckDB on the embedded side |
| Embedded CLI Reference | All flags, dot-commands, TVFs, catalog backends, storage backends |
| SQL Feature Comparison | SQE vs Trino vs Spark SQL vs DuckDB across windows, aggregates, DML, Iceberg, file-format TVFs |
| SQL Reference (book) | Every function, statement, operator, TVF, CALL procedure, GRANT extension, with origin and Trino / Snowflake / Spark / DuckDB alias columns |
| Catalog Backends | Per-backend TOML, credentials, verification queries |
| Storage Backends | S3, R2, MinIO/Ceph, Azure ADLS Gen2, Google Cloud Storage, HTTPS, hf:// |
| Operations: OpenLineage | Lineage emit, sinks, troubleshooting |
| Benchmark history | Per-suite, per-scale, per-query plots over time |
| Roadmap | Full feature checklist |
| Security Audit | 43 findings, all resolved |
| Architecture, security and scale reviews 2026-09 | 109 findings, GitLab issues #543-#651 in eight work packages (#652-#659); 74 fixed |
SQE's design and development journey is documented in "Sovereign by Design: Building a Production Query Engine on DataFusion".
Twenty chapters across five parts. Roughly 370 pages. The story of choosing DataFusion, surviving the Iceberg fork rebase, lifting the matrix from 31% to 88%, building the embedded mode that turned out to be a DuckDB-shaped surprise, and shipping column-level lineage. Source in docs/site/ebook/. Build with cd docs/site/ebook && make.
A few chapters worth reading first:
- chapter 04 ("You Are the Query") on per-query identity
- chapter 09 ("What You Cannot See") on observability
- chapters 16b and 16c on the Iceberg matrix journey from 99/189 to 164/189
- chapter 16d ("The DuckDB Drift") on building embedded mode in two days
- chapter 16e ("The Lineage Trail") on shipping OpenLineage
- chapter 17 ("What We Would Do Differently") on the retro
Engineering posts that double as design rationale:
| Post | Topic |
|---|---|
| Why We Replaced Trino with Rust | The decision to build SQE |
| Five Layers of Caching and an 8.8x Speedup | Caching strategy across the stack |
| Security Hardening: 43 Findings | Production audit |
| DataFusion 53 and the Iceberg Fork | DF 53 upgrade and vendoring decision |
| Our Nemesis: TPC-DS Q72 | The one query we cannot beat |
| Why a Public Iceberg Matrix Beats Vendor Spec Sheets | A scoreboard for the lakehouse |
| SQE Talks to Five Catalogs Now | The live verification phase plus AWS SigV4 |
| How We Accidentally Created a DuckDB | V8 to V12: file-format TVFs, hf://, Delta, smarter read_csv |
| Shipping OpenLineage | Column-level lineage from idea to merged MR |
| The Benchmark That Lied | Vacuous results, the DuckDB oracle, and the day Trino was wrong |
| The Filter That Rebuilt Itself 14,600 Times | A runtime filter re-snapshotted per batch made q12 161s instead of 2.7s |
| One dlt Pipeline, Three Roads to Apache Polaris | Direct Iceberg REST, Trino HTTP, and Arrow Flight SQL ingestion into one governed catalog |
| The MCP Server That Never Holds a Token | MCP as an OAuth resource server: no service account, a parser-gated read-only tool, three write gates |
Full archive in docs/site/blog/.
| Crate | Purpose |
|---|---|
sqe-core |
Shared types, config (TOML), errors |
sqe-sql |
SQL parser, statement classifier, GRANT/REVOKE |
sqe-auth |
Pluggable auth chain (10 providers), token cache |
sqe-catalog |
Iceberg REST client, caching, scan execution |
sqe-policy |
Policy enforcement (passthrough, in-memory, Apache Ranger) |
sqe-planner |
Plan splitting, star-schema reorder, join strategy |
sqe-coordinator |
Flight SQL server, query handler, Trino HTTP |
sqe-worker |
Stateless DataFusion executor |
sqe-cli |
Interactive SQL client (cluster + embedded modes) |
sqe-metrics |
Prometheus, OpenTelemetry, audit logger |
sqe-lineage |
OpenLineage 2-0-2 emitter; column-level lineage |
sqe-trino-compat |
Trino wire protocol |
sqe-trino-functions |
Trino-compatible scalar UDFs for DataFusion |
sqe-mcp |
Model Context Protocol endpoint (OAuth resource server, tools) |
sqe-quack-wire |
Pure-Rust port of DuckDB's BinarySerializer (Quack RPC) |
sqe-quack-server |
Quack RPC server; accepts DuckDB clients over HTTP |
sqe-quack-client |
Client side of the DuckDB Quack RPC |
sqe-bench |
Benchmark suite (7 suites, 222 queries) |
| Component | Technology |
|---|---|
| Language | Rust |
| Query Engine | Apache DataFusion 54 |
| Table Format | Apache Iceberg V2 / V3 |
| Catalogs | Polaris, Nessie, Unity Catalog OSS, AWS Glue, AWS S3 Tables, Hive Metastore, JDBC, Hadoop |
| Wire Protocols | Arrow Flight SQL + Trino HTTP + MCP Streamable HTTP |
| Storage | S3, Ceph, R2, ADLS Gen2, GCS, local filesystem, HuggingFace hf:// |
| Observability | OpenTelemetry with trace-only or all-signal OTLP, Prometheus query/session/Iceberg scan metrics, OpenLineage 2-0-2, read-only web UI (queries/tasks/workers/metrics dashboard with 12h history) on the health port |
| License | Apache 2.0 |
Issues, pull requests, and how to run tests: CONTRIBUTING.md.
Apache License 2.0. See LICENSE.