ProjectDataHopper · mattcasters · Aug 28, 2026 · Aug 28, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,15 @@ All notable changes to Data Hopper EDW (formerly hop-datavault) are documented i
 
 ## Unreleased
 
+### Partition large BV SCD2 loads (issue #141)
+
+- SCD2 table option **Hash-key partitions** (None / 4 / 8 / 16) splits a Full rebuild so each satellite `ORDER BY` covers a first-byte slice of the parent hash key
+- A generated workflow truncates the BV target once, then a driver pipeline runs the parameterized SCD2 pipeline for partition numbers `0 .. N-1` (`'${PARTITION_COUNT}'`, `'${PARTITION_NUMBER}'`)
+- Truncate is always a SQL action; Table Output and Native bulk then append each partition
+- Staging file writes `{pipeline}-${PARTITION_NUMBER}-${copy}.csv` per partition, then bulk-loads those files from the wrapper workflow
+- Incremental build mode cannot be combined with hash-key partitions
+- Business Vault Update runs partitioned SCD2 wrapper workflows before the free-pipeline orchestrator
+
 ### Read-only existing Data Vault models
 
 - Data Vault configuration checkbox **Read-only existing vault** documents an already-built raw vault so Business Vault and dimensional models can sit on top

diff --git a/docs/ai-file-schemas/models/hbv.md b/docs/ai-file-schemas/models/hbv.md
@@ -41,7 +41,8 @@ Inspect real retail samples for exact nested tags (`satellite_config`, `field_ma
 
 - Parent DV hub (via references)  
 - One or more **satellite configs** (`satelliteName`, source indicator, field mappings `sourceFieldName` → `targetFieldName`)  
-- Timeline fields from configuration (`validFromField`, `validToField`, open sentinels)  
+- Timeline fields from configuration (`validFromField`, `validToField`, open sentinels)
+- Optional `hashKeyPartitionCount` (`NONE`, `4`, `8`, `16`) for large full rebuilds — SQL truncate once, then Table Output, Native bulk, or Staging file (one CSV set per partition)  
 
 ## Anti-patterns
 

diff --git a/docs/ai-file-schemas/samples/hbv-excerpt.xml b/docs/ai-file-schemas/samples/hbv-excerpt.xml
@@ -33,6 +33,7 @@
 <tables>
 <table>
       <buildMode>FULL_REBUILD</buildMode>
+      <hashKeyPartitionCount>NONE</hashKeyPartitionCount>
       <functionalTimestampField>x_load_ts</functionalTimestampField>
       <validFromField/>
       <validToField/>

diff --git a/docs/business-vault-scd2.adoc b/docs/business-vault-scd2.adoc
@@ -53,6 +53,25 @@ For each SCD2 table the plugin builds a Hop pipeline that:
 
 Use **Debug** or **Show build pipeline** on an SCD2 table in the `.hbv` editor to inspect the generated pipeline before running a workflow.
 
+=== Hash-key partitions (large full rebuilds)
+
+When a satellite is too large to `ORDER BY` hash key and load date in one pass, set **Hash-key partitions** to 4, 8, or 16 on the SCD2 table. Full rebuild then:
+
+1. Truncates the BV target **once** with a workflow SQL action (`TRUNCATE TABLE` via the Hop database dialect).
+2. Generates partition numbers `0 .. N-1` and runs the SCD2 pipeline once per part, passing `'${PARTITION_COUNT}'` and `'${PARTITION_NUMBER}'`.
+3. Filters each satellite `TableInput` with a first-byte modulus of the parent hash key (BINARY: first octet; HEX: first two hex characters; STRING: first dash-separated token). Every version of a hub/link key stays in one partition.
+4. Writes with `TableOutput` **truncate off**, so later partitions append.
+
+**Show build pipeline** opens the parameterized SCD2 pipeline, the partition driver, and the wrapper workflow. Running the inner pipeline alone with defaults loads partition 0 of N.
+
+Constraints:
+
+* Full rebuild only (Incremental already filters by watermark).
+* Target load mode is honored: **Table Output** and **Native bulk** append after the SQL truncate; **Staging file** writes one CSV set per partition (`…-${PARTITION_NUMBER}-${Internal.Transform.CopyNr}.csv`) then bulk-loads those files in the wrapper workflow.
+* Sequential partition execution (lower peak memory, not parallel wall-clock).
+
+Integration test: `integration-tests/tests/multi-satellite-bv/update-customer-360-partitioned.hwf` uses `customer-360-partitioned.hbv` (4 parts) and the same golden current-state dataset as the unpartitioned full rebuild.
+
 === Incremental build mode
 
 Set **Build mode** to *Incremental* on the SCD2 table when you want to append new functional versions instead of truncating and reloading the BV table on every run.
@@ -79,7 +98,7 @@ The watermark is always read from the **Business Vault target table** (`MAX(wate
 
 On the first incremental run against an empty BV table the sentinel (`1900-01-01 00:00:00`) applies, so behaviour matches a full rebuild. Later runs process only satellite deltas and close prior open rows in place.
 
-Integration test: `integration-tests/tests/multi-satellite-bv/update-customer-360-incremental.hwf` loads four DV satellite waves, runs **Business Vault Update** once on `customer-360-incremental.hbv`, then validates current open rows against the same golden dataset as the full-rebuild fixture. Multi-run close-and-append behaviour is covered by unit tests in `BvScd2PipelineSupportTest`.
+Integration test: `integration-tests/tests/multi-satellite-bv/update-customer-360-incremental.hwf` loads four DV satellite waves, runs **Business Vault Update** once on `customer-360-incremental.hbv`, then validates current open rows against the same golden dataset as the full-rebuild fixture. Multi-run close-and-append behaviour is covered by unit tests in `BvScd2PipelineSupportTest`. Hash-key partitioned full rebuild: `update-customer-360-partitioned.hwf` / `customer-360-partitioned.hbv` (4 parts, same golden).
 
 == Single-satellite SCD2
 
@@ -150,6 +169,9 @@ Open intervals use sentinels from configuration (default `1900-01-01 00:00:00` a
 |Build mode
 |`FULL_REBUILD` (default) truncates and reloads the BV table; `INCREMENTAL` filters satellite history and appends/closes versions.
 
+|Hash-key partitions
+|`None` (default), or `4` / `8` / `16` parts for a large full rebuild. See <<Hash-key partitions (large full rebuilds)>>.
+
 |Incremental watermark field
 |Optional BV column used for `MAX(...)` watermark reads and incremental write filtering. Defaults to the resolved functional timestamp field.
 

diff --git a/docs/business-vault-update-action.adoc b/docs/business-vault-update-action.adoc
@@ -14,7 +14,8 @@ At runtime the action:
 * Optionally validates both models before any work is done
 * Optionally generates CREATE TABLE DDL for Business Vault tables on the BV target database
 * Generates **build pipelines** for each selected SCD2 and PIT table (and other supported types when implemented)
-* Stages pipelines and runs them through a parallel orchestrator
+* For SCD2 tables with **hash-key partitions**, generates a wrapper workflow (truncate once, then sequential partition loads; Staging file mode then bulk-loads each partition CSV) and runs it before the free-pipeline orchestrator
+* Stages remaining pipelines and runs them through a parallel orchestrator
 * Optionally publishes BV target table layouts to the data catalog
 * Aggregates pipeline results into the workflow action result
 
@@ -102,5 +103,6 @@ SCD2 and PIT read independent data paths (PIT does not consume SCD2 output). Ord
 Sample workflows:
 
 * `integration-tests/tests/multi-satellite-bv/update-customer-360.hwf` — multi-satellite SCD2 (full rebuild)
+* `integration-tests/tests/multi-satellite-bv/update-customer-360-partitioned.hwf` — same golden as full rebuild, SCD2 hash-key partitions = 4
 * `integration-tests/tests/multi-satellite-bv/update-customer-360-incremental.hwf` — multi-satellite SCD2 incremental build mode
 * `integration-tests/tests/basic/update-vault1.hwf` — SCD2 + PIT on the introductory vault1 model
diff --git a/docs/feature-overview.adoc b/docs/feature-overview.adoc
@@ -324,7 +324,7 @@ image::images/validate-resource-definitions-action-dialog.png[Validate resource
 
 === Business Vault (`.hbv`)
 
-Linked to a `.hdv` model. Defines **SCD2** consumption tables (single- or multi-satellite merge), **PIT** helpers, and **SQL views/tables** (dbt-style `ref` / `source`, optional Jinja macros). **Import dbt models** on the canvas (and the **Import dbt project** workflow action) loads a dbt-core project as SQL business tables. **Business Vault Update** validates, optionally publishes target layouts to the catalog, generates SCD2 build pipelines, and orchestrates parallel execution.
+Linked to a `.hdv` model. Defines **SCD2** consumption tables (single- or multi-satellite merge), **PIT** helpers, and **SQL views/tables** (dbt-style `ref` / `source`, optional Jinja macros). **Import dbt models** on the canvas (and the **Import dbt project** workflow action) loads a dbt-core project as SQL business tables. **Business Vault Update** validates, optionally publishes target layouts to the catalog, generates SCD2 build pipelines (hash-key partitioned full rebuilds run as truncate-then-load wrapper workflows), and orchestrates parallel execution.
 
 image::images/business-vault-dbt-import-models-dialog.png[Import dbt models dialog — project folder, model list, destination, and macro library,align="center"]
 

diff --git a/docs/getting-started-integration-tests.adoc b/docs/getting-started-integration-tests.adoc
@@ -104,6 +104,8 @@ Run `integration-tests/tests/multi-satellite-bv/update-customer-360.hwf` for end
 
 For incremental SCD2, open `customer-360-incremental.hbv` (same mappings, `buildMode=INCREMENTAL`) and run `update-customer-360-incremental.hwf`. That workflow runs the same DV load waves, then **Business Vault Update** with the incremental pipeline, and validates against the same golden current-state dataset.
 
+For a large-table Full rebuild split, open `customer-360-partitioned.hbv` (`hashKeyPartitionCount=4`) and run `update-customer-360-partitioned.hwf`. Same golden current-state dataset; the target is truncated once, then four hash-key slices are loaded.
+
 == Chapter 7 — External read-only raw vault tables
 
 When dbt or another tool loads raw vault tables, Hop can still model them for Business Vault.

diff --git a/docs/help/bv-scd2-table-dialog.adoc b/docs/help/bv-scd2-table-dialog.adoc
@@ -46,6 +46,9 @@ Raw vault satellites remain insert-only (technical load history). SCD2 rewrites
 |**Build mode**
 |`FULL_REBUILD` (truncate and reload) or `INCREMENTAL` (watermark, append, close open rows).
 
+|**Hash-key partitions**
+|`None` (default) or 4 / 8 / 16. Full rebuild only: truncate once, then load hash-key slices so each satellite `ORDER BY` is smaller. Works with Table Output, Native bulk, and Staging file (one CSV set per partition, then bulk-load).
+
 |**Incremental watermark field**
 |Optional BV column for `MAX(...)` watermark reads; defaults to the functional timestamp.
 
@@ -69,8 +72,9 @@ The functional timestamp drives interval boundaries — not the technical load d
 
 - **Full rebuild** — truncate and reload the BV table from satellite history.
 - **Incremental** — read satellite deltas above a watermark, close the prior open version, append new versions. Prefer after an initial full load when history is large.
+- **Hash-key partitions** — optional 4 / 8 / 16 split on Full rebuild. A wrapper workflow truncates the target, then runs the SCD2 pipeline per partition with `'${PARTITION_COUNT}'` and `'${PARTITION_NUMBER}'`. Honors Table Output, Native bulk, and Staging file. Disabled for Incremental.
 
-Use **Debug** or **Show build pipeline** on the canvas to inspect the generated Hop pipeline before workflow runs.
+Use **Debug** or **Show build pipeline** on the canvas to inspect the generated Hop pipeline (and the partition workflow when enabled) before workflow runs.
 
 == Field mappings
 

diff --git a/integration-tests/tests/multi-satellite-bv/customer-360-partitioned.hbv b/integration-tests/tests/multi-satellite-bv/customer-360-partitioned.hbv
@@ -0,0 +1,195 @@
+<!--
+Copyright 2026 i-Bridge bv
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<business-vault-model>
+  <name_sync_with_filename>Y</name_sync_with_filename>
+  <description>Customer 360 business vault (4 hash-key partitions): four satellites merged into one functional SCD2 table</description>
+  <dataVaultModelPath>${PROJECT_HOME}/tests/multi-satellite-bv/customer-360.hdv</dataVaultModelPath>
+  <configurationName>business-vault-customer-360</configurationName>
+  <configuration>
+    <targetDatabase>Vault</targetDatabase>
+    <generatedPipelineFolder/>
+    <scd2PipelineNamePrefix>bv-scd2-</scd2PipelineNamePrefix>
+    <pitPipelineNamePrefix>bv-pit-</pitPipelineNamePrefix>
+    <businessTablePipelineNamePrefix>bv-biz-</businessTablePipelineNamePrefix>
+    <openStartSentinel>1900-01-01 00:00:00</openStartSentinel>
+    <openEndSentinel>9999-12-31 23:59:59</openEndSentinel>
+    <scd2DefaultsFolder/>
+    <functionalTimestampField/>
+    <loadDateFieldFallback>x_load_ts</loadDateFieldFallback>
+    <validFromField>x_from_ts</validFromField>
+    <validToField>x_to_ts</validToField>
+    <targetTableBatchSize>1000</targetTableBatchSize>
+    <targetTableParallelCopies>1</targetTableParallelCopies>
+  </configuration>
+  <tables>
+    <table>
+      <functionalTimestampField>x_load_ts</functionalTimestampField>
+      <validFromField/>
+      <validToField/>
+      <includeHashKey>Y</includeHashKey>
+      <hashKeyPartitionCount>4</hashKeyPartitionCount>
+      <field_mappings>
+        <field_mapping>
+          <satelliteName>sat_customer_demo</satelliteName>
+          <sourceFieldName>segment</sourceFieldName>
+          <targetFieldName>cust_segment</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_demo</satelliteName>
+          <sourceFieldName>loyalty_tier</sourceFieldName>
+          <targetFieldName>cust_loyalty_tier</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_demo</satelliteName>
+          <sourceFieldName>demo_score</sourceFieldName>
+          <targetFieldName>cust_demo_score</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_contact</satelliteName>
+          <sourceFieldName>email</sourceFieldName>
+          <targetFieldName>cust_email</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_contact</satelliteName>
+          <sourceFieldName>phone</sourceFieldName>
+          <targetFieldName>cust_phone</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_address</satelliteName>
+          <sourceFieldName>address_line1</sourceFieldName>
+          <targetFieldName>cust_address</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_address</satelliteName>
+          <sourceFieldName>city</sourceFieldName>
+          <targetFieldName>cust_city</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_address</satelliteName>
+          <sourceFieldName>postal_code</sourceFieldName>
+          <targetFieldName>cust_postal_code</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_prefs</satelliteName>
+          <sourceFieldName>newsletter_opt_in</sourceFieldName>
+          <targetFieldName>cust_newsletter</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_prefs</satelliteName>
+          <sourceFieldName>preferred_channel</sourceFieldName>
+          <targetFieldName>cust_channel</targetFieldName>
+        </field_mapping>
+        <field_mapping>
+          <satelliteName>sat_customer_prefs</satelliteName>
+          <sourceFieldName>language_code</sourceFieldName>
+          <targetFieldName>cust_language</targetFieldName>
+        </field_mapping>
+      </field_mappings>
+      <satellite_configs>
+        <satellite_config>
+          <satelliteName>sat_customer_demo</satelliteName>
+          <functionalTimestampField/>
+          <sourceIndicatorValue>DEMO</sourceIndicatorValue>
+        </satellite_config>
+        <satellite_config>
+          <satelliteName>sat_customer_contact</satelliteName>
+          <functionalTimestampField/>
+          <sourceIndicatorValue>CONTACT</sourceIndicatorValue>
+        </satellite_config>
+        <satellite_config>
+          <satelliteName>sat_customer_address</satelliteName>
+          <functionalTimestampField/>
+          <sourceIndicatorValue>ADDRESS</sourceIndicatorValue>
+        </satellite_config>
+        <satellite_config>
+          <satelliteName>sat_customer_prefs</satelliteName>
+          <functionalTimestampField/>
+          <sourceIndicatorValue>PREFS</sourceIndicatorValue>
+        </satellite_config>
+      </satellite_configs>
+      <tableName>customer_360_bv</tableName>
+      <description>Functional customer 360 view merged from four satellites</description>
+      <tableType>SCD2</tableType>
+      <derivatives>
+        <derivative>
+          <dvTableName>sat_customer_demo</dvTableName>
+          <dvTableType>SATELLITE</dvTableType>
+        </derivative>
+        <derivative>
+          <dvTableName>sat_customer_contact</dvTableName>
+          <dvTableType>SATELLITE</dvTableType>
+        </derivative>
+        <derivative>
+          <dvTableName>sat_customer_address</dvTableName>
+          <dvTableType>SATELLITE</dvTableType>
+        </derivative>
+        <derivative>
+          <dvTableName>sat_customer_prefs</dvTableName>
+          <dvTableType>SATELLITE</dvTableType>
+        </derivative>
+      </derivatives>
+      <xloc>496</xloc>
+      <yloc>208</yloc>
+      <name>customer_360_bv</name>
+    </table>
+  </tables>
+  <notes>
+    <note>
+      <text>This SCD2 exercise uses the input from 4 different satellites to build 
+one "Customer 360" Type II Slowly Changing Dimension.
+
+The way it does this is by reading from all 4 sources at the same time and 
+using a Sorted Schema Merge (new transform) feeding into the [Repeat Fields](https://hop.apache.org/manual/latest/pipeline/transforms/repeatfields.html#_current_when_indicated) transform 
+
+to build combined records of what is essentially a timeline per customer hash key.  
+What you get is a clear indication of which record version was valid at which point in time.
+
+Click on the [customer_360_bv](customer_360_bv) table and select "Show build pipeline" to see the full pipeline.</text>
+      <noteType>INFORMATION</noteType>
+      <xloc>144</xloc>
+      <yloc>384</yloc>
+      <width>624</width>
+      <height>224</height>
+    </note>
+  </notes>
+  <dv_references>
+    <dv_reference>
+      <dvTableName>sat_customer_demo</dvTableName>
+      <dvTableType>SATELLITE</dvTableType>
+      <xloc>240</xloc>
+      <yloc>112</yloc>
+    </dv_reference>
+    <dv_reference>
+      <dvTableName>sat_customer_contact</dvTableName>
+      <dvTableType>SATELLITE</dvTableType>
+      <xloc>144</xloc>
+      <yloc>208</yloc>
+    </dv_reference>
+    <dv_reference>
+      <dvTableName>sat_customer_address</dvTableName>
+      <dvTableType>SATELLITE</dvTableType>
+      <xloc>144</xloc>
+      <yloc>304</yloc>
+    </dv_reference>
+    <dv_reference>
+      <dvTableName>sat_customer_prefs</dvTableName>
+      <dvTableType>SATELLITE</dvTableType>
+      <xloc>448</xloc>
+      <yloc>80</yloc>
+    </dv_reference>
+  </dv_references>
+  <name>customer-360-partitioned</name>
+</business-vault-model>