feat: Add multi-cloud FOCUS test data generator for FinOps Hub

[FallenHoot]

Add multi-cloud FOCUS test data generator for FinOps Hub

Description

Adds Generate-MultiCloudTestData.ps1 — a PowerShell script that generates synthetic, multi-cloud, FOCUS-compliant cost data for testing and validating FinOps Hub deployments end-to-end.

Closes #2005

What's Included

• Generate-MultiCloudTestData.ps1 (~1,430 lines) — Self-contained script that generates FOCUS 1.0–1.3 synthetic cost data for Azure, AWS, GCP, and DataCenter providers

Why This Script Is Needed

Testing a FinOps Hub deployment today requires real Cost Management export data. This script fills that gap by generating realistic synthetic data that:

1. Covers all 4 supported cloud providers with provider-specific conventions (Azure resource IDs, AWS ARNs, GCP resource paths)
2. Populates every column referenced by FinOps Hub dashboard KQL queries
3. Simulates real-world patterns: commitment discounts (Reservations + Savings Plans), Azure Hybrid Benefit, spot/dynamic pricing, marketplace purchases, negotiated discounts, and tag coverage variation
4. Generates data with proper Cost Management manifest.json files for ingestion pipeline compatibility
5. Optionally uploads to Azure Storage and manages ADF triggers

Key Features

Feature	Details
FOCUS compliance	All mandatory + conditional FOCUS columns (v1.0–1.3)
Persistent identities	Resources, billing accounts, subscriptions consistent across days
Budget scaling	Costs scaled to target budget via Python/pandas
Memory-safe	Streams rows daily to CSV, avoids OOM on 500K+ row datasets
Output formats	Parquet (pyarrow), CSV, or both
Upload support	Uploads to msexports + ingestion containers with proper blob paths

Testing

Tested with:

• Default settings (500K rows, 6 months, all providers, $500K budget)
• Single provider mode (Azure-only, 200K rows)
• Full pipeline (generate → upload → ADF trigger → ADX ingestion → dashboard validation)
• FOCUS versions 1.0, 1.2, and 1.3

Prerequisites

• PowerShell 7+
• Python 3 with pandas and pyarrow (for Parquet conversion)
• Azure CLI (for upload functionality)

Checklist

• Script follows FOCUS specification conventions
• Microsoft copyright header included
• Comment-based help with SYNOPSIS, DESCRIPTION, PARAMETERS, EXAMPLES
• No hardcoded paths or environment-specific references
• Tested with 498K+ rows successfully ingested into FinOps Hub

[RolandKrummenacher]

Review: Generate-MultiCloudTestData.ps1

Great concept — this fills a real gap for FinOps Hub end-to-end testing. The FOCUS column coverage and multi-cloud provider modeling are thorough. However, there are several issues to address before merging:

Critical

• Get-Random overflow with 12-digit AWS account IDs (lines 335, 361) — will throw at runtime
• Python dependency should be eliminated — budget scaling and Parquet output can be done in pure PowerShell, removing ~80 lines of fragile cross-language code with path-injection risk and dead code

Required by repo conventions

• Missing changelog entry (v14 section in docs-mslearn/toolkit/changelog.md)
• Missing README.md in the test directory
• Missing #Requires statement and .LINK in help
• No -WhatIf/-Confirm support for destructive operations (file creation, uploads, trigger starts)

Recommended

• Add Pester tests for helper functions
• Prefer Azure AD auth over storage account keys
• Add -Seed parameter for reproducible test data
• FOCUS version parameter is metadata-only — either vary the schema or simplify

Minor

• Inconsistent cost rounding (10 vs 2 decimal places)
• ADF trigger names hardcoded — should be parameterized or documented

[RolandKrummenacher]

FOCUS Specification Compliance Analysis

I did a detailed comparison of the script's output against the official FOCUS specification at focus.finops.org for all four claimed versions (1.0, 1.1, 1.2, 1.3). Here's what I found:

---

Critical Issue: $FocusVersion parameter is cosmetic only

The script accepts $FocusVersion (ValidateSet "1.0", "1.1", "1.2", "1.3") but never uses it to vary the output schema. The same columns are emitted regardless of version. The value is only written to x_FocusVersion. This means the output cannot be properly compliant with any single FOCUS version — it's a superset/subset mix.

---

Per-Version Column Compliance (Cost and Usage Dataset)

FOCUS v1.0 (43 columns)

• ✅ All 43 v1.0 columns are present
• ❌ 8 extra columns that don't exist in v1.0: BillingAccountType (v1.2), SubAccountType (v1.2), InvoiceId (v1.2), CommitmentDiscountQuantity (v1.1), CommitmentDiscountUnit (v1.1), ServiceSubcategory (v1.1), HostProviderName (v1.3), ServiceProviderName (v1.3)

FOCUS v1.1 (50 columns)

• ❌ 4 columns missing: CapacityReservationId, CapacityReservationStatus, SkuMeter, SkuPriceDetails
• ❌ 5 extra columns not in v1.1: BillingAccountType, SubAccountType, InvoiceId, HostProviderName, ServiceProviderName

FOCUS v1.2 (57 columns)

• ❌ 8 columns missing: CapacityReservationId, CapacityReservationStatus, SkuMeter, SkuPriceDetails, PricingCurrency, PricingCurrencyContractedUnitPrice, PricingCurrencyEffectiveCost, PricingCurrencyListUnitPrice
• ❌ 2 extra columns not in v1.2: HostProviderName, ServiceProviderName

FOCUS v1.3 (64+ columns)

• ❌ 13 columns missing: AllocatedMethodDetails, AllocatedResourceId, AllocatedResourceName, AllocatedResourceType, CapacityReservationId, CapacityReservationStatus, ContractApplied, PricingCurrency, PricingCurrencyContractedUnitPrice, PricingCurrencyEffectiveCost, PricingCurrencyListUnitPrice, SkuMeter, SkuPriceDetails

Column Naming Issue

The script uses ServiceProviderName as the mandatory provider column for all versions, but the correct Column ID is ProviderName for v1.0–v1.2. ServiceProviderName only replaces ProviderName (deprecated) in v1.3. The script does output ProviderName too, but categorizes it under "FinOps Hub / Dashboard required columns" — it should be the primary mandatory column for v1.0–v1.2.

---

Missing v1.3 Structural Features

1. Contract Commitment Dataset (entirely absent)

FOCUS v1.3 introduced a second dataset with 13 mandatory columns (ContractId, ContractCommitmentId, ContractCommitmentCategory, ContractCommitmentCost, ContractCommitmentDescription, ContractCommitmentPeriodEnd/Start, ContractCommitmentQuantity, ContractCommitmentType, ContractCommitmentUnit, ContractPeriodEnd/Start, BillingCurrency). The script only generates Cost and Usage data — no Contract Commitment dataset is produced.

2. Data Generator-Calculated Split Cost Allocation (absent)

The 4 Allocated* columns (AllocatedMethodDetails, AllocatedResourceId, AllocatedResourceName, AllocatedResourceType) support shared cost splitting (e.g., K8s clusters, shared storage). Not implemented.

3. ContractApplied column (absent)

The JSON column that bridges Cost and Usage rows to the Contract Commitment dataset is not generated.

---

Summary Table

Version	Columns Present	Columns Missing	Extra Columns	Compliant?
v1.0	43/43	0	8	❌
v1.1	46/50	4	5	❌
v1.2	49/57	8	2	❌
v1.3	51/64+	13+	0	❌

Recommendation

Either:

1. Target a single version (e.g., v1.0 or v1.3) and get that version fully correct, or
2. Use $FocusVersion to dynamically select columns — only output columns valid for the chosen version and include all required columns for that version.

The closest match today is v1.3, but it's still missing conditional/recommended columns and the entire Contract Commitment dataset. For test data generation purposes, it may be acceptable to scope this to Cost and Usage only with a documented caveat, but the column set should still match the selected version.

[RolandKrummenacher]

Additional FOCUS Spec Compliance Findings

A few more items found during deeper analysis:

---

1. ServiceSubcategory — Invalid Values Against Spec's Closed Enumeration

The FOCUS spec (v1.1+) defines a closed list of allowed ServiceSubcategory values, each with a mandatory parent ServiceCategory. ~12 out of ~30 service entries use values that are not in the spec's allowed list:

Line	Service	Category	Subcategory in Script	Issue
169	Storage Accounts	Storage	General Purpose v2	Should be Object Storage or Block Storage
170	Azure Cosmos DB	Databases	NoSQL Databases	Should be NoSQL
171	Azure Data Explorer	Analytics	Data Analytics	Not in spec — closest: Log Analytics or Other (Analytics)
172	Azure App Service	Compute	App Services	Not in spec — closest: Containers or Other (Compute)
173	Azure Functions	Compute	Serverless Compute	Should be Functions
174	Azure Key Vault	Security	Key Management	Not in spec — closest: Other (Security)
175	Bandwidth	Networking	Data Transfer	Not in spec — closest: Content Delivery or Other (Networking)
176	Marketplace - 3rd Party	Compute	Marketplace	Not a valid subcategory
220	Amazon DynamoDB	Databases	NoSQL Databases	Should be NoSQL
247	Cloud Spanner	Databases	Distributed Databases	Not in spec — closest: Other (Databases)
248	Cloud Run	Compute	Serverless Containers	Should be Containers
267	Physical Servers	Compute	Bare Metal	Not in spec — closest: Other (Compute)

Values that are correct include: Virtual Machines, Containers, Relational Databases, Object Storage, Block Storage, Content Delivery, Network Infrastructure, Data Warehouses.

---

2. Cost Column Invariants — Math Broken by Anomaly Rows

The FOCUS spec requires: ListCost = ListUnitPrice × PricingQuantity (and similarly for ContractedCost) when unit price and quantity are non-null and ChargeClass ≠ "Correction".

Unit prices are derived on line 741-742:

$listUnitPrice = [math]::Round($listCost / $pricingQuantity, 10)
$contractedUnitPrice = [math]::Round($contractedCost / $pricingQuantity, 10)

But the "data quality anomaly" block on lines 751-756 mutates costs AFTER unit prices were already calculated:

if ($qualityRoll -eq 0) {
    $effectiveCost = [math]::Round($contractedCost * 1.1, 10)   # breaks EffectiveCost invariant
} elseif ($qualityRoll -eq 1) {
    $contractedCost = [math]::Round($listCost * 1.05, 10)       # breaks ContractedCost = ContractedUnitPrice × PricingQuantity
}

~2% of rows will have cost/unit-price mismatches that violate the spec's mathematical constraints. If these are intentionally anomalous test data, they should be documented as such (e.g., via x_SourceChanges), and ChargeClass should be set to "Correction" to exempt them from the spec's invariant rules.

---

3. InvoiceId — Assigned to All Charge Categories

The script generates an InvoiceId for every row (lines 760-764), including Credit and Adjustment charges. In practice:

• Some credits and adjustments are not tied to a specific invoice and should have InvoiceId = $null
• ChargeClass = "Correction" rows reference a previously invoiced billing period and might carry the original invoice's ID, not a new one

This is a lower-severity finding (more about realism than strict spec violation), but worth considering for test data that claims multi-version FOCUS compliance.

[RolandKrummenacher]

Missing FOCUS Columns

The script is still missing several columns defined in the FOCUS specification across versions. These should either be implemented or explicitly documented as out-of-scope:

v1.1+ (4 columns)

• CapacityReservationId — Identifier for capacity reservations
• CapacityReservationStatus — Whether capacity reservation was used/unused
• SkuMeter — Meter-level SKU details
• SkuPriceDetails — JSON column with pricing metadata

v1.2+ (4 columns)

• PricingCurrency — Currency used for pricing columns
• PricingCurrencyContractedUnitPrice — Contracted unit price in pricing currency
• PricingCurrencyEffectiveCost — Effective cost in pricing currency
• PricingCurrencyListUnitPrice — List unit price in pricing currency

v1.3+ (5 columns)

• AllocatedMethodDetails — Details about cost allocation method
• AllocatedResourceId — Resource ID for split/allocated costs
• AllocatedResourceName — Resource name for split/allocated costs
• AllocatedResourceType — Resource type for split/allocated costs
• ContractApplied — JSON column bridging Cost and Usage rows to Contract Commitment dataset

Total: 13 columns missing across versions. Without these, the output cannot be fully compliant with any FOCUS version from v1.1 onward. At minimum, please document which columns are intentionally excluded and why (e.g., Contract Commitment dataset is already noted as out of scope in the help text — the same treatment should apply to these).

[FallenHoot]

@RolandKrummenacher — Thank you for the thorough review! Really appreciate the detailed feedback.

Your comments were spot-on and gave us the opportunity to go back and revisit logic that was missing during a live demo. We've addressed all the review feedback in this latest push:

What changed

PR review items — all addressed:

• AllocatedResourceType — Added as the missing FOCUS v1.3 column
• ContractApplied — Now populated with JSON contract references for committed-discount rows (v1.3+)
• Split cost allocation — ~10% of AKS/EKS/GKE rows now populate Allocated* columns with namespace-level allocation simulation
• ADF trigger names — Extracted to a reusable `` variable (was hardcoded in 2 places)
• Column emission documented — FOCUS Column Coverage summary now explicitly lists which columns are emitted per version

Additional improvements (discovered while re-testing):

• Expanded README with NukeTestData section, output formats, and additional datasets documentation
• Added NukeTestData Quick Start examples
• Removed the .duplicate backup file that was accidentally included

Pester tests

We'll look into adding Pester unit tests (for Get-RandomDecimal, New-AwsAccountId, Get-WeightedRandomService, etc.) in a follow-up PR to keep this one focused on the generator itself.

[FallenHoot]

Fix pushed (61a6d0c): Resolve OutputPath to an absolute path before use. Export-Parquet is a .NET cmdlet that uses [IO.Directory]::GetCurrentDirectory() which can differ from PowerShell's C:\Users\zaolinsk\finops-toolkit — this caused 'Could not find a part of the path' errors when running from a different working directory. Fixed by calling System.Management.Automation.EngineIntrinsics.SessionState.Path.GetUnresolvedProviderPathFromPSPath() on OutputPath after parameter binding.

[flanakin]

Can we move this to src/powershell/Public/New-FinOpsTestData.ps1 so it can be published in the PS module? I'm fine with another verb name, but we should use an approved verb.

Side question: Should this be a generic script for any purpose or do we want to make it hubs-specific? I'm fine either way, but we'd follow different conventions.

[flanakin]

Can you look at the formatting conventions we have and apply them here as well. In this case, we indent the doc properties alongside the values.

Suggested change

	.SYNOPSIS
	Generates multi-cloud FOCUS-compliant test data for FinOps Hub validation.
	.SYNOPSIS
	Generates multi-cloud FOCUS-compliant test data for FinOps Hub validation.

[flanakin]

These are mentioning hubs-specific tables. Do you want to keep this specific to hubs? If so, I'd probably change the name to New-FinOpsHubTestData. But I also see value in breaking this out to support any number of scenarios:

• New-FinOpsTestData
• Set-FinOpsStorageBlobContent
• New-FinOpsExportManifest
• Add-FinOpsHubTestData

I see these as just breaking down what you have into smaller chunks. We don't need to do this now. I'm just thinking out loud about a growth path that would be reusable for more scenarios, if/when needed.

[flanakin]

I need to double check what's in FOCUS 1.3, but I believe the best term here is ServiceProvider to account for SaaS services that we could hypothetically support in the future.

Suggested change

	.PARAMETER CloudProvider
	.PARAMETER ServiceProvider

[flanakin]

nit: MaxRowCount or maybe just RowCount?

[flanakin]

nit: Billing Profile type doesn't match EA account agreement.

[flanakin]

Is this needed? Have you ever seen a case where this check failed? I'd love to get to a point where this code isn't necessary.

[flanakin]

Don't use the Az CLI. You're in PowerShell. Stick with Az PowerShell. Applies to all commands.

[flanakin]

nit: Creating a manifest is an awesome capability in and of itself. I'd love to see this as a separate New-FinOpsExportManifest command.

[flanakin]

Approaching 3K lines is a bit much. I'd love to see this broken out into multiple files.