Team Cymru Scout: migrate playbooks from the HTTP Data Collector API to the Logs Ingestion API

[jstraney-tc]

Change(s)

Migrates the two Team Cymru Scout playbooks that still ingest through the HTTP Data Collector API over to the Logs Ingestion API:

• TeamCymruScoutLiveInvestigation — 21 ingestion actions
• TeamCymruScoutEnrichIncident — 6 ingestion actions

Every ApiConnection action posting to /api/logs through the azureloganalyticsdatacollector managed connector is now an Http action posting to a data collection rule stream, authenticated with the playbook's system-assigned managed identity.

Both templates now provision what they write to, so nothing is left for the operator to wire up by hand:

Resource added	LiveInvestigation	EnrichIncident
Microsoft.Insights/dataCollectionEndpoints	1	1
Microsoft.OperationalInsights/workspaces/tables	21	6
Microsoft.Insights/dataCollectionRules	1 (21 streams)	1 (6 streams)
Microsoft.Authorization/roleAssignments — Monitoring Metrics Publisher, scoped to the DCR	1	1

The data collector connection resource, its ARM variable, its dependsOn entry and its $connections entry are removed. TeamCymruScoutCreateIncidentAndNotify writes no logs and is unchanged.

Two commits. The first is a formatting-only normalization of the two templates — TeamCymruScoutEnrichIncident/azuredeploy.json had been serialized by PowerShell's ConvertTo-Json, which made any content change unreadable in a diff. It is split out so the migration itself can be reviewed on its own, and is verified to be a semantic no-op: both files parse to structures identical to their previous contents.

Reason for Change(s)

Support for the HTTP Data Collector API ends 2026-09-14 (migration guidance). This solution's Azure Function data connector was already moved to the Logs Ingestion API in an earlier release, but these two playbooks were not, and they account for all remaining Data Collector API usage in the solution.

Version Updated

Yes — solution 3.1.1 → 4.0.0, with a ReleaseNotes.md entry.

Major rather than minor because existing workspaces need a one-way table conversion before the migrated playbooks can write.

Compatibility

Column names and types are unchanged, so the 17 parsers and the workbook that read these tables keep working without modification. The DCR transforms project the raw Scout API fields onto the existing _s/_d/_b/_g column names, reproducing the flattening the Data Collector API applied implicitly (local.ip → local_ip_s) along with the ip/query/rank fields the playbooks graft on before ingesting.

Tables created by the old API are classic tables and must be converted once per workspace before the migrated playbooks run. Both playbook READMEs and the post-deployment metadata document the az monitor log-analytics workspace table migrate command and state that the conversion cannot be undone.

Testing Completed

No — opened as a draft pending deployment validation.

Verified statically:

• All templates parse; zero azureloganalyticsdatacollector, Log-Type or /api/logs references remain
• DCR stream declarations ↔ dataFlows ↔ outputStreams ↔ table resources are consistent in both directions
• Every transform emits only columns declared on its destination table, and reads only columns declared on its stream
• Diffing the workflow definitions against the previous commit shows exactly the 27 ingestion actions changed, with no actions added or removed
• The rules in .script/playbooksValidator.ts — PlaybookName parameter, single main workflow, resource-group location, mandatory metadata fields, valid entity types — pass for all three playbooks

Not yet done, and the reason this is a draft:

• Deployment to a Sentinel workspace
• End-to-end ingestion and a parser regression check against live data

The destination column names were derived from this solution's own parsers and from the playbooks' ParseJson schemas rather than from a live workspace, so confirming them against <Table>_CL | getschema is the main outstanding item. Will move this out of draft once that is done.

Checked that the validations are passing

Partially. The repo's Node validators could not be run locally (node_modules not installed), so the playbooksValidator rules were replicated and checked directly, as listed above. Package/ has not been regenerated — PowerShell was unavailable — so /package will be needed.

🤖 Generated with Claude Code

[Copilot]

Note

Copilot was unable to run its full agentic suite in this review.

Pull request overview

Updates the Team Cymru Scout solution metadata and playbook documentation to support the migration from the retired HTTP Data Collector API to the Azure Monitor Logs Ingestion API, and bumps the solution version accordingly.

Changes:

• Bumped solution version to 4.0.0 and updated publish metadata.
• Added release note entry for the playbook ingestion migration.
• Updated both playbook READMEs with Logs Ingestion API guidance and classic-table migration instructions.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
Solutions/Team Cymru Scout/SolutionMetadata.json	Updates last publish date metadata.
Solutions/Team Cymru Scout/ReleaseNotes.md	Adds 4.0.0 entry documenting the migration.
Solutions/Team Cymru Scout/Playbooks/TeamCymruScoutLiveInvestigation/README.md	Documents Logs Ingestion API behavior and classic-table migration steps.
Solutions/Team Cymru Scout/Playbooks/TeamCymruScoutEnrichIncident/README.md	Documents Logs Ingestion API behavior and classic-table migration steps.
Solutions/Team Cymru Scout/Data/Solution_TeamCymruScout.json	Bumps solution version to 4.0.0.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jstraney-tc]

@microsoft-github-policy-service agree [company="Team Cymru"]

[jstraney-tc]

@microsoft-github-policy-service agree company="Team Cymru"

[v-skharode]

Could you please verify the table migration examples in both README files? The placeholder --table-name "_CL" may cause users to append _CL twice if the table list command already returns the full table name. Clarifying the expected input would help avoid migration issues.

[jstraney-tc]

v-skharode thanks for the review. Confirming the table names and the migration example:

• The table names themselves are correct. Both playbooks write to _CL custom tables (Communication_Data_CL, Domain_Data_CL, etc.), and each README lists the full set the playbook writes. Those are unchanged from the previous Data Collector API version, so the 17 parsers and the workbook keep reading them without modification.

• The migration example was the issue you (and Copilot) flagged, and it's now fixed. az monitor log-analytics workspace table list returns the full name including the _CL suffix, but the migrate example used a <TableName>_CL placeholder, which would have led to a doubled suffix (Communication_Data_CL_CL) and a failed migration. Both READMEs and the post-deployment metadata in each azuredeploy.json now use --table-name "<TableName>" and explicitly say to use the name exactly as returned by table list / listed in the README, since it already includes _CL.

Pushed in the latest commit. Let me know if you'd like any further wording changes.

[v-skharode]

Thanks for the clarification and update. I verified the latest changes locally and confirmed the migration example now uses the correct table name returned by the table list command, avoiding the duplicate _CL suffix issue. The concern has been addressed.

[v-skharode]

I have a couple of follow-up questions:

The solution version has been updated from 3.1.1 to 4.0.0. Could you please clarify the reason for a major version bump rather than a patch/minor version update (for example, 3.1.2)?

I am unable to locate the updated package file/artifact associated with this version change in the PR. Could you please confirm whether the package was regenerated and included as part of this update?

[v-skharode]

Follow-up:
We are awaiting your response to the review comments shared earlier. Please provide an update on the requested clarifications and changes so that we can proceed with the review.

[jstraney-tc]

Thanks for the follow-up. Responses to both:

1. Version: agreed a major bump wasn't warranted. This change only migrates two playbooks to the Logs Ingestion API, so I've set the solution version to 3.1.2 (a patch over the published 3.1.1) rather than 4.0.0.

2. Package: you're right, the package hadn't been regenerated. I re-ran the V3 packaging tool and committed the regenerated artifacts, Package/3.1.2.zip and the updated Package/mainTemplate.json. Version 3.1.2 is now consistent across the solution, workbook, and playbook resources.

Both are in the latest commit. Let me know if you'd like any other changes.

[v-skharode]

Thanks for addressing the previous feedback. I noticed that the ARM-TTK validation is still failing on "apiVersions Should Be Recent In Reference Functions". The template appears to reference Microsoft.Logic/workflows API version 2017-07-01, which is being flagged as outdated by the validation. Could you please review the ARM-TTK failure and update accordingly?

[jstraney-tc]

v-skharode thanks for catching this. Fixed in the latest commit:

• The role-assignment principalId lookup in both migrated playbooks (TeamCymruScoutEnrichIncident and TeamCymruScoutLiveInvestigation) referenced Microsoft.Logic/workflows with API version 2017-07-01 inside the reference() call, which is what "apiVersions Should Be Recent In Reference Functions" was flagging. Both now use 2019-05-01, the latest stable Logic Apps API version.
• I also updated the apiVersion on the Microsoft.Logic/workflows resource declarations in those two playbooks to 2019-05-01 for consistency.
• The 3.1.2 package was regenerated with the V3 packaging tool, so Package/mainTemplate.json and Package/3.1.2.zip include the fix. The solution version stays at 3.1.2.

I ran Test-AzTemplate against the regenerated package locally and "apiVersions Should Be Recent In Reference Functions" now passes. Let me know if anything else comes up.

[jstraney-tc]

I see there's a few checks that have failed, but I'm not certain it's anything from the open PR — the JSON File Syntax Validation and Playbook Validation jobs both appear to have hit a GitHub API rate limit (403) before validating any files. Could you please rerun the pipeline? Is there anything additional needed from my side?