Add Datazag Threat Intelligence solution - #14888
Conversation
|
@microsoft-github-policy-service agree company="Datazag" |
v-atulyadav
added
the
New Solution
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a new Datazag Threat Intelligence Microsoft Sentinel solution that onboards STIX 2.1 indicators via TAXII 2.1 and includes DNS-based detection + hunting content.
Changes:
- Introduces solution packaging assets (metadata, release notes, ARM template, CreateUI, test parameters).
- Adds one scheduled analytic rule (ASIM DNS indicator match) and one DNS retro-hunt query.
- Updates connector-id validation list to include the new StaticUI connector id.
Reviewed changes
Copilot reviewed 10 out of 12 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| Solutions/Datazag/SolutionMetadata.json | New solution publisher/support metadata and release dates. |
| Solutions/Datazag/ReleaseNotes.md | Initial release notes entry for the new solution. |
| Solutions/Datazag/Package/testParameters.json | ARM test parameters for solution deployment validation. |
| Solutions/Datazag/Package/mainTemplate.json | Main ARM template registering content package, connector, rule template, and hunting query template. |
| Solutions/Datazag/Package/createUiDefinition.json | Create UI wizard content for solution installation. |
| Solutions/Datazag/Hunting Queries/DatazagRetroHunt_DNS.yaml | New hunting query (DNS retro-hunt against current TI set). |
| Solutions/Datazag/Data/Solution_Datazag.json | Solution manifest referencing connector/rule/query artifacts. |
| Solutions/Datazag/Data Connectors/DatazagThreatIntelligence_Connector.json | StaticUI connector definition and TAXII onboarding instructions. |
| Solutions/Datazag/Analytic Rules/DatazagDomainIndicatorMatch_DNS.yaml | New scheduled analytic rule template (ASIM DNS join with TI). |
| .script/tests/detectionTemplateSchemaValidation/ValidConnectorIds.json | Adds Datazag connector id to schema validation allow-list (currently broken). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| "description": "Matches active, high-confidence domain indicators delivered by Datazag against DNS query\nactivity normalized by the ASIM Dns schema. Because the rule reads from _Im_Dns rather than\na vendor-specific table, it works across any DNS source the workspace has normalized -\nWindows DNS, Infoblox, Cisco Umbrella, Azure Firewall, Zscaler and others - without\nmodification.\n\nA match means a host inside the estate resolved a domain that Datazag scored as\nbrand-impersonation, platform-impersonation or attacker infrastructure at or above the\nblock-confidence floor. Resolution is not proof of compromise, but it establishes that the\ndomain reached a resolver from inside the estate, which is the earliest observable point in\nmost credential-phishing chains.\n\nIndicator revisions are collapsed with arg_max by Id, so a re-derived indicator does not\nproduce a duplicate match. Withdrawn (IsDeleted), revoked and expired indicators are\nexcluded.", | ||
| "displayName": "Datazag - impersonation domain resolved in DNS", | ||
| "enabled": false, | ||
| "query": "let ioc_lookback = 14d;\nlet dns_lookback = 1h;\nlet confidence_floor = 85;\nlet DatazagDomains =\n ThreatIntelIndicators\n | where TimeGenerated >= ago(ioc_lookback)\n | where SourceSystem == \"Datazag\"\n | where ObservableKey == \"domain-name:value\"\n | where isnotempty(ObservableValue)\n // collapse indicator revisions to the most recent state of each\n | summarize arg_max(TimeGenerated, *) by Id\n | where IsDeleted == false\n | where tostring(Data.revoked) != \"true\"\n | where isempty(ValidUntil) or ValidUntil > now()\n | where Confidence >= confidence_floor\n | extend IndicatorDomain = tolower(trim_end(@\"\\.\", ObservableValue))\n | project\n IndicatorDomain,\n IndicatorStixId = tostring(Data.id),\n IndicatorRecordId = Id,\n Confidence,\n ThreatType,\n IndicatorEvidence = tostring(Data.labels),\n IndicatorSummary = tostring(Data.description),\n IndicatorValidUntil = ValidUntil;\nlet DnsActivity =\n _Im_Dns(starttime=ago(dns_lookback), endtime=now())\n | where isnotempty(DnsQuery)\n | extend IndicatorDomain = tolower(trim_end(@\"\\.\", DnsQuery))\n | project\n DnsTime = TimeGenerated,\n IndicatorDomain,\n DnsQuery,\n SrcIpAddr = column_ifexists(\"SrcIpAddr\", \"\"),\n SrcHostname = column_ifexists(\"SrcHostname\", \"\"),\n SrcUsername = column_ifexists(\"SrcUsername\", \"\"),\n DnsResponseName = column_ifexists(\"DnsResponseName\", \"\"),\n EventVendor = column_ifexists(\"EventVendor\", \"\"),\n EventProduct = column_ifexists(\"EventProduct\", \"\");\nDatazagDomains\n| join kind=innerunique DnsActivity on IndicatorDomain\n| summarize\n FirstSeen = min(DnsTime),\n LastSeen = max(DnsTime),\n QueryCount = count(),\n SourceIps = make_set(SrcIpAddr, 50),\n SourceHosts = make_set(SrcHostname, 50),\n Users = make_set(SrcUsername, 50),\n ResolvedTo = make_set(DnsResponseName, 20),\n SeenBy = make_set(strcat(EventVendor, \" \", EventProduct), 10)\n by\n IndicatorDomain, IndicatorStixId, IndicatorRecordId, Confidence,\n ThreatType, IndicatorEvidence, IndicatorSummary, IndicatorValidUntil\n| extend\n SourceIp = tostring(SourceIps[0]),\n SourceHost = tostring(SourceHosts[0]),\n UserRaw = tostring(Users[0])\n| extend\n AccountNTDomain = iff(UserRaw has @\"\\\", tostring(split(UserRaw, @\"\\\")[0]), \"\"),\n AccountName = iff(UserRaw has @\"\\\", tostring(split(UserRaw, @\"\\\")[1]), UserRaw)\n| project-away UserRaw\n| order by LastSeen desc\n", |
| "contentSchemaVersion": "3.0.0", | ||
| "displayName": "Datazag", | ||
| "publisherDisplayName": "Datazag", | ||
| "descriptionHtml": "<p><strong>Note:</strong> Please refer to the following before installing the solution:</p>\n<p>• Review the solution <a href=\"https://github.com/Azure/Azure-Sentinel/tree/master/Solutions/Datazag/ReleaseNotes.md\">Release Notes</a></p>\n<p>• There may be <a href=\"https://aka.ms/sentinelsolutionsknownissues\">known issues</a> pertaining to this Solution, please refer to them before installing.</p>\n<p>The <a href=\"https://datazag.com/\">Datazag</a> solution for Microsoft Sentinel delivers impersonation and attacker-infrastructure indicators derived from Certificate Transparency at issuance time, published as STIX 2.1 objects over a TAXII 2.1 server.</p>\n<p>Indicators are ingested using Microsoft Sentinel's built-in <strong>Threat Intelligence - TAXII</strong> data connector and land in the native <strong>ThreatIntelIndicators</strong> table. No custom table, data collection rule or workspace function is required, and indicators are available to analytics rules and hunting queries immediately. Because indicators are retained in the workspace, existing logs can also be retro-hunted against them.</p>\n<p><strong>Prerequisites:</strong></p>\n<ol type=\"a\">\n<li><p>An active Datazag subscription. Contact Datazag to obtain the API root, collection ID and credentials.</p>\n</li>\n<li><p>Analytic rules and hunting queries in this solution query DNS and web session activity through the <a href=\"https://learn.microsoft.com/azure/sentinel/normalization\">Advanced Security Information Model (ASIM)</a>. The relevant ASIM parsers must be deployed in the workspace for this content to return results.</p>\n</li>\n</ol>\n<p><strong>Data Connectors:</strong> 1, <strong>Analytic Rules:</strong> 1, <strong>Hunting Queries:</strong> 1</p>\n<p><a href=\"https://aka.ms/azuresentinel\">Learn more about Microsoft Sentinel</a> | <a href=\"https://aka.ms/azuresentinelsolutionsdoc\">Learn more about Solutions</a></p>\n", |
There was a problem hiding this comment.
Addressed both review points: hunting query description reduced to 227 characters; alertDescriptionFormat reduced from 6 parameters to 3. Data connector removed from the solution — TAXII onboarding instructions (API root, collection IDs, credential contact) now live in the solution description, and both content items declare ThreatIntelligenceTaxii as the required connector. Package regenerated at 3.0.3; arm-ttk 48/49 with only the known contentProductId/id failure.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
Hi Peter Chaplin (@peterchap), Please review the Copilot comment and implement it if needed. Since this is a new solution, only the 3.0.0 package is required. Please remove the extra packages, keep just one 3.0.0 package, and commit the changes. Also, make sure the PR includes only one package and release note should be short and clear to understand. Additionally, please let us know which type of connector you are using for this solution. We would also be happy to help migrate your solution to CCF. To review the available options, please contact the Microsoft Sentinel Partners at AzureSentinelPartner@microsoft.com. If this connector is needed, please obtain approval from the azure app assure team by email and share the screenshot with us so we can proceed. Thanks!! |
Addresses the reviewer request to ship one package at 3.0.0 for a new solution, plus the outstanding automated findings. Packages: removed 3.0.1, 3.0.2, 3.0.3, 3.0.4 and 3.0.6; added a single 3.0.0.zip. The version had been bumped through review iterations, leaving five archives and a manifest that disagreed with all of them (manifest said 3.0.4, newest zip was 3.0.6). The new package is built from the loose Package/ files, which are the current content — verified rather than assumed. 3.0.6.zip was the OLDEST archive (committed 2026-08-12) and is stale: it still filtered indicators on SourceSystem == "Datazag", the exact issue raised in review. The loose files and the YAML sources both use Data.created_by_ref, and the packaged copy is byte-identical to them. Version is now 3.0.0 in mainTemplate.json (_solutionVersion, version, and the three template-version descriptions) and Solution_Datazag.json. The four contentSchemaVersion fields also read 3.0.0 and are a different field — untouched. Release notes reduced to a single row and one leading pipe per line, fixing the extra empty column some renderers produced. Removed test.py — a local debugging script with a hardcoded machine path that should never have been submitted. It was also the subject of the CodeQL "file is not always closed" alert. Other review findings were already resolved in the working tree and are re-verified here: no merge-conflict markers in ValidConnectorIds.json (parses as valid JSON, contains DatazagThreatIntelligence), no BasePath in Solution_Datazag.json, no "web session activity" text, the analytic rule requires only the DatazagThreatIntelligence connector rather than Azure Firewall and Windows DNS, and the rule description is a clean YAML block scalar without wrapping quotes. All six JSON artifacts parse. No 3.0.1-3.0.6 references remain anywhere in the solution. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The connector id was dropped from the validation list when master was merged into this branch (c642393). ValidConnectorIds.json is a shared file that every solution appends to, so it conflicts on most upstream merges, and this entry was lost in the resolution. Without it the schema validation for the analytic rule fails: the rule and the mainTemplate both declare connectorId DatazagThreatIntelligence, and that id must appear in this list to be considered valid. Appended as a single entry — 344 -> 345, file still parses as one valid JSON array, and the diff touches only the added line. Note this is the same file that previously carried unresolved merge-conflict markers in this PR, so it is worth re-checking after any future upstream merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The analytic rule and hunting query both filtered `Confidence >= 85`. That was
right when written — the feed then carried ORANGE and low-confidence material,
so the rule protected the SOC from it. The feed now does that filtering itself:
RED-only across every lane, a publication floor of 55, and the non-RED backlog
withdrawn.
So 85 had become a SECOND gate on already-gated material, and measured against
the live collection (6,951 indicators) it was cutting 16.9% of the feed —
including almost all of one lane:
BRAND fires 2 · excluded 679
INFRASTRUCTURE fires 100 · excluded 117
PLATFORM fires 5,671 · excluded 382
Two of 681 brand indicators could fire the rule. Brand alerts score 78, below
85, so a customer installing this solution would get essentially no
brand-impersonation detections — the lane a bank or retailer most wants
alerting on. Infrastructure lost nearly half for the same reason: its
confidence is now derived from corroboration strength, which puts
bulletproof-ASN evidence at 70 and host-campaign clusters at 90.
Set to 55 rather than a lower margin so the FEED is the single place that
decides what is publishable. Two thresholds that must be kept in sync is the
drift this repo's review process rightly asks about, and it is easier to
explain one number than two.
Nothing in the PR was factually wrong — this only became visible after the feed
policy changed, so it is tuning, not a correction.
⚠️ REPLACEMENT WAS PHRASE-SCOPED, NOT `85` -> `55`. The literal "85" appears 10
times in mainTemplate.json and the STIX identity reference contains it
(identity--55a5a448-c6f1-5128-bc71-4d85b719131e). A blind substitution would
have corrupted the identity every query keys off. Verified: 4 lines changed, 8
identity references intact, all JSON parses.
Package rebuilt from the loose files and byte-identical to them. Version stays
3.0.0 — it is not published, so no new package, which is what the review asked
for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks for the review. All points addressed in the latest commit. Single 3.0.0 package Removed One note on which content went into the package: Release notes Reduced to a single row, and each row now starts with a single pipe so the table renders without an extra empty column. Copilot comments
One further change: This branch was updated with a Flagging it because it explains why a shared file appears in this PR. It is the same file that previously carried the merge-conflict markers, which is inherent to an append-only list every solution edits. One tuning change, not a review item: rule confidence floor 85 -> 55 Flagging this because it is a behavioural change to the analytic rule since your last look, rather than a response to feedback. The rule and hunting query filtered The floor is now 55, matching the feed's own, so there is a single place that decides what is publishable rather than two thresholds to keep in sync. No new package: the version remains 3.0.0 and Connector type The solution does not ship a data-ingestion connector. It uses Microsoft Sentinel's built-in Threat Intelligence - TAXII connector: the customer points that connector at our TAXII 2.1 API root and collection ID, and indicators land in the native What we include is a static connector definition ( CCF migration and App Assure Thank you for the offer. We are requesting App Assure approval now and will share the screenshot once we have it. On CCF: as above, the solution ships no ingestion connector of its own — Microsoft's built-in TAXII connector performs the ingestion — so please let us know if you still see a migration path here and we will follow up with the Sentinel Partners alias. |
Peter Chaplin (peterchap)
left a comment
Peter Chaplin (peterchap)
left a comment
There was a problem hiding this comment.
its ok
|
Hi Peter Chaplin (@peterchap), We are putting your PR on hold because your solution uses data connector which is not Sentinel Codeless Connector Framework (CCF). please let us know which type of connector you are using for this solution. We would also be happy to help migrate your solution to CCF. To review the available options, please contact the Microsoft Sentinel Partners at AzureSentinelPartner@microsoft.com. If this connector is needed, please obtain approval from the azure app assure team by email and share the screenshot with us so we can proceed. Thanks!! |
…nector
The review placed this PR on hold because the solution ships a data connector
that is not built on the Codeless Connector Framework. That is correct as
stated: `DatazagThreatIntelligence_Connector.json` was a STATIC connector
definition — no kind, no apiEndpoint, no pollingConfig — so it performed no
ingestion, but it was still a data connector artifact and it was not CCF.
Rather than seek an exemption for a connector that never ingested anything, the
connector is removed. This follows the shape of TI solutions already merged in
this repo:
HoneyLabs no Data Connectors folder; rules declare ThreatIntelligenceTaxii
ThreatConnect no Data Connectors folder; rules declare ThreatIntelligence
Datazag (now) no Data Connectors folder; rules declare ThreatIntelligenceTaxii
Ingestion was always performed by Microsoft's built-in Threat Intelligence -
TAXII connector; the removed file only documented how to configure it. Those
instructions now live in the solution Description, where the two solutions above
keep theirs, so nothing is lost to the customer.
Changes:
* deleted Data Connectors/DatazagThreatIntelligence_Connector.json
* dropped "Data Connectors" and "StaticDataConnectorIds" from
Data/Solution_Datazag.json
* analytic rule and hunting query now declare
connectorId: ThreatIntelligenceTaxii / ThreatIntelligenceIndicator
* de-registered DatazagThreatIntelligence from ValidConnectorIds.json
(345 -> 344; this reverts the entry added earlier in this PR, and the file
returns to its upstream contents)
* setup steps folded into the solution Description
Package REGENERATED with the repo's own tool
(Tools/Create-Azure-Sentinel-Solution/V3/createSolutionV3.ps1) rather than
hand-edited — the connector was woven through 28 references in mainTemplate.json
and editing that by hand would have been a transcription-error surface. Result:
0 DatazagThreatIntelligence references, 0 dataConnector references, 1
ThreatIntelligenceTaxii reference, and mainTemplate drops from 36,003 to 25,642
bytes.
⚠️ THE GENERATOR BUMPED THE VERSION TO 3.0.1 AND IT HAS BEEN REVERTED. The review
asked for a single 3.0.0 package for a new solution, so the version is pinned
back to 3.0.0 in mainTemplate.json and Solution_Datazag.json and 3.0.1.zip was
deleted. The three contentSchemaVersion fields also read 3.0.0 and are a
different field — untouched. Package rebuilt and byte-identical to the loose
files.
arm-ttk during regeneration: 48 pass, 1 fail — "IDs Should Be Derived From
ResourceIDs" on contentProductId/id. That check is failed by the generator's own
output rather than by anything specific to this solution.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thank you — that is a fair point, and we have resolved it by removing the connector rather than seeking an exemption for it. What the solution shipped, and why it is gone
The solution now ships no data connector This follows the shape of TI solutions already merged in this repository:
Ingestion was always performed by Microsoft's own built-in TAXII connector — the customer points it at our TAXII 2.1 API root and collection ID, and indicators land in the native Changes in the latest commit:
The package was regenerated with On your three questions
If we have misread the precedent — for example if HoneyLabs and ThreatConnect predate the CCF requirement and were grandfathered rather than being an accepted current pattern — please let us know and we will follow whichever route you prefer. |
|
Hi Peter Chaplin (@peterchap), |
…cription, remove data connector
|
Hi Peter Chaplin (@peterchap), could you please change the package version back from 3.0.3 to 3.0.0? Since this is the initial version of the solution, it should remain 3.0.0. Please update it. Thanks! |
|
Version reset to 3.0.0 and package regenerated. Stale zips removed; Package/ now contains 3.0.0 only. |
|
Logos/Datazag.png was missing from the branch — the solution's Logo field references it and it would have 404'd on the listing. Added in the latest commit and rebased onto your ReleaseNotes update. No other changes. |
|
Hi Peter Chaplin (@peterchap), solutions logo should be in svg format please update it same update in data file url as well. Thanks! |
|
Logo converted to SVG (Logos/Datazag.svg) and the Logo URL in the data file updated to match. Package regenerated at 3.0.0. |
Datazag Threat Intelligence solution
New solution publishing STIX 2.1 impersonation and attacker-infrastructure
indicators over TAXII 2.1, ingested via Sentinel's built-in
Threat Intelligence - TAXII connector.
Contents:
Version updated:
Testing Completed:
Partial. mainTemplate.json deployed successfully to a Sentinel
workspace with no custom parsers, functions or tables. Connector,
analytic rule template and hunting query all appear in their
galleries. createUiDefinition.json validated in CreateUISandbox.
Both KQL queries parse against ThreatIntelIndicators, which is
receiving live data from the feed in that workspace.
The analytic rule and hunting query join against ASIM _Im_Dns.
The test workspace has no DNS telemetry, so the join has been
validated for syntax but not for match behaviour. Happy to
take guidance if behavioural validation is required.
Checked that the validations are passing and have addressed any issues: