[JamesAde11]

Change(s)

Add a query-time ASIM EmailEvent 1.0.0 normalization parser for the native EmailEvents source table, with regular and filtering variants.

Proposed replacement for #15354. These PRs are alternative implementations, not separate additions to merge together. The earlier draft remains open until this replacement is accepted.

The contribution includes:

• Fresh regular and filtering parser YAML definitions.
• Registration in the EmailEvent unifiers, with parameter forwarding and source/global exclusions.
• Four parser/unifier changelogs and generated ARM deployment templates.
• The native source-table definition required by the repository KQL validator.
• Ten fully synthetic source records covering the 55-column Sentinel contract, each with Type=EmailEvents. No customer email records are redistributed.

Development helpers, bespoke tests, local tester/filter-harness adaptations, raw execution evidence and environment configuration are not included.

Reason for Change(s)

Normalize Microsoft Defender for Office 365 delivery records into the common EmailEvent contract so queries can use normalized email fields. This is query-time normalization: the PR does not create a normalized destination table, deploy functions into customer workspaces, or register an installed built-in parser.

The implementation was built afresh from the native source contract and schema requirements following the official ASIM parser creation skills. The skill files were followed directly rather than invoked through a registered skill. The read-only validation adaptations below are explicitly disclosed.

Mapping decisions

• Classify source records as EmailDelivery; preserve one output row per source record.
• Map native TimeGenerated to event start/end time. Do not substitute a Defender hunting envelope or a later remediation timestamp.
• Keep header From and SMTP envelope sender separate; map the actual recipient to Dst and retain native To/Cc arrays.
• Keep delivery outcomes separate from threat verdicts. Successful delivery does not mean a message is safe.
• Map Delivered/Junked to delivery Success, Replaced to Modified/Success, and Blocked or Failed/Dropped locations to Failure. Only explicit supported locations are normalized.
• Preserve missing/unknown information and diagnostics rather than inventing required values or policy causes.
• Prefer a valid IPv4 address, otherwise a valid IPv6 address; reject network prefixes as individual source addresses.
• Support disabled, optional unmapped-field packing and the standard EmailEvent filters, including literal IPv4/IPv6 prefixes.
• Do not enrich from other event tables, expand records, aggregate or deduplicate.

Version Updated

• New source parsers: 0.1.0.
• EmailEvent unifiers: 1.0.0 to 1.0.1.
• Target EmailEvent schema remains 1.0.0.

Testing Completed

Actual KQL execution used read-only inline synthetic fixtures in Log Analytics. No source records were ingested and no parser/tester functions were deployed.

Check	Result
Exact packaged YAML mapping checks, both variants	21 cases and 78 expected-field assertions per variant passed, 156 field assertions total
Exact packaged YAML and both unifiers: outer-parameter forwarding, every filter, disabled, pack and source/global exclusions	60 passed
Supplemental runtime controls	31 passed; separate optional-column-absence and future-field-packing checks also passed
Existing repository parser-template checks	30 passed
Official schema tester, both variants	Zero Error-level findings; 11 recommended-field warnings retained per variant
Separately corrected recipient metadata with the official data-tester logic	Zero Error-level findings on 17 conformance records per variant
Recipient-array negative controls	Numeric and mixed arrays remained rejected
Generated deployment assets	Six ARM JSON files parsed; four query/parameter definitions matched YAML; a generated ARM signature/query executed successfully inline

Skill adaptations and retained limitations

• Inline fixtures replaced querying a populated source table.
• Official tester definitions were inlined rather than deployed.
• Uncorrected upstream metadata rejects valid To/Cc string arrays because their element types are blank. Those two errors per variant are retained in local evidence. A separate query-only correction fills the three recipient-array element types; numeric/mixed negative controls still fail. The correction is not bundled here, and deployed testers reading uncorrected metadata may continue reporting the false failure.
• The official filter harness does not list EmailEvent and assumes a single filtered column. Four full-header recipient-reduction failures were retained. A local-only adaptation checks exact event identities and multiplicities across Dst/To/Cc instead; that adapted full-header run passed 60 checks. This is not an unchanged official-suite pass.

Eleven missing-recommended-field warnings per variant remain: DstIpAddr, DvcDomain, DvcHostname, DvcIpAddr, EmailProtocolName, EmailReplyToAddress, EventSeverity, EventUid, SrcDomain, SrcHostname and SrcUserUpn. Values are not fabricated to silence these warnings.

Hosted ingestion compatibility

• The packaged CSV filename matches the script's extraction of vendor, product and schema. Every row routes to native EmailEvents; the validator catalog's CustomTables directory does not make it a custom log.
• The current shared ingestion script does not support EmailEvents as a built-in and skips it. Neither its built-in schema lookup nor custom schema-selection branch runs. The native table reference also lists Ingestion API support as No.
• Exact upstream preparation functions were exercised locally without their write-side entry point. Timestamp rebasing and numeric conversion were verified. Dynamic values remain JSON strings; Boolean conversion depends on a schema-driven branch that is not reached for this source.
• sampleDataValidator selects JSON samples, not this CSV. A green check would not establish CSV ingestion or preservation of native dynamic types.

A maintainer-supported hosted validation route is still required. SafeToRun approval alone does not fix this ingestion limitation. No shared CI/tooling changes or custom-table routing workarounds are included.

Checked that the validations are passing and have addressed any issues that are present

The targeted local and engine-executed checks above passed within their stated scope. GitHub checks must be evaluated separately on the final commit. Hosted ASIM tests may require write-capable maintainer SafeToRun approval; new commits invalidate earlier approval even if the label remains visible. Skipped checks are not passes.

Current representative production-data validation, performance evaluation and deployment validation remain pending. Deployment links target upstream master and are not evidence that the new templates are deployable before merge.

Review requested

Requested reviewers: Derrick Lee (@yummyblabla) and Ofer Shezaf (@oshezaf).

Please review the field and delivery-action mappings, source compatibility, filtering/exclusion behavior, recipient metadata dependency, documented workflow adaptations and remaining validation requirements.

This is a draft for review.

[Derrick Lee (yummyblabla)]

You can leave this README.md untouched. Remove the changes here.