Repository navigation
ASIM: Add Proofpoint POD Message EmailEvent parsers - #15409
adbhogeGit wants to merge 19 commits into
Conversation
Add parameterless and filtering parsers, unifier registrations, fixtures, focused offline validation, and generated ARM assets. Live validation is deferred by user direction. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Remove generated ARM changes and retain source-only offline checks and documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
…roofpoint-message-parser
|
ASIM parsers have been changed. ARM templates were regenerated from the updated KQL function YAML files. |
There was a problem hiding this comment.
🔵 Needs a closer look
The reported KQL catalog failures, deferred live validation, and unverified source mappings require maintainer review before merging.
No open issues 🎉
What changed in this PR
Adds Proofpoint POD message support to ASIM EmailEvent 1.0.0, making ProofpointPODMessage_CL available through the standard EmailEvent unifiers.
Changes:
- Adds parameterless and filtering parsers that preserve one event per source message, with optional source details.
- Registers both parsers in the unifiers and includes generated ARM deployment artifacts.
- Adds synthetic fixtures, offline validation checks, changelogs, and documentation of mapping limitations and deferred live validation.
| File | Description |
|---|---|
| Sample Data/ASIM/ProofpointPODMessage_CL_Schema.csv | Defines the source fixture schema. |
| Sample Data/ASIM/Proofpoint_POD_EmailEvent_IngestedLogs.csv | Adds four synthetic message scenarios. |
| Parsers/ASimEmailEvent/Tests/ProofpointPODMessage.Tests.ps1 | Checks contracts, KQL types, filter bindings, and registrations offline. |
| Parsers/ASimEmailEvent/README.md | Documents mappings, filters, limitations, and validation status. |
| Parsers/ASimEmailEvent/Parsers/vimEmailEventProofpointPODMessage.yaml | Adds the filtering Proofpoint parser. |
| Parsers/ASimEmailEvent/Parsers/imEmailEvent.yaml | Registers the filtering parser and forwards parameters. |
| Parsers/ASimEmailEvent/Parsers/ASimEmailEventProofpointPODMessage.yaml | Adds source-to-EmailEvent normalization. |
| Parsers/ASimEmailEvent/Parsers/ASimEmailEvent.yaml | Registers the parameterless parser. |
| Parsers/ASimEmailEvent/CHANGELOG/vimEmailEventProofpointPODMessage.md | Records the filtering parser’s initial release. |
| Parsers/ASimEmailEvent/CHANGELOG/imEmailEvent.md | Records the filtering unifier update. |
| Parsers/ASimEmailEvent/CHANGELOG/ASimEmailEventProofpointPODMessage.md | Records the parameterless parser’s initial release. |
| Parsers/ASimEmailEvent/CHANGELOG/ASimEmailEvent.md | Records the parameterless unifier update. |
| Parsers/ASimEmailEvent/ARM/vimEmailEventProofpointPODMessage/vimEmailEventProofpointPODMessage.json | Adds the filtering parser deployment template. |
| Parsers/ASimEmailEvent/ARM/vimEmailEventProofpointPODMessage/README.md | Provides filtering parser deployment links. |
| Parsers/ASimEmailEvent/ARM/imEmailEvent/imEmailEvent.json | Updates the deployed filtering unifier. |
| Parsers/ASimEmailEvent/ARM/FullDeploymentEmailEvent.json | Includes both source parsers in full deployment. |
| Parsers/ASimEmailEvent/ARM/ASimEmailEventProofpointPODMessage/README.md | Provides parameterless parser deployment links. |
| Parsers/ASimEmailEvent/ARM/ASimEmailEventProofpointPODMessage/ASimEmailEventProofpointPODMessage.json | Adds the parameterless parser deployment template. |
| Parsers/ASimEmailEvent/ARM/ASimEmailEvent/ASimEmailEvent.json | Updates the deployed parameterless unifier. |
Review effort: Balanced
Register the exact CCP custom-table schema and normalize JSON-encoded dynamic columns so CI fixture ingestion produces typed EmailEvent arrays. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Document that checked-in parser ARM templates are generated and committed by the repository GitHub action, rather than manually edited. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Keep the parser mappings native to their dynamic connector columns; the hosted recipient-array error comes from missing ArrayValuesType metadata, and the filter validator lacks EmailEvent support. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
…roofpoint-message-parser
Derrick Lee (yummyblabla)
left a comment
Derrick Lee (yummyblabla)
left a comment
There was a problem hiding this comment.
Don't
| @@ -0,0 +1,216 @@ | |||
| $ErrorActionPreference = 'Stop' | |||
There was a problem hiding this comment.
We don't need this file
| - [Normalization and the Advanced Security Information Model (ASIM)](https://aka.ms/AboutASIM) | ||
| - [Deploy all of ASIM](https://aka.ms/DeployASIM) | ||
| - [ASIM Email Event normalization schema reference](https://aka.ms/ASimEmailEventDoc) No newline at end of file | ||
| - [ASIM Email Event normalization schema reference](https://aka.ms/ASimEmailEventDoc) |
There was a problem hiding this comment.
Changes not needed in this README
Normalize observed DMARC policies and message actions in both parser variants. Remove the standalone PowerShell harness and restore the schema README as requested in review. Leave ARM generation to the repository action. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
…roofpoint-message-parser
|
Schema tester Output: |
Only map valid FQDNs into SrcFQDN and SrcDomain, and retain valid single-label hostnames in SrcHostname. Leave malformed and IP-shaped connection.host values out of hostname fields. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
|
ASim DataTester output: |
Use the current local calendar date in parser changelogs and metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Describe how connection.host maps to ASIM host fields in both Message parser metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Required items, please complete
Change(s):
ProofpointPODMessage_CLtable, with unifier registrations, changelogs and synthetic fixtures.KS204failure.connection.hostbefore mapping ASIM source identity fields: valid FQDNs map toSrcFQDN/SrcDomain, valid single-label hostnames map toSrcHostname, and IP-shaped or malformed values are not mislabeled. This rule is documented in parser metadata.356fd3691bcontains only action-generated ARM README updates after parser commit840108b770.Reason for Change(s):
KS204, incorporates supplied source value enumerations, and fixes EmailEvent DataTester errors forSrcHostname,SrcDomain, andSrcFQDNwithout fabricating host identity.Version Updated:
0.1.2, with paired changelogs and updated LastUpdated metadata.Testing Completed:
localhost, bracketed/unbracketed IP literals, malformed values and invalid hostname labels.git diff --check; source changes are parser metadata/query and changelogs. No manual ARM edits.840108b770confirms no errors forSrcHostname,SrcDomain, orSrcFQDN. It reports onlyEmailToRecipientsarray element-type validation, plus the unsupported-schema filtering check. Template validation and sample-data ingestion passed. The repository action then generated ARM README output in head356fd3691b.Checked that the validations are passing and have addressed any issues that are present:
pass->Pass;quarantine/reject->FailwithEmailDmarcPolicy=Quarantine/Reject;none->None. Matching is case-insensitive.temperror/permerrorremain inEmailDmarcOriginalResultand produce no normalized result because EmailEvent 1.0.0 has no matching normalized error enum.reject/discard->Blocked;hold->Quarantined;redirect->Redirected;continue/accept/retry->NA. Unknown values remainNA. Inspection events retainEventResult=NA; filter acceptance is not proof of inbox delivery.EmailToRecipientsarrays of sanitized email strings fail because the ASIM tester catalog leavesArrayValuesTypeblank. The filter test reports EmailEvent is unsupported. Shared validator support is tracked separately in PR #15412; this PR does not modify validator code or the tester catalog.ProofpointPODMessage_CL. No user-workspace queries, deployment or ingestion were performed; synthetic fixtures are not observed telemetry.