Skip to content

[ASIM] Add Proofpoint POD MailLog ingestion-time parser - #15463

Merged
adbhogeGit merged 16 commits into
masterfrom
adbhoge-microsoft-proofpoint-maillog-draft-pr
Nov 2, 2026
Merged

adbhogeGit merged 16 commits into
masterfrom
adbhoge-microsoft-proofpoint-maillog-draft-pr

Conversation

@adbhogeGit

@adbhogeGit adbhogeGit commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Required items, please complete

Change(s):

  • Add the Proofpoint POD MailLog ingestion-time EmailEvent parser targeting ASimEmailEventLogs.
  • Add its source-table-specific CSV fixture and separate ingestion-time parser changelog; retain the existing query-time parser history independently.
  • Normalize missing envelope recipients before filtering and delivery classification while preserving the original sm bag in AdditionalFields. Keep IsDelivery's explicit boolean wrapper for DCR compatibility.
  • Change the parameterless and filtering MailLog query-time parsers to read source id directly with tostring(id). The ingestion-time parser already used this mapping. Update the QTP changelogs and generated ARM templates.
  • Scope the change to MailLog; no Message parser, converter KQL, or conversion reports are included.

Reason for Change(s):

  • Support ingestion-time normalization of Proofpoint POD MailLog SMTP receipt and delivery records using the existing query-time parser's mappings.
  • Preserve the source's legitimate event ID rather than substituting an empty value when the column is missing.

Version Updated:

  • N/A for detection templates. The ingestion-time parser is version 0.1.3; its separate changelog retains the initial 0.1.2 history. Both MailLog QTPs are version 0.1.3 with their own updated changelogs.

Testing Completed:

  • Passed the documented ingestion-time unit suite: 164 tests (python -m unittest discover -s ".script\tests\asimParsersTest\ingestTimeParser\tests" -p "test_asim_*.py").
  • Passed repository static ingestion-time parser validation with the paired query-time parser eligible for parity (python .script\tests\asimParsersTest\ingestTimeParser\asim_ingest_time_validation.py validate Parsers\ASimEmailEvent\IngestTimeParsers\ASimEmailEventProofpointPODMailLog.yaml --repo-root . --query-time-path Parsers\ASimEmailEvent\Parsers\ASimEmailEventProofpointPODMailLog.yaml).
  • Passed typed fixture preparation against ProofpointPODMailLog_CL_Schema.csv.
  • The tests used the session-local Python runtime because the system python alias was unavailable. Validation artifacts were saved outside the repository.
  • No direct user-workspace queries, deployments, or ingestion were performed. The authorized repository canary is responsible for live validation.

Checked that the validations are passing and have addressed any issues that are present:

  • The focused offline checks pass. Shared-canary run https://github.com/Azure/Azure-Sentinel/actions/runs/41118622329 (attempt 1, head 728c24c) confirms the visibility fix: 25 inputs, 24 selected, 1 rejected; all 24 expected target rows became visible, with no missing timestamps, in 137.145 seconds.
  • DCR-output schema validation and data validation passed (zero data errors); query-time execution returned 24 rows. Cleanup passed.
  • In that earlier canary, parity failed. The prepared raw-replay schema omits id, leaving QTP EventOriginalUid/EventUid empty while ITP IDs are populated (proposal [ASIM] Preserve source id in ingestion validation fixtures #15510 was closed without merging). Aligning rows by their unique TimeGenerated also exposes separate source-host mismatches: SrcHostname is empty in ITP for 13 receipt rows; SrcDomain/SrcDomainType/SrcFQDN differ for 11, and Src differs for the two local receipts. These are not fixed by recipient-array normalization. This PR is not claimed to pass full parity yet.
  • Current ingestion canary run 41142174672, attempt 1 at head 72d53da0bef5891c14f21d02683dde84987197ed, prepared 25 rows, selected 24 and rejected one. All 24 expected target rows became visible; DCR-output schema, data validation (zero errors), raw replay ingestion, and cleanup passed.
  • QTP replay then failed with HTTP 400 / SEM0100: 'extend' operator: Failed to resolve scalar expression named 'id'. Fixture preparation excludes id as reserved metadata, so the replay schema lacks the source column. Parity did not run.
  • Legacy ASIM template run 41142174667 at the same head failed because it applied query-time parser template checks to the ingestion-time YAML. The ingestion-time-specific static validation passed.
  • These CI failures require separately authorized maintainer changes to validation tooling. No tooling or fixture workaround is included in this PR, and full parity is not claimed.

Aditi Bhoge and others added 3 commits October 27, 2026 19:23
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adbhogeGit
adbhogeGit marked this pull request as ready for review October 28, 2026 19:43
@adbhogeGit
adbhogeGit requested review from a team as code owners October 28, 2026 19:43
Aditi Bhoge and others added 4 commits October 28, 2026 16:39
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Aditi Bhoge and others added 5 commits October 29, 2026 09:44
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Aditi Bhoge and others added 3 commits October 30, 2026 16:57
@contentautomationbot

Copy link
Copy Markdown

ASIM parsers have been changed. ARM templates were regenerated from the updated KQL function YAML files.
To find the new ARM templates, pull your branch.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adbhogeGit
adbhogeGit merged commit 7fdf038 into master Nov 2, 2026
32 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants