Skip to content

Fix Proofpoint POD MailLog ingestion-time relay-host normalization parity - #15512

Closed
adbhogeGit wants to merge 6 commits into
adbhoge-microsoft-proofpoint-maillog-draft-prfrom
adbhoge-microsoft-maillog-parity-fix
Closed

adbhogeGit wants to merge 6 commits into
adbhoge-microsoft-proofpoint-maillog-draft-prfrom
adbhoge-microsoft-maillog-parity-fix

Conversation

@adbhogeGit

@adbhogeGit adbhogeGit commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Required items, please complete

Change(s):

  • Stack this incremental correction on #15463, verified at 728c24c32ec119e898a5400544a6012e6a0a5cae. Base branch: adbhoge-microsoft-proofpoint-maillog-draft-pr, not master.
  • Split the parsed/fallback relay host into @ components, select the last component using the parenthesized positive index (array_length(RelayHostParts) - 1), and remove trailing dots with anchored extract. Both source and destination mappings consume the same normalized host.
  • Keep the ITP at 0.1.4 with its separate PR #15512 changelog. Per the user's follow-up, change both the parameterless and filtering QTP from tostring(column_ifexists("id", "")) to tostring(id), bump each QTP to 0.1.3, refresh LastUpdated, and update each matching changelog. The ITP already uses direct tostring(id) and is unchanged by the ID follow-up.
  • Preserve the original 25-row fixture, raw sm, explicit Boolean IsDelivery, valid source-ID values and EventUid prefixing, TimeGenerated, filter, output shape, IP validation, non-ID optional fields, and all unrelated mappings. These two source-ID fallbacks were the only column_ifexists/columnifexists usages in the three MailLog parsers. No fan-out, aggregation, generated ARM, harness, tooling, test implementation, schema, workflow, script, or Tools/ changes.

Reason for Change(s):

  • Canary run 41118622329, attempt 1 tested the parent head, not this correction: 25 inputs / 24 selected / 1 rejected; target visibility 24/24, DCR-output schema and ASIM data passed; QTP executed 24 rows but full parity failed.
  • Timestamp-aligned captured rows identify independent host defects: SrcHostname differs in 13 receipt rows, SrcDomain/SrcDomainType/SrcFQDN in 11, and Src in two local receipts. These are 11 host [IPv4] inputs and two user@localhost inputs. Matching relay IPs and delivery-device host fields isolate the failure boundary to compound RelayHost normalization before mapping.
  • The artifacts do NOT establish whether negative indexing or regex replacement is the engine-level cause. Replace the compound expression with a documented positive-index/anchored-extraction form without claiming an unproven platform defect. Last-@ behavior, ordinary FQDNs, absent relays, localhost, multiple trailing dots, and destination/local-delivery fallback match QTP semantics offline.
  • Unresolved tooling blocker: source-ID replay still requires an authorized maintainer tooling fix. #15510 was closed WITHOUT merge at 2026-10-30T21:26:30Z; it is prior art, not an open or forthcoming dependency. Fixture preparation omits legitimate source id from raw/QTP replay; all 24 captured QTP UIDs are empty while ITP UIDs are populated. Preserve valid IDs; do not duplicate or alter protected harness implementation, clear IDs, or weaken parity.
  • #15501 merged at 2026-10-30T22:32:18Z (head 3a124189f45e147facb9f81a3d66ac792ae231da). Its direct source-ID mapping and accepted CI failure do not fix shared replay or demonstrate passing full parity. This PR addresses parser-side host normalization and the user-directed direct source-ID requirement, not the underlying replay defect or all discovered issues.
  • User-accepted CI limitation: the Proofpoint source contract defines id as string in .script/tests/KqlvalidationsTests/CustomTables/ProofpointPODMailLog_CL.json and Sample Data/ASIM/ProofpointPODMailLog_CL_Schema.csv. Require that source column rather than silently substitute an empty ID. The fixture preparer still excludes id, including from explicit schemas. Consequently, raw replay without that column is expected to fail at qtpExecution with a missing-id error, not produce successful parity. The user explicitly accepted this Proofpoint edge-case failure. This is not a harness repair, parity waiver, authorization for Azure access, or proof of runtime success.
  • References: transformation KQL, dynamic arithmetic subscripts, extract.

Version Updated:

Testing Completed:

  • Offline only. $Python below resolves to the existing local CPython 3.13 executable; $Artifacts is the external session-artifact directory. Exact command invocations from the repository root and their results:
& $Python .script\tests\asimParsersTest\ingestTimeParser\asim_ingest_time_validation.py validate 'Parsers\ASimEmailEvent\IngestTimeParsers\ASimEmailEventProofpointPODMailLog.yaml' --repo-root . --query-time-path 'Parsers\ASimEmailEvent\Parsers\ASimEmailEventProofpointPODMailLog.yaml' --output "$Artifacts\static-validation.json"
  • Passed: valid=true, zero findings, paired QTP eligible.
& $Python .script\tests\asimParsersTest\ingestTimeParser\asim_fixture_preparation.py prepare 'Sample Data\ASIM\Proofpoint_POD MailLog_EmailEvent_ProofpointPODMailLog_CL_IngestedLogs.csv' --expected-source-raw-table ProofpointPODMailLog_CL --target-id asimemaileventproofpointpodmaillog --target-group ASimEmailEventLogs --repository Azure/Azure-Sentinel --pull-request 0 --run-id 41118622329 --run-attempt 1 --now '2026-10-30T20:00:00Z' --output "$Artifacts\prepared-fixture.json"
  • Passed: 25 original rows prepared; repeat invocation reused the identical immutable artifact. No ingestion performed.
& $Python -m unittest discover -s '.script\tests\asimParsersTest\ingestTimeParser\tests' -p 'test_asim_*.py'
  • Passed: 164 tests. The printed Output path must not replace the source fixture diagnostic belongs to an expected negative test; suite result is OK.
& $Python "$Artifacts\verify_host_normalization.py"
git -c core.longpaths=true diff --check
  • Passed before the direct-ID follow-up, at parser head 5fa425176652825771842b7f5cecfd8870dbd5bf: session-only Python semantic checks against the exact captured parser pair, all 24 timestamp-aligned rows, 12 explicit host examples, and 3,456 synthetic mapping combinations (multiple @, empty last component, no relay, hostname/FQDN, trailing dots, IPv4/IPv6/invalid IP, receipt/delivery, local fallback, and device variants). The script asserts the query is unchanged outside relay-host normalization, retains 25/24/1 selection, and checks raw sm preservation. Modeling only the revised host fields leaves exactly the known UID differences; AdditionalFields bags are compared structurally because serialization order differs. This historical model is not a claim that the revised direct-ID QTP executes against the captured replay. These checks are NOT KQL/DCR execution, a substitute for shared canary, or new repository harness tests. Whitespace check passed.

  • Direct-ID follow-up commands and results (same $Python and $Artifacts):

& $Python "$Artifacts\verify_direct_id.py"
& $Python .script\tests\asimParsersTest\ingestTimeParser\asim_ingest_time_validation.py validate 'Parsers\ASimEmailEvent\IngestTimeParsers\ASimEmailEventProofpointPODMailLog.yaml' --repo-root . --query-time-path 'Parsers\ASimEmailEvent\Parsers\ASimEmailEventProofpointPODMailLog.yaml' --output "$Artifacts\direct-id-static-validation.json"
& $Python -m unittest discover -s '.script\tests\asimParsersTest\ingestTimeParser\tests' -p 'test_asim_*.py'
git -c core.longpaths=true diff --check
  • Passed: offline content assertions confirm that each QTP query changes only the source-ID expression, each version/changelog matches, other metadata/filters are preserved, and the ITP is byte-for-text unchanged from 50ca9f982e0542af78a1dec33841c4508afc10e8. Static validation valid=true, zero findings, paired QTP eligible. Complete ingestion-time unit suite: 164 tests, OK (8.900s); expected negative-test output-path diagnostic is not a failure. Whitespace passed. These checks do not execute either QTP in a KQL engine or a DCR.

Checked that the validations are passing and have addressed any issues that are present:

  • Offline checks above pass. Live workspace testing is explicitly deferred: no direct Azure queries, deployments, writes, synthetic ingestion, or new canary run.
  • No live DCR/parity success is claimed for this revision. The previous canary passed schema/data but failed full parity. With the new direct-ID QTP, replay omitting id is expected to fail qtpExecution instead; this expected CI failure has not been executed or observed for this revision. The user accepts it, but it does not repair fixture replay or validate parity. The workflow targets master only, so this stacked PR may not trigger shared canary. Source-ID replay needs an authorized maintainer tooling fix; closed [ASIM] Preserve source id in ingestion validation fixtures #15510 is not an available dependency. Once that unresolved blocker is fixed on the trusted base and the stacked changes can be tested through the authorized existing-PR GitHub canary, rerun full DCR/filter/target/schema/data/QTP/parity validation before considering runtime verification complete. Accepted CI failure in [ASIM] Add Proofpoint TAP permitted-click ingestion-time EmailEvent parser #15501 is not evidence of success.
  • General README dotnet test KQL/YAML checks were not run; the documented ITP-specific static validator, fixture preparation, and complete ingestion-time unit suite were used instead.

Aditi Bhoge and others added 5 commits October 30, 2026 14:44
Use a positive last-component array index and anchored extraction while preserving all other mappings and the original fixture.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace only the id column_ifexists fallback, retaining the already-direct ingestion-time mapping and all non-ID behavior. Document expected replay QTP failure when fixture tooling omits id.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@contentautomationbot

Copy link
Copy Markdown

ASIM parsers have been changed. ARM templates were regenerated from the updated KQL function YAML files.
To find the new ARM templates, pull your branch.

Record the current PR base ancestry without importing content changes and restore query-time changelogs to the base. Preserve only the ingestion-time relay-host correction and matching changelog.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adbhogeGit adbhogeGit closed this Oct 31, 2026
@v-atulyadav v-atulyadav self-assigned this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants