Repository navigation
Fix Proofpoint POD MailLog ingestion-time relay-host normalization parity - #15512
Closed
adbhogeGit wants to merge 6 commits into
Closed
adbhogeGit wants to merge 6 commits into
adbhogeGit wants to merge 6 commits into
Conversation
Use a positive last-component array index and anchored extraction while preserving all other mappings and the original fixture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace only the id column_ifexists fallback, retaining the already-direct ingestion-time mapping and all non-ID behavior. Document expected replay QTP failure when fixture tooling omits id. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…aillog-parity-fix
|
ASIM parsers have been changed. ARM templates were regenerated from the updated KQL function YAML files. |
Record the current PR base ancestry without importing content changes and restore query-time changelogs to the base. Preserve only the ingestion-time relay-host correction and matching changelog. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required items, please complete
Change(s):
728c24c32ec119e898a5400544a6012e6a0a5cae. Base branch:adbhoge-microsoft-proofpoint-maillog-draft-pr, not master.@components, select the last component using the parenthesized positive index(array_length(RelayHostParts) - 1), and remove trailing dots with anchoredextract. Both source and destination mappings consume the same normalized host.0.1.4with its separate PR #15512 changelog. Per the user's follow-up, change both the parameterless and filtering QTP fromtostring(column_ifexists("id", ""))totostring(id), bump each QTP to0.1.3, refresh LastUpdated, and update each matching changelog. The ITP already uses directtostring(id)and is unchanged by the ID follow-up.sm, explicit BooleanIsDelivery, valid source-ID values and EventUid prefixing, TimeGenerated, filter, output shape, IP validation, non-ID optional fields, and all unrelated mappings. These two source-ID fallbacks were the onlycolumn_ifexists/columnifexistsusages in the three MailLog parsers. No fan-out, aggregation, generated ARM, harness, tooling, test implementation, schema, workflow, script, or Tools/ changes.Reason for Change(s):
host [IPv4]inputs and twouser@localhostinputs. Matching relay IPs and delivery-device host fields isolate the failure boundary to compound RelayHost normalization before mapping.@behavior, ordinary FQDNs, absent relays, localhost, multiple trailing dots, and destination/local-delivery fallback match QTP semantics offline.idfrom raw/QTP replay; all 24 captured QTP UIDs are empty while ITP UIDs are populated. Preserve valid IDs; do not duplicate or alter protected harness implementation, clear IDs, or weaken parity.3a124189f45e147facb9f81a3d66ac792ae231da). Its direct source-ID mapping and accepted CI failure do not fix shared replay or demonstrate passing full parity. This PR addresses parser-side host normalization and the user-directed direct source-ID requirement, not the underlying replay defect or all discovered issues.idas string in.script/tests/KqlvalidationsTests/CustomTables/ProofpointPODMailLog_CL.jsonandSample Data/ASIM/ProofpointPODMailLog_CL_Schema.csv. Require that source column rather than silently substitute an empty ID. The fixture preparer still excludesid, including from explicit schemas. Consequently, raw replay without that column is expected to fail at qtpExecution with a missing-iderror, not produce successful parity. The user explicitly accepted this Proofpoint edge-case failure. This is not a harness repair, parity waiver, authorization for Azure access, or proof of runtime success.Version Updated:
0.1.3->0.1.4for host normalization; Parser.LastUpdated and Change.Date remain2026-10-30. Both QTP variants0.1.2->0.1.3, LastUpdatedOct 30, 2026, for direct source-ID access; each has its own version-matched changelog linked to Fix Proofpoint POD MailLog ingestion-time relay-host normalization parity #15512.Testing Completed:
$Pythonbelow resolves to the existing local CPython 3.13 executable;$Artifactsis the external session-artifact directory. Exact command invocations from the repository root and their results:Output path must not replace the source fixturediagnostic belongs to an expected negative test; suite result is OK.Passed before the direct-ID follow-up, at parser head
5fa425176652825771842b7f5cecfd8870dbd5bf: session-only Python semantic checks against the exact captured parser pair, all 24 timestamp-aligned rows, 12 explicit host examples, and 3,456 synthetic mapping combinations (multiple @, empty last component, no relay, hostname/FQDN, trailing dots, IPv4/IPv6/invalid IP, receipt/delivery, local fallback, and device variants). The script asserts the query is unchanged outside relay-host normalization, retains 25/24/1 selection, and checks raw sm preservation. Modeling only the revised host fields leaves exactly the known UID differences; AdditionalFields bags are compared structurally because serialization order differs. This historical model is not a claim that the revised direct-ID QTP executes against the captured replay. These checks are NOT KQL/DCR execution, a substitute for shared canary, or new repository harness tests. Whitespace check passed.Direct-ID follow-up commands and results (same
$Pythonand$Artifacts):50ca9f982e0542af78a1dec33841c4508afc10e8. Static validation valid=true, zero findings, paired QTP eligible. Complete ingestion-time unit suite: 164 tests, OK (8.900s); expected negative-test output-path diagnostic is not a failure. Whitespace passed. These checks do not execute either QTP in a KQL engine or a DCR.Checked that the validations are passing and have addressed any issues that are present:
idis expected to fail qtpExecution instead; this expected CI failure has not been executed or observed for this revision. The user accepts it, but it does not repair fixture replay or validate parity. The workflow targets master only, so this stacked PR may not trigger shared canary. Source-ID replay needs an authorized maintainer tooling fix; closed [ASIM] Preserve source id in ingestion validation fixtures #15510 is not an available dependency. Once that unresolved blocker is fixed on the trusted base and the stacked changes can be tested through the authorized existing-PR GitHub canary, rerun full DCR/filter/target/schema/data/QTP/parity validation before considering runtime verification complete. Accepted CI failure in [ASIM] Add Proofpoint TAP permitted-click ingestion-time EmailEvent parser #15501 is not evidence of success.dotnet testKQL/YAML checks were not run; the documented ITP-specific static validator, fixture preparation, and complete ingestion-time unit suite were used instead.