[agam-99]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title:
  • https://issues.apache.org/jira/browse/GOBBLIN-2271

Description

• Here are some details about my PR, including screenshots (if applicable):

GaaSJobObservabilityEvent.datasetsMetrics[] (DatasetMetric) only models writes: bytesWritten, entitiesWritten, successfullyCommitted, dataQualityStatus, and, since GOBBLIN-2269, snapshotsCommitted / partitionsCommitted. Data-retention (deletion) jobs have no way to report per-dataset work through this channel. A job that removes aged-out dataset versions can already emit one datasetTaskSummaries entry per dataset root, but the deletion counts are dropped when GaaS parses them into DatasetTaskSummary.

This PR adds three optional, retention-only counts and threads them through. It follows the same pattern as GOBBLIN-2269 (#4206).

• GaaSJobObservabilityEvent.avsc: DatasetMetric gains three fields, each ["null","long"] with default: null:

  • versionsTotal: versions considered under the dataset root (deletable + retained);
  • versionsDeleted: versions deleted, or on a dry run, those that would be;
  • directoriesPruned: now-empty parent directories pruned after the deletes.

  Because they are nullable with null defaults, the schema change is backward compatible, and every existing producer and consumer is unaffected.

• DatasetTaskSummary: adds the three fields as nullable Long, intentionally not @NonNull. That keeps the 5-arg @RequiredArgsConstructor used by AbstractJobLauncher and CommitActivityImpl intact. The fields are populated by Gson during deserialization of events that carry them. toDatasetMetric maps them through, and they stay null for write jobs.

The first producer is the ddm-executor retention run (linkedin-multiproduct/ddm-executor#507). It emits one row per dataset root in a single JobSummaryTimer event, rather than one event per dataset, which would collide on the job-execution identity. Each row carries recordsWritten = bytesWritten = 0 plus these counts.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

New DatasetTaskSummaryTest:

• testToDatasetMetricMapsRetentionCountsFromJson: parses the exact JSON a retention run emits, via GsonUtils.GSON_WITH_DATE_HANDLING (the path GaaSJobObservabilityEventProducer uses), and asserts the counts reach DatasetMetric.

The existing GaaSJobObservabilityProducerTest still passes, so write-job datasetsMetrics are unchanged.

End-to-end check, done locally and not committed:

• A real ddm-executor retention run (ddm-executor PRs Fixed build #506 + Unit tests for Gobblin Yarn part II #507) produced datasetTaskSummaries=[{"datasetUrn":"abfss://…","recordsWritten":0,"bytesWritten":0,"successfullyCommitted":true,"versionsTotal":3,"versionsDeleted":1,"directoriesPruned":1}].
• That captured JSON, fed through GaaSJobObservabilityEventProducer on this branch, yields datasetsMetrics=[{…, versionsTotal:3, versionsDeleted:1, directoriesPruned:1}], and the values survive the AvroBinarySerializer round-trip.
• On master, the same input is accepted and the three fields are dropped, which confirms that this change is what carries them.

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

• Using a long positional constructor call is brittle and easy to break as the Avro schema evolves… · New
• The constant name SUMMARY_LIST is a bit ambiguous since it’s actually a Type token for a list… · New
• This test validates the new retention counters, but it doesn’t assert other key fields present in… · New

What changed in this PR

Adds retention (deletion) counters to per-dataset observability metrics so data-retention jobs can report work (versions considered/deleted and directories pruned) through GaaSJobObservabilityEvent.datasetsMetrics[].

Changes:

• Extend DatasetMetric Avro schema with nullable retention-only fields: versionsTotal, versionsDeleted, directoriesPruned
• Thread the new fields through DatasetTaskSummary.toDatasetMetric(...)
• Add unit tests validating JSON deserialization + mapping behavior for retention vs write jobs

File	Description
gobblin-runtime/​src/​main/​java/​org/​apache/​gobblin/​runtime/​DatasetTaskSummary.java	Maps new retention counters from DatasetTaskSummary into DatasetMetric
gobblin-runtime/​src/​test/​java/​org/​apache/​gobblin/​runtime/​DatasetTaskSummaryTest.java	Adds tests for JSON->DatasetTaskSummary->DatasetMetric mapping and null behavior for write jobs
gobblin-metrics-libs/​gobblin-metrics-base/​src/​main/​avro/​GaaSJobObservabilityEvent.avsc	Adds three backward-compatible nullable fields to DatasetMetric schema

🧠 Review effort: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.