You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Discussion] Adding energy consumption metrics to MLPerf Inference Benchmark #2558
MLPerf Inference currently reports throughput and latency metrics. As AI sustainability becomes a key concern, standardized energy efficiency metrics would complement existing benchmarks.
Observation
Through systematic benchmarking of quantized LLM inference (NF4, INT8, FP16) across NVIDIA Ada Lovelace and Blackwell architectures, we found that:
Quantization's energy impact is non-trivial and model-size dependent
For models <3B parameters, NF4 quantization increases energy by 25-56%
INT8 mixed-precision adds 17-33% energy overhead vs FP16
These trade-offs are not captured by throughput/latency alone
Suggestion
Consider adding optional energy reporting to the MLPerf Inference benchmark:
Energy per query/token (J)
Average power draw (W)
Energy efficiency (tokens/J)
This would enable apples-to-apples energy comparison across hardware and quantization configurations.
Data
(links updated to the current dataset family, Oct 2026)
Main benchmark dataset v1.1.0 — 360+ configurations, 4 GPU architectures, CC BY 4.0: DOI 10.5281/zenodo.19647290
This looks like something we may be interested in, but in order to understand if this is worth considering for the wider benchmark I'd suggest you attend the MLC Power working group meeting to determine best-fit.
Feel free to join there to discuss direction and figure out potential next-steps.
Thank you very much for the encouraging response and for pointing me toward the MLC Power Working Group — that sounds like exactly the right forum for this discussion.
I should mention that I am currently managing a health condition that limits my ability to participate in live meetings for the time being. I very much hope this will be temporary, and I look forward to joining the Working Group sessions as soon as I am able. In the meantime, I would be grateful if we could begin the conversation asynchronously — I am happy to provide any written materials, technical summaries, or data the group might find useful for an initial evaluation.
To give a brief overview: our study identifies a crossover effect in quantization energy efficiency — quantization actually increases energy consumption for models below a critical size threshold (~3.2–3.9B parameters for NF4, ~4.0–4.6B for INT8), while delivering 15–23% savings above it. We validated this across three GPU architectures (A800, RTX 4090D, RTX 5090) using NVML-based power sampling at 10 Hz, covering 270 configurations.
Permanent archive: Zenodo (360 configurations including FP8 reserved for future work)
I would be happy to prepare a written brief or slide deck that could be shared with the Working Group on my behalf, if that would help move the discussion forward. Please do let me know how I can best contribute given the circumstances.
Thank you again for the warm welcome — I look forward to collaborating with the community.
The Q&A document specifically addresses integration with the existing MLPerf Power infrastructure (PTDaemon compatibility, output format alignment with the power-dev repository), and positions our work as a complementary data contribution — not a replacement for the current physical analyzer standard.
Could you forward these to the Power WG chairs (power-chairs@mlcommons.org), or let me know the best asynchronous channel (mailing list, Discord) to share them directly?
The energy overhead observed with NF4 and INT8 quantization on smaller models (<3B params) likely stems from increased memory access patterns and dequantization kernel overhead that don't amortize well at smaller model scales — the compute savings from reduced precision are outweighed by the additional memory transactions per inference pass. The current MLPerf Inference logging infrastructure in mlperf_loadgen captures latency and query metadata but has no hooks for power sampling, so any integration would need an external power measurement interface (e.g., NVML or RAPL) synchronized with the loadgen query boundaries to get accurate per-query energy attribution. The Zenodo dataset and the HuggingFace Optimum PR (#2410) already provide a reasonable methodology baseline that could inform what the schema for optional energy fields in the results JSON should look like.
Hi @JiwaniZakir,
Thank you for the detailed technical analysis — your points are well taken.
On the dequantization overhead mechanism: Your diagnosis aligns precisely
with our measurements. On the RTX 5090 (Blackwell), for example, Llama 3.2
1B under NF4 shows a 47% energy *increase* over FP16, while Llama 3.1 8B
under the same quantization shows a 19% energy *decrease*. The crossover
threshold shifts with GPU architecture — we observe it at ~3.4B params for
NF4 and ~1.9B for INT8 across our tested hardware.
On loadgen synchronization: Our current NVML-based profiler samples at 10
Hz with a 30-second thermal stabilization window, achieving SNR of
12:1–28:1 (CV < 2%). We have not yet implemented direct synchronization
with mlperf_loadgen query boundaries, but the NVML polling loop can be
aligned to loadgen's query start/complete timestamps via the existing
mlperf_log_detail entries. We would be glad to prototype this integration
if the WG considers it a viable path.
On the energy fields schema: We agree that an optional schema extension is
the right approach. Based on the structure of our Zenodo dataset (360+
configurations across 5 GPU generations), we could draft a minimal schema
proposal covering:
- energy_per_query_joules (mean ± std)
- gpu_power_draw_watts (mean, peak)
- measurement_method (e.g., nvml, rapl, external_analyzer)
- sampling_rate_hz
- thermal_stabilization_seconds
This would be additive — fully optional fields that do not affect existing
result validation.
Next step question: Would it be helpful for us to put together a short RFC
or draft PR sketching this schema, or is there a preferred process within
the Power WG for proposing measurement extensions?
Best, Hongping
Zakir Jiwani ***@***.***> 于2026年4月6日周一 00:30写道:
*JiwaniZakir* left a comment (mlcommons/inference#2558)
<#2558 (comment)>
The energy overhead observed with NF4 and INT8 quantization on smaller
models (<3B params) likely stems from increased memory access patterns and
dequantization kernel overhead that don't amortize well at smaller model
scales — the compute savings from reduced precision are outweighed by the
additional memory transactions per inference pass. The current MLPerf
Inference logging infrastructure in mlperf_loadgen captures latency and
query metadata but has no hooks for power sampling, so any integration
would need an external power measurement interface (e.g., NVML or RAPL)
synchronized with the loadgen query boundaries to get accurate per-query
energy attribution. The Zenodo dataset and the HuggingFace Optimum PR (
#2410 <#2410>) already provide
a reasonable methodology baseline that could inform what the schema for
optional energy fields in the results JSON should look like.
—
Reply to this email directly, view it on GitHub
<#2558 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A7YICI6JA3MVSRBXAVGNDV34UKC3LAVCNFSM6AAAAACWLPTMP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOBZGE2TCMBXGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
The energy overhead for sub-3B models with NF4 is counterintuitive but aligns with what we see in practice — dequantization kernel overhead dominates at smaller model sizes where memory bandwidth savings are insufficient to compensate. For the proposed metrics, tracking energy per token rather than per query would be more meaningful for variable-length workloads like LLM inference. The submission checker PR should also validate that power measurement intervals align with the LoadGen timing boundaries to avoid off-by-one accounting in reported J/token figures.
Energy per token vs. per query: Agreed. For variable-length LLM workloads, J/token is the right normalization. We can compute this directly from our existing measurements: total energy over the generation phase divided by output token count. Updated schema proposal:
energy_per_token_joules (mean ± std across runs)
total_energy_joules (full inference pass, for cross-validation)
gpu_power_draw_watts (mean, peak)
output_token_count
measurement_method (nvml | rapl | external_analyzer)
sampling_rate_hz
thermal_stabilization_seconds
On LoadGen timing alignment: Understood — the power measurement window must match LoadGen's query start/complete boundaries exactly to avoid J/token accounting errors. Our approach would be:
Subscribe to LoadGen's QuerySamplesComplete callback to mark measurement boundaries
Align the NVML polling window to these timestamps (post-hoc trimming of samples outside the boundary)
The submission checker can validate that power_measurement_start and power_measurement_end fall within the LoadGen-reported query_start / query_complete interval, rejecting submissions with misaligned windows
We're ready to put this into a draft PR if that would be a useful next step. Would the right target repo be mlcommons/inference (loadgen side) or mlcommons/power-dev (power measurement side), or both?
The proposed metrics (J/query, W average, tokens/J) are a reasonable starting point, but energy-per-token should be normalized against a reference workload to account for variance in batch sizes and sequence lengths across submissions. Without that normalization, comparisons across hardware will be misleading since power draw is heavily influenced by utilization patterns. It might also be worth considering whether this should integrate with existing power measurement tools like NVIDIA's NVML or AMD's ROCm SMI to reduce implementation burden on submitters.
On normalization: Agreed — raw J/token without controlling for batch size and sequence length would produce misleading cross-hardware comparisons. Our proposed approach:
Energy measurements inherit the workload parameters already defined by each MLPerf scenario (Offline / Server / SingleStream), so J/token values are comparable within the same scenario
The schema explicitly records the workload context to make normalization transparent:
text
energy_per_token_joules (mean ± std)
total_energy_joules
gpu_power_draw_watts (mean, peak)
output_token_count
batch_size
input_sequence_length
output_sequence_length
scenario (offline | server | singlestream)
measurement_method (nvml | rocm_smi | rapl | external_analyzer)
sampling_rate_hz
thermal_stabilization_seconds
This way, reviewers can filter and compare J/token across submissions under identical workload conditions, and flag outliers where utilization patterns diverge.
On tool integration: Good call on ROCm SMI — we've added it to the measurement_method enum. To reduce implementation burden on submitters, we can provide a lightweight wrapper script that abstracts the sampling interface across NVML and ROCm SMI, exposing a unified start_measurement() / stop_measurement() API. Submitters would only need to call two functions regardless of their GPU vendor. We have a working NVML implementation that can serve as the reference; extending it to ROCm SMI should be straightforward since both expose similar polling interfaces.
The energy-per-token metric would need careful normalization to account for batch size variance across submissions, otherwise comparisons between datacenter and edge configurations will be misleading. One practical starting point would be piggybacking on DCGM or RAPL interfaces for power sampling, which are already available on most target hardware — that reduces the instrumentation burden for submitters significantly. The calibration dataset mention from @cfRod is relevant here too, since workload consistency is a prerequisite for any energy comparison to be meaningful.
On normalization: Fully aligned — our previous reply proposed recording batch_size, input_sequence_length, output_sequence_length, and scenario (Offline / Server / SingleStream) alongside J/token, so that comparisons are scoped to identical workload conditions. This should address the datacenter vs. edge variance you raised.
On DCGM: Good suggestion. DCGM provides persistent sampling with lower integration overhead than raw NVML polling, which is a better fit for datacenter submitters. We've added it to the measurement method enum:
text
measurement_method: nvml | dcgm | rocm_smi | rapl | external_analyzer
Our wrapper script can abstract across all four interfaces with the same start_measurement() / stop_measurement() API.
On @cfRod's calibration dataset: Agreed that workload consistency is a prerequisite for meaningful energy comparisons. Could you point me to @cfRod's calibration dataset discussion? We want to make sure our schema and measurement protocol align with that work rather than duplicating or conflicting with it.
The RTX 5090 result for Llama 3.2 1B aligns with what we saw on Ada Lovelace — the dequantization penalty scales inversely with model size, which suggests the overhead is dominated by memory bandwidth pressure rather than compute. For a standardized reporting format, it would be worth separating static power baseline from inference-active power so the per-query joule figures are hardware-comparable across TDP classes. Happy to share the measurement scripts and post-processing tooling we used if it helps bootstrap a reference implementation for the optional energy reporting fields.
On static vs. active power separation: Strong point. Bundling idle baseline into J/token inflates the metric for low-TDP edge devices relative to high-TDP datacenter GPUs. Updated schema adds an explicit baseline field:
text
static_power_baseline_watts (idle GPU, no inference load)
inference_active_power_watts (mean, peak — during generation)
energy_per_token_joules (computed from active power only)
Submitters would record a short idle-phase measurement before the inference run, and the submission checker can validate that energy_per_token is derived from the active delta rather than total system draw.
On sharing measurement scripts: That would be very valuable. We have a working NVML-based profiler (10 Hz sampling, thermal stabilization, LoadGen timestamp alignment) that could serve as one half of a reference implementation. If you share your scripts and post-processing tooling, we can merge both into a unified toolkit covering the full pipeline: sampling → LoadGen boundary alignment → static/active separation → schema-compliant output. Happy to set up a shared branch or repo for this — whatever works best on your end.
The J/token normalization approach makes sense for variable-length workloads, though we should be careful to scope it strictly to the generation phase — prefill energy can dominate for long-context inputs and would skew the metric if included. On the dequantization overhead: the 47% figure on Blackwell for Llama 3.2 1B is consistent with what we're seeing, and it suggests the overhead is architecture-agnostic at small model sizes, which strengthens the case for making this a standardized reporting dimension rather than a one-off observation. For the benchmark integration, would it make sense to propose J/token as a required field only for LLM tasks, with J/query retained for CV workloads where output length is fixed?
On prefill vs. generation scoping: Important distinction. For long-context inputs, prefill can dominate total energy and would inflate J/token if included. Updated approach:
energy_per_token_joules (computed from generation_energy_joules / output_token_count only)
The phase boundary can be detected via the first-token timestamp in LoadGen logs (TTFT marks the prefill→generation transition).
On architecture-agnostic overhead: This is consistent across our dataset as well — the dequantization penalty at sub-3B scales holds across all 5 GPU generations we tested (Turing through Blackwell). The cross-architecture consistency supports treating this as a standardized reporting dimension rather than hardware-specific errata.
On task-type differentiation: Agreed — clean separation:
Task type
Primary energy metric
Rationale
LLM (variable output)
energy_per_token_joules (required)
Output length varies
CV / fixed-output
energy_per_query_joules (required)
Output size is deterministic
We can drive this with a task_type field (llm | cv | other) that determines which energy metric the submission checker validates as required vs. optional.
Following up on our six rounds of technical exchange — thank you again for the thorough and substantive engagement. I wanted to briefly summarize where we've landed and propose a concrete path forward.
What our exchange has established:
The quantization-energy crossover effect is methodologically sound under scrutiny (static power cancellation, SNR analysis, thermal protocol, framework rationale)
GPU-level NVML measurement complements MLPerf system-level power rather than competing with it
Our dataset covers the 1B–5B parameter range currently absent from MLPerf Inference benchmarks — precisely where the crossover occurs
Concrete proposal for WG consideration:
I would like to formally propose one of the following contributions, whichever the WG finds most actionable:
Near-term: Contribute our open dataset (Zenodo DOI: 10.5281/zenodo.19647290) as reference data for a potential MLPerf quantization energy extension
Medium-term: Submit a Technical Note proposing energy-per-token as a supplementary metric in MLPerf Inference, alongside the existing power-constrained performance metric
Longer-term: Co-develop a new MLPerf scenario covering 1B–5B models with dequantization-on-the-fly evaluation
What I need from the group:
Does the WG see value in pursuing any of the above?
If so, what is the appropriate submission channel — a formal RFC, a Working Group meeting slot, or an async written proposal?
As noted previously, I am currently managing a health condition that limits live meeting participation. I remain fully available for asynchronous collaboration and can prepare any written briefs, data summaries, or slide materials the group requires.
Hi @JiwaniZakir (and everyone following this thread),
Quick parallel update: I've completed a full systematic write-up of the quantization-energy crossover work that has anchored much of our discussion here. The paper, When Does Quantization Save Energy? Empirical Analysis of the Energy-Efficiency Crossover Effect Across GPU Generations, covers:
the 270-configuration measurement campaign (3 GPUs × 6 model families × 3 precisions × 5 batch sizes, 1.1B–9B parameters),
a closed-form crossover threshold $N^*$ tied to memory bandwidth and dequantization overhead (Eqs. 1–5), and
the Tesla T4 supplementary measurement of Qwen2.5-3B, which confirms the below-crossover regime in the 2–5B model-size gap.
I've added an acknowledgment recognizing your independent cross-architecture validation on Ada Lovelace — that exchange genuinely shaped my confidence in the architecture-agnostic framing, and I wanted to make sure it's reflected in the record.
If at some point you have time to skim and want to share any high-level reaction — framing, methodology caveats, or anything that strikes you — I'd genuinely value it, but absolutely no urgency on my side. The schema work remains the priority for me, and I'll continue waiting on the prefill/generation separation thread whenever you're ready to pick back up.
I've been working on energy measurements for quantized LLM inference
and wanted to share an observation that might be relevant to the WG's
discussion on benchmark design.
We found that weight-only quantization (NF4/INT8 via bitsandbytes)
does not always reduce energy. On RTX 4090D, models below ~3.9B
parameters incur +25-35% energy overhead, while models above ~6B
achieve -23% savings. The crossover point varies by GPU architecture.
This suggests that energy benchmarks may need to account for the
interaction between precision format and model size, rather than
treating quantization as uniformly energy-saving.
Interactive demo: https://hongping-zh.github.io/quant-energy/
Dataset (CC BY 4.0): zenodo.org/records/19647290
Preprint: papers.ssrn.com/sol3/papers.cfm?abstract_id=6854700
I'm interested in how the Power WG handles precision-format variability
in benchmark design. Are there existing guidelines, or is this something
the WG is still developing?
A brief follow-up on the measurement side of this thread, since the June update.
Since then, the measurement procedure behind those crossover numbers has been packaged so it no longer requires any of my scripts or setup: a pinned Docker image (ghcr.io/hongping-zh/ecocompute-mlcube, SBOM and schema-verified, built by CI) produces a schema-validated energy.json from one command, and a browser-side validator (13 checks) plots submitted reports against the published curves — https://quantenergy.tech/replications/.
Why I'm bringing this here: the weakest point of the data that has informed this discussion is that nearly all of it comes from one measurer. For the WG's benchmark-design question, more independent points — especially on hardware the dataset doesn't cover — would make the crossover evidence considerably harder to dismiss. The biggest gaps right now:
H100 / H200 (Hopper) — no measured data at all; current Hopper curves are borrowed from Ampere and labelled estimated
A100 / RTX 3090 (Ampere) and L40S (Ada) — no coverage
Consumer Blackwell (RTX 5080/5070), and independent RTX 4090 / 5090 replications of the crossover itself
If anyone here has such a card: one command, about an hour including downloads (or the free-T4 Colab notebook for a zero-install start — the T4 is already covered in the main dataset, so a Colab run is directly comparable). Reports are validated and credited publicly on the replications page; everything is CC BY 4.0 and folded back into the dataset archive with attribution.
Happy to answer questions about the protocol (versioned: EcoCompute Protocol v1.1, DOI 10.5281/zenodo.22958675) or to adapt the container if something in your environment doesn't fit.
Congrats to the WG on Endpoints v1.0 — and on making power reporting mandatory, a first for MLPerf.
One observation from the measured-energy side, relevant to how those new power numbers will be read: Endpoints normalizes by provisioned power (rated kW), which is the right procurement-side proxy. But measured task-level energy can diverge from the rating plate in ways buyers can't see: on the same GPU at the same rated power, our NVML-sampled crossover data shows energy per token ranging from −28% to +46% depending on model size and quantization format, with the sign flipping around 3.7B parameters on Ada. None of that variation is visible in a provisioned-kW denominator.
That's precisely the gap the containerized protocol fills — measured energy.json under a versioned protocol, one command. If anyone in the WG wants measured ground truth to sit alongside the provisioned-power metric — especially on H100/H200, where we currently have zero measured points — happy to help set it up.
Hi all — an update from the measurement side that bears on the schema discussion upthread (April–June).
Since the June note, we completed a cross-restart deep dive on RTX 5090 covering four precisions (FP16 / INT8 / NF4 / FP8), published as a standalone record (DOI 10.5281/zenodo.22855133). Two findings from it that I think matter for benchmark design, beyond the crossover itself:
1. The crossover is a property of the card + software stack, not of the silicon. On the same RTX 5090, upgrading bitsandbytes (0.45 → 0.50.2) moved the NF4 crossover from ~4.8B to ~1.1–1.5B parameters — a displacement roughly 20× the spread we observe between cross-restart A/B sessions (same card, full reboot between rounds, max 5.2 pp on relative energy). For the schema: energy submissions need to pin the full software stack (driver + framework + quantization library versions), otherwise J/token figures are not comparable across submission rounds even on identical hardware.
2. Same bit width, different code paths, opposite energy scaling. Characterizing both FP8 paths in torchao 0.18 (stable): weight-only FP8's energy penalty rises with model size (+187% → +816%, 1.1B → 7B), while dynamic-activation FP8 falls (+323% → +82%). The anomaly itself was independently confirmed by the torchao maintainers (pytorch/ao#4094, closed with a reproduced conclusion). For the schema: the quantization code path should be recorded, not just the precision format — "FP8" alone does not predict the energy sign.
Concretely, both findings argue for two additions to the schema sketch from April:
Main dataset (360+ configurations, 4 GPU architectures, n=2, CV < 2%): DOI 10.5281/zenodo.19647290. All of the above is reproducible with the pinned MLCube container (ghcr.io/hongping-zh/ecocompute-mlcube, one command, schema-validated energy.json): hongping-zh/ecocompute-mlcube.
If the WG wants to pick the schema thread back up, I'm happy to draft the two-field extension as a short RFC.
Hi all — a third comparability variable for the schema thread, following the software-stack and code-path findings in the Nov 2 update: this one is thermal.
3. Thermal state alone moves J/token by 4%+ — same card, same stack, same model.
We ran a controlled cold-vs-steady experiment on a single RTX 4090: three cold arms (warm-up 0, GPU cooled back to the session baseline of 37–39 °C between arms) and three steady arms (120 warm-up iterations, steady plateau 53–54 °C confirmed by an external 5-second nvidia-smi log). Same model (TinyLlama-1.1B, NF4, batch 1, greedy decoding, 10 × 256-token generations per run), same software stack throughout, direct NVML sampling at 10 Hz with zero dropped samples:
Cold: 2033.8 ± 25.2 mJ/token (within-arm CV 1.2%)
Steady: 2116.3 ± 33.0 mJ/token (CV 1.6%) → +4.1%
A discarded arm that launched at an intermediate 47 °C (insufficient cooldown — excluded from the accepted set, but documented) landed in between the two groups, consistent with a monotonic temperature–energy gradient
So two submissions differing only in undisclosed thermal state differ by more than many of the effects the WG would want this metric to resolve. For the schema sketch, this argues for:
thermal_state
mode: cold | steady
temperature_start_c / temperature_end_c
temperature_steady_c # or an explicit steady_state_reached: false
warmup_iterations # 0 must be a legal, labelled value
cooldown_s
The cold-start case is structurally unmeasurable under warm-up mandates. When we validated our six runs against our own protocol (EcoCompute Protocol v1.1, DOI 10.5281/zenodo.22958675), the steady arms pass fully conformant — but each cold arm retains exactly one violation: "warm-up missing or zero." Our validator is not wrong; it encodes an ecosystem-wide assumption that measurement follows warm-up. That assumption leaves no path for the serverless / first-query scenario, where the cold-start premium is the quantity of interest. A schema that mandates warm-up cannot express that regime; warmup_iterations: 0 with mode: cold labels it honestly instead of excluding it.
Dataset — all six arms with raw power traces, the full external temperature log, and the tool-original reports alongside the final ones: DOI 10.5281/zenodo.24046432 (CC BY 4.0). Two caveats are recorded in the dataset description rather than hidden: the runs used bitsandbytes 0.50.2 instead of the pinned 0.43.3 (CUDA 13 kernel availability; every report carries a differs_from_reference_pins flag), and the thermal metadata fields our current tool emits as null were backfilled post hoc from the external log, with the unmodified originals preserved in the archive.
That completes the third leg of the comparability argument: stack_versions (Nov 2), quantization_code_path (Nov 2), thermal_state (this note). All three fields are already specified in the EcoCompute Protocol and carried in the dataset records, so they're ready to pick up whenever the WG wants to return to the schema thread.
Volta is now the fifth GPU architecture class with schema-validated energy reports (Turing, Ampere, Ada, Blackwell, Volta) and the first one measured entirely outside the maintainers' own hardware. Each report validates with zero violations against the v1.1-core profile of the energy-report schema, with the MLCommons energy reporting declarations produced natively by the container rather than backfilled. All three model sizes save energy vs FP16, and community replications now span two architecture classes (Ampere, Volta).
Activity
arav-agarwal2 commentedon Mar 30, 2026
Hello!
This looks like something we may be interested in, but in order to understand if this is worth considering for the wider benchmark I'd suggest you attend the MLC Power working group meeting to determine best-fit.
Feel free to join there to discuss direction and figure out potential next-steps.
hongping-zh commentedon Mar 31, 2026
Hi @arav-agarwal2,
Thank you very much for the encouraging response and for pointing me toward the MLC Power Working Group — that sounds like exactly the right forum for this discussion.
I should mention that I am currently managing a health condition that limits my ability to participate in live meetings for the time being. I very much hope this will be temporary, and I look forward to joining the Working Group sessions as soon as I am able. In the meantime, I would be grateful if we could begin the conversation asynchronously — I am happy to provide any written materials, technical summaries, or data the group might find useful for an initial evaluation.
To give a brief overview: our study identifies a crossover effect in quantization energy efficiency — quantization actually increases energy consumption for models below a critical size threshold (~3.2–3.9B parameters for NF4, ~4.0–4.6B for INT8), while delivering 15–23% savings above it. We validated this across three GPU architectures (A800, RTX 4090D, RTX 5090) using NVML-based power sampling at 10 Hz, covering 270 configurations.
All data and tools are openly available:
I would be happy to prepare a written brief or slide deck that could be shared with the Working Group on my behalf, if that would help move the discussion forward. Please do let me know how I can best contribute given the circumstances.
Thank you again for the warm welcome — I look forward to collaborating with the community.
Best regards,
Hongping Zhang
EcoCompute-AI Research
zhanghongping1982@gmail.com
hongping-zh commentedon Apr 1, 2026
Hi @arav-agarwal2,
Following up — I've prepared two documents for the Power Working Group's review:
Executive Summary (2 pages): Executive_Summary__The_Quantization_Energy_Crossover_Effect.pdf
Technical Q&A (10 pages): MLC_Power_WG_Technical_QA_with_References.pdf
The Q&A document specifically addresses integration with the existing MLPerf Power infrastructure (PTDaemon compatibility, output format alignment with the power-dev repository), and positions our work as a complementary data contribution — not a replacement for the current physical analyzer standard.
Could you forward these to the Power WG chairs (power-chairs@mlcommons.org), or let me know the best asynchronous channel (mailing list, Discord) to share them directly?
Thanks,
Hongping
JiwaniZakir commentedon Apr 5, 2026
The energy overhead observed with NF4 and INT8 quantization on smaller models (<3B params) likely stems from increased memory access patterns and dequantization kernel overhead that don't amortize well at smaller model scales — the compute savings from reduced precision are outweighed by the additional memory transactions per inference pass. The current MLPerf Inference logging infrastructure in
mlperf_loadgencaptures latency and query metadata but has no hooks for power sampling, so any integration would need an external power measurement interface (e.g., NVML or RAPL) synchronized with the loadgen query boundaries to get accurate per-query energy attribution. The Zenodo dataset and the HuggingFace Optimum PR (#2410) already provide a reasonable methodology baseline that could inform what the schema for optional energy fields in the results JSON should look like.hongping-zh commentedon Apr 6, 2026
JiwaniZakir commentedon Apr 6, 2026
The energy overhead for sub-3B models with NF4 is counterintuitive but aligns with what we see in practice — dequantization kernel overhead dominates at smaller model sizes where memory bandwidth savings are insufficient to compensate. For the proposed metrics, tracking energy per token rather than per query would be more meaningful for variable-length workloads like LLM inference. The submission checker PR should also validate that power measurement intervals align with the LoadGen timing boundaries to avoid off-by-one accounting in reported J/token figures.
hongping-zh commentedon Apr 6, 2026
Hi @JiwaniZakir,
Great points — both well taken.
Energy per token vs. per query: Agreed. For variable-length LLM workloads, J/token is the right normalization. We can compute this directly from our existing measurements: total energy over the generation phase divided by output token count. Updated schema proposal:
energy_per_token_joules (mean ± std across runs)
total_energy_joules (full inference pass, for cross-validation)
gpu_power_draw_watts (mean, peak)
output_token_count
measurement_method (nvml | rapl | external_analyzer)
sampling_rate_hz
thermal_stabilization_seconds
On LoadGen timing alignment: Understood — the power measurement window must match LoadGen's query start/complete boundaries exactly to avoid J/token accounting errors. Our approach would be:
Subscribe to LoadGen's QuerySamplesComplete callback to mark measurement boundaries
Align the NVML polling window to these timestamps (post-hoc trimming of samples outside the boundary)
The submission checker can validate that power_measurement_start and power_measurement_end fall within the LoadGen-reported query_start / query_complete interval, rejecting submissions with misaligned windows
We're ready to put this into a draft PR if that would be a useful next step. Would the right target repo be mlcommons/inference (loadgen side) or mlcommons/power-dev (power measurement side), or both?
Best, Hongping
JiwaniZakir commentedon Apr 6, 2026
The proposed metrics (J/query, W average, tokens/J) are a reasonable starting point, but energy-per-token should be normalized against a reference workload to account for variance in batch sizes and sequence lengths across submissions. Without that normalization, comparisons across hardware will be misleading since power draw is heavily influenced by utilization patterns. It might also be worth considering whether this should integrate with existing power measurement tools like NVIDIA's NVML or AMD's ROCm SMI to reduce implementation burden on submitters.
hongping-zh commentedon Apr 6, 2026
Hi @JiwaniZakir,
On normalization: Agreed — raw J/token without controlling for batch size and sequence length would produce misleading cross-hardware comparisons. Our proposed approach:
Energy measurements inherit the workload parameters already defined by each MLPerf scenario (Offline / Server / SingleStream), so J/token values are comparable within the same scenario
The schema explicitly records the workload context to make normalization transparent:
text
energy_per_token_joules (mean ± std)
total_energy_joules
gpu_power_draw_watts (mean, peak)
output_token_count
batch_size
input_sequence_length
output_sequence_length
scenario (offline | server | singlestream)
measurement_method (nvml | rocm_smi | rapl | external_analyzer)
sampling_rate_hz
thermal_stabilization_seconds
This way, reviewers can filter and compare J/token across submissions under identical workload conditions, and flag outliers where utilization patterns diverge.
On tool integration: Good call on ROCm SMI — we've added it to the measurement_method enum. To reduce implementation burden on submitters, we can provide a lightweight wrapper script that abstracts the sampling interface across NVML and ROCm SMI, exposing a unified start_measurement() / stop_measurement() API. Submitters would only need to call two functions regardless of their GPU vendor. We have a working NVML implementation that can serve as the reference; extending it to ROCm SMI should be straightforward since both expose similar polling interfaces.
Best, Hongping
JiwaniZakir commentedon Apr 6, 2026
The energy-per-token metric would need careful normalization to account for batch size variance across submissions, otherwise comparisons between datacenter and edge configurations will be misleading. One practical starting point would be piggybacking on DCGM or RAPL interfaces for power sampling, which are already available on most target hardware — that reduces the instrumentation burden for submitters significantly. The calibration dataset mention from @cfRod is relevant here too, since workload consistency is a prerequisite for any energy comparison to be meaningful.
hongping-zh commentedon Apr 6, 2026
Hi @JiwaniZakir,
On normalization: Fully aligned — our previous reply proposed recording batch_size, input_sequence_length, output_sequence_length, and scenario (Offline / Server / SingleStream) alongside J/token, so that comparisons are scoped to identical workload conditions. This should address the datacenter vs. edge variance you raised.
On DCGM: Good suggestion. DCGM provides persistent sampling with lower integration overhead than raw NVML polling, which is a better fit for datacenter submitters. We've added it to the measurement method enum:
text
measurement_method: nvml | dcgm | rocm_smi | rapl | external_analyzer
Our wrapper script can abstract across all four interfaces with the same start_measurement() / stop_measurement() API.
On @cfRod's calibration dataset: Agreed that workload consistency is a prerequisite for meaningful energy comparisons. Could you point me to @cfRod's calibration dataset discussion? We want to make sure our schema and measurement protocol align with that work rather than duplicating or conflicting with it.
Best, Hongping
JiwaniZakir commentedon Apr 6, 2026
The RTX 5090 result for Llama 3.2 1B aligns with what we saw on Ada Lovelace — the dequantization penalty scales inversely with model size, which suggests the overhead is dominated by memory bandwidth pressure rather than compute. For a standardized reporting format, it would be worth separating static power baseline from inference-active power so the per-query joule figures are hardware-comparable across TDP classes. Happy to share the measurement scripts and post-processing tooling we used if it helps bootstrap a reference implementation for the optional energy reporting fields.
hongping-zh commentedon Apr 6, 2026
Hi @JiwaniZakir,
On static vs. active power separation: Strong point. Bundling idle baseline into J/token inflates the metric for low-TDP edge devices relative to high-TDP datacenter GPUs. Updated schema adds an explicit baseline field:
text
static_power_baseline_watts (idle GPU, no inference load)
inference_active_power_watts (mean, peak — during generation)
energy_per_token_joules (computed from active power only)
Submitters would record a short idle-phase measurement before the inference run, and the submission checker can validate that energy_per_token is derived from the active delta rather than total system draw.
On sharing measurement scripts: That would be very valuable. We have a working NVML-based profiler (10 Hz sampling, thermal stabilization, LoadGen timestamp alignment) that could serve as one half of a reference implementation. If you share your scripts and post-processing tooling, we can merge both into a unified toolkit covering the full pipeline: sampling → LoadGen boundary alignment → static/active separation → schema-compliant output. Happy to set up a shared branch or repo for this — whatever works best on your end.
Best, Hongping
JiwaniZakir commentedon Apr 6, 2026
The J/token normalization approach makes sense for variable-length workloads, though we should be careful to scope it strictly to the generation phase — prefill energy can dominate for long-context inputs and would skew the metric if included. On the dequantization overhead: the 47% figure on Blackwell for Llama 3.2 1B is consistent with what we're seeing, and it suggests the overhead is architecture-agnostic at small model sizes, which strengthens the case for making this a standardized reporting dimension rather than a one-off observation. For the benchmark integration, would it make sense to propose J/token as a required field only for LLM tasks, with J/query retained for CV workloads where output length is fixed?
hongping-zh commentedon Apr 7, 2026
Hi @JiwaniZakir,
On prefill vs. generation scoping: Important distinction. For long-context inputs, prefill can dominate total energy and would inflate J/token if included. Updated approach:
prefill_energy_joules(prompt processing phase)generation_energy_joules(autoregressive decoding phase)energy_per_token_joules(computed fromgeneration_energy_joules / output_token_countonly)The phase boundary can be detected via the first-token timestamp in LoadGen logs (TTFT marks the prefill→generation transition).
On architecture-agnostic overhead: This is consistent across our dataset as well — the dequantization penalty at sub-3B scales holds across all 5 GPU generations we tested (Turing through Blackwell). The cross-architecture consistency supports treating this as a standardized reporting dimension rather than hardware-specific errata.
On task-type differentiation: Agreed — clean separation:
energy_per_token_joules(required)energy_per_query_joules(required)We can drive this with a
task_typefield (llm|cv|other) that determines which energy metric the submission checker validates as required vs. optional.Best,
Hongping
hongping-zh commentedon Apr 9, 2026
Hi @arav-agarwal2 and the MLC Power Working Group,
Following up on our six rounds of technical exchange — thank you again for the thorough and substantive engagement. I wanted to briefly summarize where we've landed and propose a concrete path forward.
What our exchange has established:
The quantization-energy crossover effect is methodologically sound under scrutiny (static power cancellation, SNR analysis, thermal protocol, framework rationale)
GPU-level NVML measurement complements MLPerf system-level power rather than competing with it
Our dataset covers the 1B–5B parameter range currently absent from MLPerf Inference benchmarks — precisely where the crossover occurs
Concrete proposal for WG consideration:
I would like to formally propose one of the following contributions, whichever the WG finds most actionable:
Near-term: Contribute our open dataset (Zenodo DOI: 10.5281/zenodo.19647290) as reference data for a potential MLPerf quantization energy extension
Medium-term: Submit a Technical Note proposing energy-per-token as a supplementary metric in MLPerf Inference, alongside the existing power-constrained performance metric
Longer-term: Co-develop a new MLPerf scenario covering 1B–5B models with dequantization-on-the-fly evaluation
What I need from the group:
Does the WG see value in pursuing any of the above?
If so, what is the appropriate submission channel — a formal RFC, a Working Group meeting slot, or an async written proposal?
As noted previously, I am currently managing a health condition that limits live meeting participation. I remain fully available for asynchronous collaboration and can prepare any written briefs, data summaries, or slide materials the group requires.
Thank you for your continued consideration.
Best regards, Hongping
hongping-zh commentedon Apr 29, 2026
Hi @JiwaniZakir (and everyone following this thread),
Quick parallel update: I've completed a full systematic write-up of the quantization-energy crossover work that has anchored much of our discussion here. The paper, When Does Quantization Save Energy? Empirical Analysis of the Energy-Efficiency Crossover Effect Across GPU Generations, covers:
I've added an acknowledgment recognizing your independent cross-architecture validation on Ada Lovelace — that exchange genuinely shaped my confidence in the architecture-agnostic framing, and I wanted to make sure it's reflected in the record.
Resources:
If at some point you have time to skim and want to share any high-level reaction — framing, methodology caveats, or anything that strikes you — I'd genuinely value it, but absolutely no urgency on my side. The schema work remains the priority for me, and I'll continue waiting on the prefill/generation separation thread whenever you're ready to pick back up.
Best,
Hongping
hongping-zh commentedon Jun 26, 2026
Hi all,
I've been working on energy measurements for quantized LLM inference
and wanted to share an observation that might be relevant to the WG's
discussion on benchmark design.
We found that weight-only quantization (NF4/INT8 via bitsandbytes)
does not always reduce energy. On RTX 4090D, models below ~3.9B
parameters incur +25-35% energy overhead, while models above ~6B
achieve -23% savings. The crossover point varies by GPU architecture.
This suggests that energy benchmarks may need to account for the
interaction between precision format and model size, rather than
treating quantization as uniformly energy-saving.
Interactive demo: https://hongping-zh.github.io/quant-energy/
Dataset (CC BY 4.0): zenodo.org/records/19647290
Preprint: papers.ssrn.com/sol3/papers.cfm?abstract_id=6854700
I'm interested in how the Power WG handles precision-format variability
in benchmark design. Are there existing guidelines, or is this something
the WG is still developing?
Best,
Hongping Zhang
hongping-zh commentedon Oct 14, 2026
Hi @arav-agarwal2 and the Power WG,
A brief follow-up on the measurement side of this thread, since the June update.
Since then, the measurement procedure behind those crossover numbers has been packaged so it no longer requires any of my scripts or setup: a pinned Docker image (
ghcr.io/hongping-zh/ecocompute-mlcube, SBOM and schema-verified, built by CI) produces a schema-validatedenergy.jsonfrom one command, and a browser-side validator (13 checks) plots submitted reports against the published curves — https://quantenergy.tech/replications/.Why I'm bringing this here: the weakest point of the data that has informed this discussion is that nearly all of it comes from one measurer. For the WG's benchmark-design question, more independent points — especially on hardware the dataset doesn't cover — would make the crossover evidence considerably harder to dismiss. The biggest gaps right now:
estimatedIf anyone here has such a card: one command, about an hour including downloads (or the free-T4 Colab notebook for a zero-install start — the T4 is already covered in the main dataset, so a Colab run is directly comparable). Reports are validated and credited publicly on the replications page; everything is CC BY 4.0 and folded back into the dataset archive with attribution.
Happy to answer questions about the protocol (versioned: EcoCompute Protocol v1.1, DOI 10.5281/zenodo.22958675) or to adapt the container if something in your environment doesn't fit.
hongping-zh commentedon Oct 14, 2026
Congrats to the WG on Endpoints v1.0 — and on making power reporting mandatory, a first for MLPerf.
One observation from the measured-energy side, relevant to how those new power numbers will be read: Endpoints normalizes by provisioned power (rated kW), which is the right procurement-side proxy. But measured task-level energy can diverge from the rating plate in ways buyers can't see: on the same GPU at the same rated power, our NVML-sampled crossover data shows energy per token ranging from −28% to +46% depending on model size and quantization format, with the sign flipping around 3.7B parameters on Ada. None of that variation is visible in a provisioned-kW denominator.
That's precisely the gap the containerized protocol fills — measured
energy.jsonunder a versioned protocol, one command. If anyone in the WG wants measured ground truth to sit alongside the provisioned-power metric — especially on H100/H200, where we currently have zero measured points — happy to help set it up.hongping-zh commentedon Nov 2, 2026
Hi all — an update from the measurement side that bears on the schema discussion upthread (April–June).
Since the June note, we completed a cross-restart deep dive on RTX 5090 covering four precisions (FP16 / INT8 / NF4 / FP8), published as a standalone record (DOI 10.5281/zenodo.22855133). Two findings from it that I think matter for benchmark design, beyond the crossover itself:
1. The crossover is a property of the card + software stack, not of the silicon. On the same RTX 5090, upgrading bitsandbytes (0.45 → 0.50.2) moved the NF4 crossover from ~4.8B to ~1.1–1.5B parameters — a displacement roughly 20× the spread we observe between cross-restart A/B sessions (same card, full reboot between rounds, max 5.2 pp on relative energy). For the schema: energy submissions need to pin the full software stack (driver + framework + quantization library versions), otherwise J/token figures are not comparable across submission rounds even on identical hardware.
2. Same bit width, different code paths, opposite energy scaling. Characterizing both FP8 paths in torchao 0.18 (stable): weight-only FP8's energy penalty rises with model size (+187% → +816%, 1.1B → 7B), while dynamic-activation FP8 falls (+323% → +82%). The anomaly itself was independently confirmed by the torchao maintainers (pytorch/ao#4094, closed with a reproduced conclusion). For the schema: the quantization code path should be recorded, not just the precision format — "FP8" alone does not predict the energy sign.
Concretely, both findings argue for two additions to the schema sketch from April:
Main dataset (360+ configurations, 4 GPU architectures, n=2, CV < 2%): DOI 10.5281/zenodo.19647290. All of the above is reproducible with the pinned MLCube container (
ghcr.io/hongping-zh/ecocompute-mlcube, one command, schema-validatedenergy.json): hongping-zh/ecocompute-mlcube.If the WG wants to pick the schema thread back up, I'm happy to draft the two-field extension as a short RFC.
hongping-zh commentedon Nov 8, 2026
Hi all — a third comparability variable for the schema thread, following the software-stack and code-path findings in the Nov 2 update: this one is thermal.
3. Thermal state alone moves J/token by 4%+ — same card, same stack, same model.
We ran a controlled cold-vs-steady experiment on a single RTX 4090: three cold arms (warm-up 0, GPU cooled back to the session baseline of 37–39 °C between arms) and three steady arms (120 warm-up iterations, steady plateau 53–54 °C confirmed by an external 5-second nvidia-smi log). Same model (TinyLlama-1.1B, NF4, batch 1, greedy decoding, 10 × 256-token generations per run), same software stack throughout, direct NVML sampling at 10 Hz with zero dropped samples:
So two submissions differing only in undisclosed thermal state differ by more than many of the effects the WG would want this metric to resolve. For the schema sketch, this argues for:
thermal_state
mode: cold | steady
temperature_start_c / temperature_end_c
temperature_steady_c # or an explicit steady_state_reached: false
warmup_iterations # 0 must be a legal, labelled value
cooldown_s
The cold-start case is structurally unmeasurable under warm-up mandates. When we validated our six runs against our own protocol (EcoCompute Protocol v1.1, DOI 10.5281/zenodo.22958675), the steady arms pass fully conformant — but each cold arm retains exactly one violation: "warm-up missing or zero." Our validator is not wrong; it encodes an ecosystem-wide assumption that measurement follows warm-up. That assumption leaves no path for the serverless / first-query scenario, where the cold-start premium is the quantity of interest. A schema that mandates warm-up cannot express that regime; warmup_iterations: 0 with mode: cold labels it honestly instead of excluding it.
Dataset — all six arms with raw power traces, the full external temperature log, and the tool-original reports alongside the final ones: DOI 10.5281/zenodo.24046432 (CC BY 4.0). Two caveats are recorded in the dataset description rather than hidden: the runs used bitsandbytes 0.50.2 instead of the pinned 0.43.3 (CUDA 13 kernel availability; every report carries a differs_from_reference_pins flag), and the thermal metadata fields our current tool emits as null were backfilled post hoc from the external log, with the unmodified originals preserved in the archive.
That completes the third leg of the comparability argument: stack_versions (Nov 2), quantization_code_path (Nov 2), thermal_state (this note). All three fields are already specified in the EcoCompute Protocol and carried in the dataset records, so they're ready to pick up whenever the WG wants to return to the schema thread.
hongping-zh commentedon Nov 10, 2026
Community update (cc @arav-agarwal2): a second independent contributor has run the EcoCompute measurement container and produced the project's first Volta-architecture results — three NF4 runs on a Tesla V100S (1.1B / 3B / 7B: hongping-zh/ecocompute-mlcube#42, hongping-zh/ecocompute-mlcube#43, hongping-zh/ecocompute-mlcube#44).
Volta is now the fifth GPU architecture class with schema-validated energy reports (Turing, Ampere, Ada, Blackwell, Volta) and the first one measured entirely outside the maintainers' own hardware. Each report validates with zero violations against the v1.1-core profile of the energy-report schema, with the MLCommons energy reporting declarations produced natively by the container rather than backfilled. All three model sizes save energy vs FP16, and community replications now span two architecture classes (Ampere, Volta).
Raw reports and the public gallery: https://quantenergy.tech/replications/