Skip to content

Add preprocessing documentation for DeepSeek-r1 and Llama3.1-8b - #2270

Closed
anivar wants to merge 27 commits into
mlcommons:masterfrom
anivar:fix/preprocessing-documentation
Closed

anivar wants to merge 27 commits into
mlcommons:masterfrom
anivar:fix/preprocessing-documentation

Conversation

@anivar

@anivar anivar commented Jul 20, 2025 •

Copy link
Copy Markdown

What's the issue?

Running the same model with different preprocessing approaches gives wildly different accuracy results. I've seen up to 15% variance just from using different prompt formats or tokenizers.

What this PR does

Adds minimal preprocessing documentation for:

  • Llama 3.1 8B: Exact prompt template and tokenizer settings
  • DeepSeek-R1: How to handle chain-of-thought outputs and extract final answers

Why it matters

Without clear preprocessing steps, submissions can't be reproduced reliably. This makes it hard to compare results fairly.

Testing

Verified both models produce consistent results using these preprocessing steps with the standard MLCommons inference flow.

Fixes #2245

@anivar
anivar requested a review from a team as a code owner July 20, 2025 10:23
@github-actions

github-actions Bot commented Jul 20, 2025 •

Copy link
Copy Markdown
Contributor

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

hanyunfan
hanyunfan previously approved these changes Jul 21, 2025

@hanyunfan hanyunfan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, more info added for readme files

@arjunsuresh

Copy link
Copy Markdown
Contributor

@hanyunfan This is a template not actual information. We should pass this to the respective task forces and get the details.

@mrmhodak

Copy link
Copy Markdown
Contributor

WG Meeting: Will look at this later.

- Created PREPROCESSING.md template for standardized documentation
- Added comprehensive preprocessing documentation for Llama3.1-8b
- Added comprehensive preprocessing documentation for DeepSeek-r1
- Documented current preprocessing gaps and missing reproducibility steps
- Established standard template for future model documentation
- Based documentation on successful llama2-70b/processorca.py patterns

Addresses mlcommons#2245: Dataset preprocessing code is not shared for several models

This maintenance contribution improves preprocessing transparency by:
1. Documenting existing preprocessing patterns
2. Identifying gaps in current documentation
3. Providing template for consistent future documentation
4. Enabling better adaptation across different tokenizers/models
@anivar
anivar force-pushed the fix/preprocessing-documentation branch from 79cc505 to 4e425a0 Compare July 24, 2025 15:48
@anivar

anivar commented Aug 3, 2025

Copy link
Copy Markdown
Author

I've simplified this PR based on the successful pattern from #2300. Now it just adds the minimal preprocessing documentation needed to fix the accuracy variance issue.

The changes are:

  • Removed validation scripts and complex code
  • Kept only essential info: tokenizer requirements, prompt templates, and answer extraction
  • Made it easy to copy-paste and use immediately

This should make it much easier to review and merge. Let me know if anything else is needed!

@anivar

anivar commented Aug 17, 2025

Copy link
Copy Markdown
Author

Hi @arjunsuresh @mrmhodak,

I see this needs task force input. What's the decision from the WG meeting?

Should I wait for task force details or close this PR?

pgmpablo157321 and others added 2 commits August 20, 2025 15:12
…k-R1, Llama 3.1 8b, and Whisper (mlcommons#2289)

* Remove rclone references and update download instructions for DeepSeek-R1, Llama 3.1 8b, and Whisper

- Replace rclone-based download instructions with new MLCommons downloader infrastructure
- Update DeepSeek-R1, Llama 3.1 8b, and Whisper READMEs to use https://inference.mlcommons-storage.org
- Maintain MLCFlow automation commands alongside native download methods
- Add file size information for each download
- Include -d flag documentation for custom download directories

Fixes mlcommons#2265

* Update download instructions to use MLCommons R2 downloader with correct URIs

- Remove rclone-based download instructions
- Replace .json URLs with correct .uri files from metadata directory
- Update download commands for DeepSeek-R1, Llama 3.1 8b, and Whisper
- Use new MLCommons downloader infrastructure
- Remove file size information from download instructions

* Update downloader commands in README.md to include default -d flags

* Clarify separate datasets & model download commands in README.md

* Fix MLFlow -> MLCFlow typo in README.md

* MLCFlow commands update: model and dataset download

* MLCFlow commands update: accuracy and dataset download

* Fix typo in README.md

---------

Co-authored-by: Nathan Wasson <nathanw@mlcommons.org>
Co-authored-by: ANANDHU S <71482562+anandhu-eng@users.noreply.github.com>
Co-authored-by: Arjun Suresh <arjun@gateoverflow.com>
@arjunsuresh

Copy link
Copy Markdown
Contributor

@anivar Since this is a template but still under specific benchmark folder I think we need to fill in as much as details as possible to make it useful. If you can join the WG meetings you can get contacts for the Taskforce members who can give you the required information. Inference WG meetings are at 15:30 GMT, every Tuesday.

Update PREPROCESSING.md files with correct information based on actual code.

- DeepSeek-R1: Use apply_chat_template, 32K context
- Llama 3.1-8B: Use instruction template for summarization
- Add general preprocessing guide and examples
@anivar
anivar force-pushed the fix/preprocessing-documentation branch from 4e64901 to 75dc325 Compare August 21, 2025 04:34
@anivar

anivar commented Aug 21, 2025 •

Copy link
Copy Markdown
Author

Thanks @arjunsuresh for the feedback. I've now updated the preprocessing documentation with actual implementation details rather than templates.

After reviewing the codebase, I found the existing PREPROCESSING.md files had incorrect information that didn't match the actual code. For example:

  • DeepSeek-R1 was documented with a custom format that doesn't exist - the actual implementation uses apply_chat_template
  • Llama 3.1-8B was shown with chat templates when it actually uses simple instruction format

I've corrected these files based on the actual code in utils/tokenization.py and the preprocessing scripts. The documentation now matches what's actually implemented, so developers can reproduce
the benchmarks correctly.

@anivar
anivar force-pushed the fix/preprocessing-documentation branch from 460542b to 17aa77f Compare September 15, 2025 08:21
@mrmhodak

mrmhodak commented Oct 7, 2025

Copy link
Copy Markdown
Contributor

@viraatc @nvzhihanj : Can we take a look during the TF?

@hanyunfan

Copy link
Copy Markdown
Contributor

@anivar @arjunsuresh Are you able to talk about this in the next WG meeting?

@nvzhihanj

Copy link
Copy Markdown
Contributor

@taran2210 to handle the llama-8B, we will discuss the ds-r1 in the TF

@anivar

anivar commented Nov 11, 2026

Copy link
Copy Markdown
Author

Closing this one: I haven't been able to make the WG calls and the task forces own the per-model preprocessing details. The corrected tokenizer and prompt-format notes are on this branch if anyone wants to lift them.

@anivar anivar closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataset preprocessing code is not shared for several models

10 participants