You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@pgmpablo157321 discovered that we didn't have SS dataset and accuracy for the 5000 sub-samples athttps://github.com/mlcommons/inference/tree/master/language/llama3.1-8b#5000-samples-edge . @attafosu I believe Taran and Tianmu mentioned this will be released but we couldn't find them.
I can help with this missing SingleStream dataset issue. I've been working on preprocessing documentation improvements (PR #2270) and noticed similar gaps in dataset availability documentation.
For the 5000 samples edge case, it appears we need:
The specific SingleStream dataset split (5000 samples)
Corresponding accuracy metrics/ground truth
Clear documentation on where these files should be hosted
I'll investigate the dataset structure and work with the team to ensure these files are properly generated and documented. This aligns with my ongoing work to improve dataset preprocessing documentation across models.
Activity
[-]Llama-8B missing SingleStream dataset and accuracy[/-][+]Llama-8B missing SingleStream accuracy[/+]anivar commentedon Jul 20, 2025
I can help with this missing SingleStream dataset issue. I've been working on preprocessing documentation improvements (PR #2270) and noticed similar gaps in dataset availability documentation.
For the 5000 samples edge case, it appears we need:
I'll investigate the dataset structure and work with the team to ensure these files are properly generated and documented. This aligns with my ongoing work to improve dataset preprocessing documentation across models.