Skip to content

[Fix] fix MathCanvas-Bench md5 & summarize_mathcanvas_results#1502

Open
shiwk24 wants to merge 1 commit intoopen-compass:mainfrom
shiwk24:fix/mathcanvas-bench-eval
Open

[Fix] fix MathCanvas-Bench md5 & summarize_mathcanvas_results#1502
shiwk24 wants to merge 1 commit intoopen-compass:mainfrom
shiwk24:fix/mathcanvas-bench-eval

Conversation

@shiwk24
Copy link
Copy Markdown
Contributor

@shiwk24 shiwk24 commented Mar 31, 2026

Summary

Fixes two issues affecting MathCanvas-Bench usage and reporting.

Changes

  • Dataset TSV / MD5: Update MathCanvas-Bench checksum in image_vqa.py so the local TSV matches the current file and avoids unnecessary re-download / mismatch warnings. The TSV contains one minor data update at index 165: the question image was redrawn in Python to improve the coordinate system rendering. This does not affect the problem content or solving.
  • Summary stats (summarize_mathcanvas_results in mathcanvas.py): Align by_question_image_count and by_knowledge keys with the dataset so per-category and image-presence metrics count correctly (previously keys did not match).

Notes

No change to the judging logic; reporting / aggregation only, plus the hash update.

@clintonsteiner
Copy link
Copy Markdown

@dependabot rebase

@kennymckormick
Copy link
Copy Markdown
Member

@shiwk24
There is one data sample changed in the new TSV file, please give a short introduction.

@shiwk24
Copy link
Copy Markdown
Contributor Author

shiwk24 commented Apr 6, 2026

@shiwk24 There is one data sample changed in the new TSV file, please give a short introduction.

Sure, I have updated it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants