Minerva

Stream: Machine Learning for Theorem Proving

[Yuhuai Tony Wu]

I'd like to share with folks here about our recent work, Minerva (blog post, paper): a language model that can reason mathematics, general science in natural language, using step-by-step reasoning. Some remarkable achievements with this model:

1) 51% acc on MATH, a math competition dataset, which was predicted to happen in 2025.
2) 65% acc in 2022 Poland’s National Math Exam which is more than the national average!
3) 30% acc on “undergrad” problems in physics, biology, chemistry, economics, etc.
4) 82% on GCSE Mathematics - this is a UK exam for 16 years old taken by about 700K students each year.

You can find some example solutions at sample explorer.

Appreciate any comments, questions, or other feedback!

[Yuhuai Tony Wu]

One clear limitation is the lack of verification with natural language models, and especially when it comes to proofs. But I'm optimistic to make use of the autoformalization abilities of these models to help verify, and further provide feedback signals to ground meanings etc.

[Jason Rute]

Nice work @Yuhuai Tony Wu!

[Floris van Doorn]

This is very impressive. Some of the solutions in the samples are really good!