Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?
Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?
29
𝕊304
resolved Feb 13
Resolved as
78%
Start earning real cash prizes today.
Step 1: Verify and claim your free sweepcashṀ
Step 2: Correctly predict on sweepstakes markets.
Step 3: Withdraw winnings!
S1.00
won → $1

From a recent arXiv preprint,

We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress.

This question resolves to YES if the state-of-the-art average accuracy score on the FrontierMath benchmark, as reported prior to midnight, January 1st 2028 Pacific Time, is above 85.0% for any fully-automated computer method. Credible reports include but are not limited to blog posts, arXiv preprints, and papers. Otherwise, this question resolves to NO.

I will use my discretion in determining whether a result should be considered valid. Obvious cheating, such as including the test set in the training data, does not count.

  • Update 2024-21-12 (PST): There is no maximum inference budget for this benchmark - models can use any amount of compute to solve the problems. (AI summary of creator comment)

This question is managed and resolved by Manifold.
Get
Ṁ1,000
and
S3.00


Sort by:
15d

The sweepstakes market for this question has been resolved to partial as we are shutting down sweepstakes. Please read the full announcement here. The mana market will continue as usual.

Only markets closing before March 3rd will be left open for trading and will be resolved as usual.

Users will be able to cashout or donate their entire sweepcash balance, regardless of whether it has been won in a sweepstakes or not, by March 28th (for amounts above our minimum threshold of $25).

19d

Haven't read the paper. But are all the problems provable in the dataset? Or does the model have to find counterexamples otherwise

@travelling_salesman
All questions have numeric answers in this dataset.

21d

Seems good to check benchmark errors for this type of question, to see if the criterium is even achievable. Epoch AI found that "approximately 1 in 20 problems had errors", so it definitely looks reachable

filled a Ṁ1,000 YES at 86% order27d

o3-mini gets 32%, getting 28% of T3 problems correct (when using python, which seems allowed in this market)

27d

within 2 months of announcement the sota went from 2% to 32%. 32% was achieved with a distilled model. Uhhh

SOTA was nearly 13% at the announcement looking at this table. Probably even higher if this python program was used.

So in some ways while frontiermath appears easier than it did at launch, the capabilities jump of o3 may actually be lower than we thought before

2mo

[Asking this on all the FrontierMath markets.]

If FrontierMath changes (e.g. if Tier 4 is added to what's considered to be the official FrontierMath benchmark), how does that affect the resolution of this question?

It seems to me like the fair way to do it is to go based on the original FrontierMath benchmark (modulo small tweaks/corrections), but I'm not totally sure that in the future we will have benchmark scores that are separated out by original vs. new problems.

2mo

@EricNeyman Excellent question! IMO the best option is for the guys from Epoch to create separate benchmark for Tier 4 or to fork FrontierMath and give it a new name like FrontierMath-Enhanced, or something like that.

"the fair way to do it is to go based on the original FrontierMath benchmark" - I agree.

2mo

@Metastable I agree! But it would also be nice to have clarity on what happens if Epoch doesn't separate them cleanly like that.

2mo

@EricNeyman We don't want assertions like "o3 first scored 25% on FrontierMath in 2024" to get muddied by Tier 4. It will definitely be separately tabulated in some form. I would interpret this and all similar markets as being about base FrontierMath, and I'm sure that's how Matthew intends these markets to be resolved.

2mo

@MatthewBarnett is there no max inference budget for this?

bought Ṁ5,000 YES2mo

Holy crap.

2mo

@Bayesian Guys, it's time to stop being surprised. Just update all the way.

bought Ṁ750 NO2mo

@AdamK Nah, priced in bro

bought Ṁ1 YES2mo

@RyanGreenblatt Well, I guess they are now that I moved all the prices.

3mo

Related market:

What is this?

What is Manifold?
Manifold is the world's largest social prediction market.
Get accurate real-time odds on politics, tech, sports, and more.
Win cash prizes for your predictions on our sweepstakes markets! Always free to play. No purchase necessary.
Are our predictions accurate?
Yes! Manifold is very well calibrated, with forecasts on average within 4 percentage points of the true probability. Our probabilities are created by users buying and selling shares of a market.
In the 2022 US midterm elections, we outperformed all other prediction market platforms and were in line with FiveThirtyEight’s performance. Many people who don't like trading still use Manifold to get reliable news.
How do I win cash prizes?
Manifold offers two market types: play money and sweepstakes.
All questions include a play money market which uses mana Ṁ and can't be cashed out.
Selected markets will have a sweepstakes toggle. These require sweepcash S to participate and winners can withdraw sweepcash as a cash prize. You can filter for sweepstakes markets on the browse page.
Redeem your sweepcash won from markets at
S1.00
→ $1.00
, minus a 5% fee.
Learn more.
© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules