About Writings Contact
Manifold logoManifold logo
All models are from the Gemma 3 family. Thanks Google!
Highlight mode:
Model for highlight:
Legend · Probability of the true token
p < 1e-5
1e-5–1e-4
1e-4–1e-3
0.001–0.01
0.01–0.1
0.1–0.5
p ≥ 0.5
Select models to compare • B = Base, I = Instruct • Click size to toggle both

A prediction on AI and mathematics

Or: AI might eventually lead to the end of the world, but in the meantime there will be beautiful mathematics

Ever since 2020 it has seemed to me that superhuman AI mathematicians are pretty close, and nowadays they seem quite a bit closer. While the exact year and date seem to me to depend heavily on various local contingencies - who decides to invest how many person-hours and capital into what, etc., I'll say that I would be quitesurprised if superhuman AI mathematicians are not here within 10 years.

I am not too fussed about the exact definition of "superhuman" here because it seems very likely that, whenever we get anything that fits any "reasonable" definition of a superhuman AI mathematician, within very short time - a year or two - we'll have AI mathematicians which are obviously super(superhuman AI mathematicians), and in another short time we'll get super(super(superhuman ...)) - certainly not ad infinitum, but it will for sure be ad nauseam for some. One corollary of this prediction is that I expect there to be as much progress in mathematics in the next 10ish years as there has been up until now.

I wouldn't really care as much about that, though, if it was only mathematics in play. But I imagine these AIs are likely to be used for making an astounding amount of progress Why the focus on superhuman AI mathematicians, as opposed to superhuman ML scientists? Why pull mathematics into this, if it is mostly to serve an instrumental value? One reason is that pure mathematics, as opposed to most applied subjects, provides such an incredibly fertile ground for almost unlimited self-play, that it seems to me quite likely that AIs are first going to be superhuman pure mathematicians first and applied mathematicians (including ML scientists) second - though not with much time inbetween, as I expect mathematical ability to generalize far. - we very quickly get many years of (current) progress in finding more efficient architectures, training procedures, etc. etc. For, sure, ML has a sizeable empirical component, so the AIs will probably need to spin up an experiment or two here or there, but these experiments should likely provide< much more valuable information than most human-level ones. So, soon after superhuman AI mathematicians, we have superhuman everything.

To put it more straightforwardly: superhuman AI mathematicians seem to me to be both a sufficient and a necessary component of the near-term transformation of the world. If you get superhuman AI mathematicians you can leverage them into transformation of the world - and if you don't have them, the road to transformation is much longer.

So it is worth some forethought trying to figure out how to leverage this state of affairs in order to advance AI safely. In fact, from my point of view, one of the single highest points of leverage over the future is being prepared for the period in which we will be able to direct our AI systems in order to build a deep, deep understanding of AI systems themselves.