On our existential crisis and trustworthy AI

Michel Schellekens, Professor of Computer Science at University College Cork

I am a mathematician and a (theoretical) computer scientist. AI affects both vocations deeply. On the back of the recent AI-driven Navier-Stokes result, computer science colleagues joked that I am no longer needed as a mathematician. Of course the tables can be turned. No job seems safe, especially, and ironically, jobs involving programming. AI-assisted programming now disrupts its field of origin, the characteristically disruptive area of computer science. This meta-disruption is on the mind of everyone working in computing.

I am concerned about mathematics too. I value our mathematical culture deeply and share the concern of our field being disrupted by AI. I signed the recent declaration on “A Severe Misalignment of AI in Mathematics” by Fields medalists since at the very least it sends a signal that change is required. The current disruption is reminiscent of the upheaval mathematics experienced upon Gödel’s shocking results, capturing inherent limitations of our capacity to reach true statements from axioms.

The impact of AI is more immediate. This industrial revolution forces us to confront our potential obsolescence as mathematicians and/or programmers. Whether this worry is entirely justified remains to be seen, but, even if only partly true, the possibility inevitably brings questions on job security, our value in society and the meaning of our work. This visceral experience is corrosive to motivation. Many wrestle with it as do our students. The uncertainty of where the actual boundary of AI capacity to aid mathematical research or programming will land makes life hard for practicing mathematicians, computer scientists and students in the field.

Of course, computer science colleagues would counter that the actual task of dreaming up and engineering new software applications is what matters and that AI helps to develop these faster.

The same holds for mathematical pursuits.

Personally, mathematics, aside from an astonishingly beautiful garden to visit, has also always been a means to an end, engaged in on the back of a particular interest. This makes me less of a pure mathematician, perhaps, even though my research pursues a pure mathematics foundation for the design automation of optimal algorithms. This goal requires crossing mathematical fields according to need, similar to the case for physics, i.e. it is never purely driven by a desire to develop a single mathematical area in its own right.

In these mathematical pursuits I would like nothing more than to avail of an AI tool that helps me reach these goals. I have access to AI of course and view it as a useful tool, but, judging from the latest controversies, it remains deeply frustrating to only have access to environments that cannot be fully trusted or give the appearance of not being trustworthy.

Trustworthiness is the biggest hindering block to pursuing new ideas freely using AI. If novel ideas we pursue permeate everywhere immediately in the system, then clearly there is a massive deterrent to use AI by scientists. A company that develops trustworthy environments in which AI can be used without it absorbing all new ideas or scooping these, is a company that would merit our subscription. Currently there are two options on the horizon.

The proposal of a CERN for AI-assisted science is a brilliant idea and would offer a much needed resource.

Similarly, the SAIR’s Open Math Model initiative is a hopeful development and invites our support.

Until AI manages to identify a new problem and model or solve it independently, or “dream up” a goal and solve it creatively, there is still plenty of scope for mathematicians to be creative, set goals, and, in combination with AI, create and explore new fields, applications and results.

Perhaps this is what future mathematics education could focus on. Not only to develop mathematical theorem proving or problem solving muscle in a particular field, but to increase focus on the creative aspects of developing new fields. E.g., by setting new axioms to model newly encountered structures, processes or problems in combination with learning about the mechanisms of discovery. This can be combined with experimenting, learning and researching aided by AI.

Of course, many mathematicians make such creative leaps, combining areas in new ways or modelling unknown aspects of nature or mathematics itself. Perhaps, ultimately, AI will achieve this too.

Until that day, we will not have lost all purpose, even though the readjustment is taking its bitter toll.

Even on the bitter lesson front, all is not doom and gloom. Moshe Vardi’s Sweeter Lesson discusses how the success of domain specific applications is over-shadowed by the focus on general methods that leverage computation.

There is plenty of scope for AI-improvement by incorporating the human knowledge of our chosen domain of study. Mathematics captures domain-specificity via its axiomatisations. This potential is under-utilised due to the swing from pure logic-based AI-approaches to the highly successful stochastic modelling of neural networks. The middle ground of domain specificity remains largely unexplored and this presents opportunities for mathematicians and physicists who are ideally suited to contribute in this area. The middle ground forms, by its nature, hard but worthwhile territory to capture, offering a potential counter weight to the error-proneness and energy-consuming nature of general purpose AI.

It is tempting to dismiss the pursuit of domain-specificity in light of forceful applications such as Navier-Stokes. The dust has not entirely settled on this matter, but even accepting the outcome as fully correct and independently obtained, casting out domain-specificity means demanding too little. Designing guaranteed optimal solutions in novel contexts in a computationally efficient and error-free manner (without bolting on a verifier) is well beyond current AI’s capability. Bacon’s “nature to be commanded must be obeyed” remains a valid counter point to the bitter lesson.

The day on which AI overtakes us on the mathematical creativity front may be far off. Meanwhile I hope that our community will soon be served with a trustworthy version of AI that allows us to further science, rather than a version that currently does not allow for unrestrained scientific exploration due to an untrustworthy set up.


Received 17 September 2026.

7 responses to “On our existential crisis and trustworthy AI”

  1. Michael Rozynski Avatar
    Michael Rozynski

    “If novel ideas we pursue permeate everywhere immediately in the system, then clearly there is a massive deterrent to use AI by scientists.”

    It cuts both ways: suppose the ‘novel’ but utterly stupid ideas that permeate everywhere in the system far outnumber the novel and correct ideas.

    Which will surely happen given that the crackpots far outnumber the experts in any given profession or human endeavor.

    Wouldn’t it be nice then if the systems would choke on their own triumphs?

    1. intruigued Avatar
      intruigued

      Interesting take. This risks happening in computer science too with sloppy code being generated. I’m not sure how this will pan out. The companies may try to label data or find ways to filter out rubbish, but the slop already generated by a flood of flimsy scientific articles is probably a headache for them.

  2. Manuel Avatar
    Manuel

    I am not fully understanding the logic why Ai would replace us once it can pose its own beautiful problems and solve them autonomously.

    This viewpoint somehow seems to assume that untrained humans just magically could make any sense out of such AI endeavors. But appropriate training takes humans many years and requires a lot of hard work, and it requires a healthy infrastructure, like universities that host active research carried out by seniors that motivates the young ones on their way.

    If Ai is just running and doing its own thing better than any human, we arrived back in the old days where we could’t properly understand natures laws at all and started the practice of science to understand as humans the world around us.

    1. Anonymous Avatar
      Anonymous

      Well said!

    2. intruigued Avatar
      intruigued

      If AI can set its own research targets, mathematicians will be even more robbed of their creative contributions and less inclined to pursue mathematics, unless it perhaps becomes a kind of joint effort. If AI achieves this independent target setting and brand new theory development then companies, who will be able to interpret the AI’s goals, will take over the creative part of mathematics. It will require expertise to interpret the results, but there is currently a joy in pursuing mathematics and setting our own goals. I would imagine it is a lot less joyful when the system does the entire thinking for us and creatively paves new ways which then is interpreted by experts.

    3. Vaughn Gerrits Avatar
      Vaughn Gerrits

      in the old days, only the most wealthy could afford to do natural philosophy. that’s not the future I want.

      1. intruigued Avatar
        intruigued

        We’ll have little choice in influencing this. But the open models are bound to be used to build alternatives that scientists can use safely and freely or at modest cost (in the EU I bet).

Comments are moderated. Read our comment policy.

Add to the discussion

New posts by email.

Prefer a feed reader? Subscribe by RSS.

Also on Mathstodon.

Latest comments across the site.

Discover more from Proofs and Prompts

Subscribe now to keep reading and get access to the full archive.

Continue reading