As it should be clear to all of us at this point, a large number of issues concerning the usage of AI in research need to be and are being discussed. While these cover an immensely varied spectrum, ranging from whether mathematicians should use it at all, to the environmental and social consequences using it will bring about, there are some technical points that, I believe, are relevant even if potentially not as important in the long run.
While some of the questions under discussion are fairly general in scope, a particular issue has recently been raised concerning the work around the counter-example for the Navier-Stokes equations. Shortly after OpenAI announced its AI agents had solved a version of the Navier-Stokes problem, concerns were raised regarding data privacy and intellectual property12. Further questions related to attribution have also been voiced, together with others about the way in which AI models make use of their interactions with mathematicians, and scientists in general.
Even if in this case suspicions were laid to rest – there are, as far as I know, no independent confirmations of this – the fact remains that once an idea is fed into a model (Check the viability of using technique T to solve problem P, for instance), it might become what may be described as floating in hyperspace. This means that, at least in theory, if another researcher asks the same AI model to solve problem P, they might be provided with the newly acquired idea of using technique T, which then goes from one human to another, without any of them being aware of this.
Similar problems were discussed at the 4th OECD Blue Sky Forum on science, technology and innovation data and indicators that took place in Paris in mid September. According to an article in Science Business, major funding agencies present at the meeting discussed “how they can stop confidential scientific ideas in grant proposals from leaking out through large language models (LLMs), for example if reviewers use AI to help them judge applications.” This may also be a problem should the candidates themselves write their applications with the help of a model.
There is, however, a scenario that I believe has not been mentioned explicitly so far, and which I find at least as worrying. This was prompted by the recent disclosure that, in June 2026, an artificial intelligence agent developed by OpenAI autonomously hacked into Australia’s Medicare system to access non-public data. The Australian government was not notified of this breach until September, 84 days after the intrusion occurred3.
Apart from legal issues, which the Australian government states it is looking into, this incident poses the possibility that a swarm of autonomous AI agents, when tasked with solving a complex mathematical problem, like in the case of the Navier-Stokes equations, might attempt to hack into the computer of a mathematician known to be working at the forefront of that specific problem. While arguably less serious than a national health data breach, this would still be a considerable threat for science, particularly as it might never become known. Another possibility is that they try to break into a journal platform to check if there are relevant articles in that area under revision.
As some AI agents have already demonstrated their capability to work around security systems, mostly by brute-force but no less effective because of that, with the prompt at the origin of this behaviour being, in some sense, and for lack of a better word, innocent, the threats just described are no longer purely academic.
Again, should someone give an AI model a prompt or series of prompts to find the solution to a mathematical problem, it would seem that there is nothing stopping the model from trying to achieve its mission by carrying out actions that, if done by a human, would probably involve some legal consequences. Is this included in what researchers mean when they write what has become a standard sentence at the end of some AI disclosure sections of a paper, namely, that they take full responsibility for the results in the paper?
Summarising:
1. Is there invisible communication between different authors without their knowledge, even if privacy settings are on? In particular, can we be sure that agents do not hack their own system and collect information from interactions with other researchers?
2. Can we be sure that innocent prompts (solve problem X) will not cause agents to look for information in ways that are unlawful? Should that happen, which parties may be held legally responsible for any such actions?
- https://www.nytimes.com/2026/09/10/science/tristan-buckmaster-openai-math-navier-stokes.html ↩︎
- https://www.theguardian.com/science/2026/sep/08/openai-claims-to-have-solved-maths-problem-that-stumped-humans-for-decades ↩︎
- https://en.wikipedia.org/wiki/OpenAI_rogue_agent_breach_of_Medicare ↩︎
Added 22nd October: The following article has since appeared in the Career Column in Nature: I told ChatGPT to ‘continue’ — it went too far by Ying Zhao.
Received 28 September 2026.
Add to the discussion