Use LLMs to check your work

Anonymous

SERIES · AI is making papers cheap to produce but they remain costly to referee. Proofs and Prompts is publishing a series of posts from 26 October on the future of publishing, papers and credit. Questions such as how to disclose AI use, who bears the cost of checking, what authorship means and what papers of the future might look like will be addressed. This post is part of the series. Each is written independently and reflects only its authors’ views. All posts in the series

AI disclosure. This essay was composed on the Notes app of the author’s iPhone. This app provides AI assistance with spelling, grammar, and auto-completion of words being typed.

Most discussions about LLMs in mathematics seem to focus on their (rapidly advancing) proof discovery abilities and the effects these abilities may have on professional mathematics. I write here about a different use of these technologies, to make the following point: we have a professional duty to use LLMs to proofread our work. I use “proofread” to mean both copy editing of no mathematical substance, as well as the detection and correction of mathematical errors and gaps in arguments. By “our work,” I mean our research papers, though the arguments below extend with straightforward modifications to teaching materials.

First, a fundamental community norm is that we should make all reasonable efforts to ensure that our proofs are correct. Modern LLMs are tools that, when used appropriately, can provide this assurance with high confidence and little effort.

Second, incorrect or inadequately justified proofs exact a substantial human cost. I have refereed many papers in my career; the vast majority had at least some minor mathematical issues, and a not insignificant minority had issues that required new contributions by the author(s) to patch. Identifying these issues and writing a detailed referee report to alert the author(s) takes many hours, especially for long and technical papers.

Before LLMs, I executed my referee responsibilities with pride, knowing that authors were (typically) putting forth the best work they could, and that operating on the frontier of mathematical knowledge would inevitably involve some mistakes. Now, however, when I see clear mistakes in papers I am reviewing, I just feel disrespected. The author(s) had a tool they could have used to eliminate the mistakes and did not, wasting my time. (Certainly, LLMs are not perfect, but it is easy to tell when a cursory check has not been done.)

Recent events in my personal life have emphasized for me the value and precariousness of human life, and how death and life-limiting illness can strike unpredictably. Submitting work that has not gone through an LLM check to a journal may cost a conscientious referee many hours of their time. To ask someone to spend their most precious resource correcting errors that could have been caught with the push of a button is both rude and professionally negligent.

Some may argue that abstention from LLM checking is justified on ethical grounds. If this objection rests on the behavior of frontier labs, one can simply use open-weight models. If the objection rests on resource use, a simple calculation shows that the resource use from a few minutes of LLM computation pales in comparison to other activities we regularly undertake to advance mathematics, such as conference travel. I am sure there will be other creative objections, but it is hard for me to see how they can outweigh the cost refereeing erroneous mathematics has on human life (especially in combination with the cost to all other readers who might be interested in the result).

I also believe we have a duty to use LLMs for literature searches, to ensure correct attribution, and to make sure our arguments are as simple and straightforward as possible. The reasons are similar.


Received 30 September 2026.

Filed under
More by →

14 responses to “Use LLMs to check your work”

  1. Grigori Avramidi Avatar
    Grigori Avramidi

    I second this. But, I feel uncomfortable to even suggest this to coauthors who are more skeptical of llm use and want to keep a particular paper completely llm free.

    One worry to keep in mind is that this may lead to disjointed papers that are more correct but harder to read (if there are humans who grudgingly fix gaps pointed out by an llm), just like ordinary refereeing often leads to such papers. So, there is a balance to be struck.

    On a related note, I have been asked to referee ai assisted papers by a journal that forbids me to use ai assistance to referee said paper. This is absurd and needs to change.

  2. Gordon Royle Avatar
    Gordon Royle

    What even is the purpose of refereeing at all now? An LLM can verify correctness and novelty so what else? Formerly a human referee might give an opinion on the significance of the result or quality of the exposition but I am now hopelessly uncalibrated. Many people are rushing out AI-assisted papers proving results that would previously have been acceptable but now are routine. Is a complicated exposition using a wide range of unfamiliar techniques poor writing or is it actually the simplest possible given that only an LLM would have combined so many disparate ideas? Personally I think that sites such as hexagon will rapidly dominate and journals will fade into irrelevance. If you need a result then search hexagon, verify the result with your own LLM and then use it.

    1. Anonymous Avatar
      Anonymous

      A slippery slope.

      What is even the purpose of any humans interacting with mathematics at all now? An LLM can produce novel mathematics now, at a faster pace than humans, and even produce conjectures, so what else?

      Just let the LLM produce, verify, referee, and expand upon everything, and take humans out of the loop entirely. In fact, humans are so inefficient, we should just get rid of them in favor of the machine god.

      1. Gordon Royle Avatar
        Gordon Royle

        At the moment, I don’t see any realistic trajectory that doesn’t end up largely writing humans out of the loop – not that I want this to occur, just that I can’t see how it can be avoided. I don’t think asking human reviewers to referee AI-generated/AI-assisted papers counts as keeping humans in the loop.

    2. A fool holding onto their humanity Avatar
      A fool holding onto their humanity

      What are you even doing here, dude? Why do *you* need a result if you’re just prompting “your” AI to read other “people’s” AI’s work. Why not just save the AI the trouble of having a human in-the-loop and go lie down and rest, at least AI seemingly can’t do that.

      1. Gordon Royle Avatar
        Gordon Royle

        I’m just making a gloomy prediction of where things are heading, certainly not saying I like it. There are many people (including some of the biggest names in the biz) using AI in all sorts of ways to accelerate their research. As a result, papers or paper-like artefacts are accumulating so rapidly that only a fraction will ever be human refereed. At the same time, many of these documents contain correct results and valid proofs and so people will start to use them and cite them. (Even with AI, verification is easier than discovery.) So the tradition of results carefully built on refereed papers themselves built on refereed papers all the way down is, I believe, no longer sustainable.

    3. Joker Avatar
      Joker

      I don’t like this but my horrible experience of editor not sending out my submission for review for over a year and a half and then rejected it after I got a revision request from the referee just prevents me from refuting the argument.

    4. Wes Avatar
      Wes

      It seems very possible that AI will soon be able to ask good questions and build theory, at which point, if we go this route, mathematics will be done primarily by and for AI with no human involvement and human mathematics will be relegated to hobbyists.

  3. M Avatar
    M

    Assuming everyone actually would have equal access to capable LLMs, I agree. I also do not want to review work that could easily have been improved before taking my time.

    But a problem with this is review fatigue: if we only read papers where all easy to spot mistakes are already eliminated then we will also get worse at detecting any hard to spot errors. We will get used to just waiving things through.

    Recommended reading: Cory Doctorow about (reverse) centaurs.

    1. Kevin Russell Avatar
      Kevin Russell

      The review-fatigue worry matches what we saw from the other side. On our project we run AI agents as research sweepers, and we published that we caught ‘60% sweep-agent hallucination’ at what we call Stage 3.5. In another ledger, verification killed 9/9 hallucinated breach IDs. So I’d frame the duty as two separate checks: an LLM read catches typos, gaps and missing citations, but it is itself a claim that needs checking. Anything load-bearing should be something a referee can re-run or mechanically check. Would it help referees if AI-use disclosures said which steps were LLM-read and which were mechanically checked, so “pre-screened” doesn’t quietly turn into “verified”?

  4. Sash Avatar
    Sash

    I’ll go further, we have an “obligated” duty to first use a billion tokens at least to solve any mathematical problem, if not we should wait for the next reset, and spend a few billion more tokens and only if that doesn’t work and we’re sure no new model release is coming in the next few weeks, should we be professionally obligated to use our minds to do something, anything else is not the most efficient use of resources.

    Actually this is not enough, we are obligated to spend a trillion tokens with every new frontier model to try to solve every existing open problem since it is a mathematicians duty to produce as many true statements as possible.

    Why only focus on open problems. If it is our duty to society to maximize number of true statements, we should actually just ask the LLM to open-endedly search a field for true statements that it can prove. We can categorize each statement by number of tokens it took for the proof, to understand how deep the search was. Depending on our grant money, I think we are obligated to spend at least a billion tokens on this every day (~10$) and then every week we should release a book of our findings so that other LLMs can build on our work. Imagine just how much further our towers of abstraction can go.

    I’m not being sarcastic. I think there should be Mathematicians who do this, there should be complete AI maxing, along with complete AI manning and then like a controlled experiment we should see what is actually more valuable to a society and funnel grant money to those people.

  5. rdavis Avatar
    rdavis

    To ask someone to spend their most precious resource correcting errors that could have been caught with the push of a button is both rude and professionally negligent.

    On the other hand, I find it rude to tell those who choose not to use AI to compromise on their morals. If you don’t want to correct those kinds of errors, then you’re free to decline any requests to review papers that have not indicated pre-screening by AI.

  6. Wes Avatar
    Wes

    As long as the best models are commercial LLMs, I think mathematicians should and will be concerned about data confidentiality. Unless you have a local model or strong enterprise protections, using LLMs to check your work gives access to that work to commercial AI companies.

  7. Anonymous Avatar
    Anonymous

    The objection is that you leak all IP to BigAI. The author suggests local models. The price to run Kimi K3 locally is about $1000,000 plus electricity, if you even get Nvidia H100 cards on the market because they all go to the oligarchy:

    https://openmodelmap.com/kimi-k3/hardware

Comments are moderated. Read our comment policy.

Add to the discussion

New posts by email.

Prefer a feed reader? Subscribe by RSS.

Also on Mathstodon.

Latest comments across the site.

  1. Pehaps a useful point to keep in mind is that when the convex hull involves points from a far away...