In response to Open AI’s October 6th release of hundreds of results, the Advisory Group on Mathematics and Artificial Intelligence (AGMAI) released a statement saying, “it is ultimately up to the mathematical community to assess the extent to which our recommendations were followed successfully.” By far the most important recommendation that AGMAI had made was for AI companies to stop “testing advanced mathematical problems on proprietary models that remain inaccessible to the broader scientific community.” They identify the obvious reasons for this: it concentrates control over a powerful means of production of mathematical results in the hands of whoever has access to the best models and the most compute, so essentially the AI companies and whoever they decide grant access to. While OpenAI has offered some conciliatory words and paltry financial resources, they explicitly said from the outset that they have no intention of listening to AGMAI on the core issue of testing open problems on internal models. Thus, I conclude that AGMAI’s recommendations were not followed in any substantive fashion, and that AI companies have no intention of changing this in the future. I’ll agree that it was worth a shot, but working with AI companies got us essentially nothing and now it’s time for a different approach. As a start, AGMAI should dissolve itself rather than continue to provide a veneer of legitimacy to OpenAI while OpenAI ignores their central recommendation.
Right now, mathematicians still have some influence because we have things the AI companies want: our data and the legitimacy of our mathematical institutions. The latter is why OpenAI bothered forming AGMAI in the first place. When we work with commercial AI to solve problems, we provide AI companies with valuable training data. In some cases, access to this data might even allow AI companies to solve these problems first using more powerful internal models and without properly crediting the work they built on: behavior that would be condemned as academic dishonesty if you or I did it. When we cite commercial AI services in our papers, we provide free advertising for AI companies. When we tell the media that OpenAI’s release of results is exciting and a big advance for mathematics, we legitimize their unethical research practices. The more we do these things, the more we strengthen the position of AI companies and weaken our own with respect to deciding the future of mathematics. We should stop and seriously ask ourselves if this is what we want to do.
A counterargument is that if we continue to work with AI companies, we can persuade them to adopt norms more aligned with those of the mathematical community. Beyond their behavior thus far, I find this unlikely because the AI companies and many of those associated with them have been quite forthcoming about their beliefs and vision for the future, and if I take them at their word, I can’t see things leading anywhere positive for the existing mathematical community. Many AI company employees (and some mathematicians) sincerely believe that what they are doing has a nonnegligible probability of literally killing everyone, yet they race ahead and implicitly believe themselves justified in making the choice to do so on behalf of humanity. OpenAI’s October 6th release reeks of the Silicon Valley “move fast and break things” ethos, where a private company acts recklessly and destructively to capture value for itself while leaving others to clean up the resulting messes. Also along these lines, recently released documents reveal that insiders at OpenAI believed they were acting unethically and in some cases illegally by scraping training data and using it for their own profit without asking consent from or providing compensation to the humans who produced the data, yet they continue to operate in this manner. Anthropic talks a big game about safety while privately shopping a toothless safety bill that would insulate them from regulation. Sam Altman uses the language of effective altruism to justify near-term harm for the vague promise of a future of material abundance, often eliding the fact that in this future AI companies would amass sweeping control over the means of production, in our context of mathematical results, but aspirationally of all economic value. Are these the people and institutions we want to decide the future of mathematics, let alone of society at large?
There might still be time. If we unite, organize, and exert our influence against the harmful practices of AI companies, we can defend our institutions and community. Imagine if we had all collectively responded to OpenAI’s October 6th release by rejecting the legitimacy of OpenAI’s unethical research practices rather than with shell-shocked chaos. Sure, the preprints they produced would still exist, but I believe this would have had a powerful effect. And maybe AI companies say they don’t need us (their ultimate goal either way) and that they will build their own mathematics. Then we should make them do so without mathematicians or mathematical institutions.
Note too, that opposing AI companies doesn’t mean we need to disavow AI. Many mathematicians have written thoughtfully, here and elsewhere, about how we can harness the potential of AI to advance our understanding of mathematics while ensuring a healthy future for our human communities. Tao and others have written about the need for us to build our own AI infrastructure in which we can ensure equitable access and proper attribution. We should make this a priority so that we are not dependent on the AI companies, and once we have achieved this, we can reap the rewards. However each of us feels about the use of AI in mathematics research personally, we now exist in a world in which AI can accelerate mathematical progress. Opposing AI companies is not about pretending the technology doesn’t exist or that it won’t have an impact. Rather, it is about recognizing that the survival of our institutions and communities depends on our political choices just as much as our mathematical and philosophical choices. As Tao recently stated, mathematics is now on a “war footing,” and if we choose poorly, we will lose the war.
I know firsthand that this is a difficult ask. For one, opposing the AI companies can feel hopeless because they are already so rich and powerful. They promote a narrative of inevitability and want us to feel this way, but we can at least find out for ourselves if it’s true rather than preemptively give up. Another challenge is that many of us have responded thus far in the ways we did because we sincerely love mathematics and discovery rather than out of any intention to aid the AI companies. When I hear a major problem in my field has been resolved, my first instincts are toward excitement and curiosity. I want to come together with colleagues to celebrate a milestone and unpack the new ideas, questions, and directions opened by the advance. Even though the October 6th release was hundreds of slop papers, there are almost certainly insights to be gleaned, and for some of us, to set this aside goes against our integrity as mathematicians and scientists. But right now, that is what we must do to survive. If we allow ourselves to become distracted by shiny new theorems and paltry donations while our institutions are captured and our future generations live in fear for their livelihoods, then the future of mathematics will be written by AI companies rather than mathematicians.
Received October 26.
Add to the discussion