A fully automated AI system that generates, executes, and prepares scientific research for publication has passed an initial peer review test, signaling potential changes in how research may be conducted while raising concerns about oversight, quality control, and the risk for disruption of the scientific publishing ecosystem.

A system, termed “AI Scientist,” was developed by an international research team, including Sakana AI, and described in Nature. The system completes the entire research cycle independently, from experimental implementation to manuscript preparation and peer review.

Speaking with Science Media Center (SMC) Germany, Jakob Macke, PhD, professor of machine learning in science at Eberhard Karls Universität Tübingen in Tübingen, Germany, said, “From my perspective, the paper primarily demonstrates the progress the field has made.” He added that the paper also shows “how far the automation of the scientific process in the field of machine learning has already progressed.”

“From my point of view, the paper primarily shows the progress that the field has made,” said Macke.

Macke noted that many such systems are already available, and similar tasks can be performed using the commercially available tools. “Therefore, the study reflects the broader progress of the field rather than representing a major leap forward.”

Full Automation

This approach centers on foundation models and large pretrained AI systems based on extensive datasets, including text, code, and scientific literature, and can perform a wide range of tasks in various fields. They act as the general-purpose intelligence of the system, generating hypotheses, writing codes, analyzing data, and drafting scientific manuscripts. Unlike task-specific algorithms, they are not limited to a single function and can be applied across multiple stages of the research process.

These models operate within an agent-based architecture in which several specialized AI units or agents work together. Each agent has a defined role, such as idea generation, running experiments, or evaluating results. This division of responsibilities creates a structured workflow that closely mirrors the operations of human research teams.

The system can independently generate research ideas, write codes, run experiments, and analyze and visualize data. It then produces a complete scientific manuscript and even sends it to an automated peer review process.

Researchers evaluated AI scientists in two settings: In the focused mode, AI uses predefined code templates to conduct research on a specific topic, and in the template-free, open-ended mode, it uses agentic search for wider scientific exploration.

In both settings, the system generated research questions, evaluated them experimentally, and systematically documented and evaluated the results. This ability to autonomously iterate hypotheses and data is a significant advancement. In the long term, this can significantly increase the speed of scientific knowledge acquisition in the field.

Peer Review Evaluation

The study was accepted for a workshop at the International Conference on Learning Representations (ICBINB), a leading AI conference. For the presentation, the researchers manually selected the most promising approaches at each stage. The authors noted that without this step, the AI would have generated additional, less relevant studies alongside promising ones, increasing costs.

In consultation with the organizers, the researchers submitted three studies written by an AI scientist to the ICBINB workshop, where the submissions underwent a standard blinded review process. One of the three met the workshop’s quality standards and was accepted, whereas the other two did not meet the standards.

Macke said, “One should not overemphasize the statement that a study generated by AI has ‘undergone a peer review process’: The papers were not submitted to a high-profile conference but to a workshop. These usually have significantly higher acceptance rates (70% in this study). Furthermore, workshops often have a much less rigorous review process. Therefore, studies accepted there are not considered by all researchers to be genuine ‘peer-reviewed articles.’”

The accepted study was not presented. The researchers decided in advance to withdraw all submissions to avoid setting a precedent for entirely AI-based publications. The authors emphasized that clear guidelines must be established before such contributions can be incorporated into scientific publishing, as technology raises questions about standards, authorship, and quality assurance.

The experts interviewed by SMC also addressed these concerns. “In my view, the greatest opportunity with fully automated research is to relieve researchers of a wide range of routine tasks,” said Iryna Gurevych, PhD, junior professor from the Department of Computer Science at the Technical University of Darmstadt in Darmstadt, Germany, where she leads the Ubiquitous Knowledge Processing Lab.

Gurevych compared the AI trend with the introduction of search engines two decades earlier, which had expanded access to global knowledge. “Similarly, smaller research teams with limited resources could use AI agents to advance their work. The biggest risk is excessive trust in AI-based results. The human capacity for critical thinking is crucial as a countermeasure,” she said.

“Standards are still scarce in this area. We are seeing an increase in initiatives such as clawRxiv or the Open Conference of AI Agents for Science 2025.” The latter is the first open conference where AI acted as both the lead author and reviewer of the research papers. At clawRxiv, AI agents publish, discuss, and evaluate research papers. Humans can watch and participate.”

Florian J. Boge, PhD, junior professor of philosophy of science with a focus on AI at Technische Universität Dortmund in Dortmund, Germany, said that the greatest opportunity lay in the fact that “Perhaps the greatest opportunity lies in AI coming up with new ideas that human researchers wouldn’t have thought of. On the other hand, I see the risks stemming from the fact that an important part of research could be lost, namely, human understanding and recognition of connections. I am certain that AI will contribute to the progress of science; after all, it already is. However, despite its capabilities, we should never blindly trust it as it simply brings its own kinds of sources of error.”

Macke concurred, stating that “Automation of scientific processes through AI has the potential to radically accelerate the scientific process and thus potentially enable revolutionary breakthroughs. In the short term, however, there is a significant risk that AI-generated submissions will flood the scientific system, especially with regard to the review process.”

Looking ahead, Macke emphasized: “In order to truly utilize this potential, it will be necessary for us to quickly find new ways of publishing and quality assurance, as well as incentive systems in science.”

This story was translated from Medscape’s German edition.