In recent days, the first round of evaluation of the ICLR 2027 summit on machine learning has been made public.
For the authors of the paper, this is usually a stressful time. After months or even two years of research work, they end up being peer-reviewed. Reviewers carefully examine methods, experiments and theoretical guidelines, pointing out the problems that really exist in the research.
But this year, when some researchers opened OpenReview, the comments made people cry and laugh.
The ICLR 2027 peer review was once a big spit in the AI circle.
Someone asked the authors to explain the bad standards.
Someone put the A.I. prompt directly on it.
The first is Hanchen Li at the University of California, Berkeley.
He posted that he had received an ICLR review that asked them to explain what "standard deviation" is.
He joked that it was like going back to high school.
The original post did not show the full text of the paper and the context of the review, and it was impossible to judge whether the reviewer wanted the authors to supplement the statistical definition, or whether there were any questions about the specific use of the paper.
However, the post quickly sparked a debate.
Another researcher, Ravid Shwartz Ziv, showed a more comedic effect. He posted comments from three reviewers.
The first wrote: "The paper is well-written and addresses an important problem. However, the authors should consider additional baselines".
At the third place, the scene suddenly changed: "Certainly! Here is a review of the paper". It was only afterwards that the official review began.
The two comments seem to have been copied from the same template, while the third left a more pronounced trace of AI. What's even more disturbing is that these comments eventually entered into a formal peer review process.
Another problem faced by researcher Konstantin Grotov involves AI-assisted writing. He said that his paper submitted to the ICLR received an opinion review focusing mainly on the use of AI.
The problem is that the paper has already revealed AI-assisted writing as requested.
Grotov subsequently explained that the reviewer could of course question the use of AI, but that specific technical arguments should be drawn to explain how this use affected the quality, results or credibility of the paper.
The following is a more subtle case.
Researcher Artur Chakhvadze shared a review in which the reviewer admitted that he did not have sufficient professional background to assess the paper well.
Chakhvadze, on the other hand, acknowledged this: at least the reviewer honestly acknowledged the limits of his abilities. He also said that he had seen many reviewing papers that did not understand the basics of the field, but still gave a high confidence rating of three quarters.
This post has sparked controversy. Some people believe that the reviewer proactively states that he is not familiar with the direction of the research, and is already much more responsible than the confusion and confusion of drafting review opinions. Some people also ask: Since the professional background does not match, why was this paper initially assigned to him? Chakhvadze then added a sentence: the reviewer himself must not be wrong, and it is really worth questioning the process of allocating and managing the review.
Some of the posts are actually of different types: some people don't fully understand the paper, some may rely on AI to generate comments, and some are assigned to areas of research they're not familiar with.
AI is good at finding trouble
But you don't have to know what kind of problem is worth rejecting.
In this discussion, one point of view is particularly noteworthy.
Jiaxin Pei, of the University of Texas at Austin, noted that his most headache-filled review problem in 2026 was the "critical issues" that AI reviewers seemed to have between the "limitations" of the paper and the "critical issues" that really affected the paper's formation.
He argues that large language models can easily find hundreds of limitations in a single paper, even the best papers in history being the exception.
This phrase highlights an important shortcut when AI participates in peer review. A research paper is almost impossible to do face-to-face. The scale of experiments may be limited, some scenarios may not be covered, methods may depend on specific assumptions, and theoretical analysis may be applicable. These are normal research limitations.
What really determines whether the paper is established is whether these limitations undermine its core contributions.
The reviewer can, of course, ask for more data sets, model sizes and control experiments. But if the existing experiments are sufficient to support the paper's core conclusions, additional experiments do not necessarily constitute a reason for rejecting the paper. Conversely, if the paper's most critical experimental design is flawed, or there are irreparable errors in the theoretical inference, even if other parts are beautifully written, they may not be established.
This requires the ability to judge the importance of the problem. A large model can easily confuse the two. It can constantly list potential problems, but not necessarily determine which problems really affect the study's conclusions. Thus, judgments may become longer, requiring more and more complementary experiments, and the core issues that really determine the quality of the paper are instead flooded.
Jiaxin Pei's post has sparked a lot of research.
It has been pointed out that this problem is permeating the peer review mechanism itself. Today, at every AI academic conference, there seems to be a group of reviewers: one who scores lower on a paper, one who asks the author to add something irrelevant, or one who simply adds a little to existing experiments.
If the answer is negative, it may be just an ordinary limitation rather than enough to negate the critical flaws of the entire paper. This is precisely the ability to peer review that is most needed: to know what is worthy of attention and what is acceptable.
What is the meaning of this verse?
If you look at these cases together, you will see that the controversy surrounding the ICLR is not merely a question of the level of reviewers.
In a larger context, AI is changing the production and evaluation of academic research at the same time.
Peer review itself is a job that relies heavily on expertise and time spent. The pace of growth in the number of papers is not necessarily followed by the number of professional reviewers. The ICLR 2027 Project Chairs team has previously explained this issue in detail. On September 2, the ICLR 2027 Project Chairs team published an official article analyzing the new pressures facing peer review in the AI era.
The lead team noted that the field of machine learning has grown rapidly in recent years, with the number of new entrants far exceeding senior researchers, and people with the expertise to review may not be able to keep up with the growth of posts. Generative AI also further reduced the threshold for paper production, and even people who are not familiar with a particular field of research can use AI to produce a text that looks like a research paper. At the same time, reviewers turn the review work over to AI, which is also a new problem to be solved.
To ease the strain on review resources, ICLR 2027 has introduced a series of measures, including limiting the number of posts, strengthening the reciprocal review system, and more.
The obligation to review is further tightened: authors who appear in three or more posts must have completed at least six review work. In addition, each post requires at least one author with a prescribed qualification to register as a reviewer and commit to review at least three papers. If not all authors are eligible, this requirement may be waived.
In terms of AI use, ICLR does not completely prohibit reviewers from using large models, but allows limited, responsible use of AI-assisted writing of review opinions and meta-reviews. However, this is a clear premise: reviewers must disclose the use of AI. If any part of the review opinion is generated or modified using a large model, they must also submit their original independently written paper evaluation, as well as a complete record of their interactions with the large model.
The ICLR also explicitly states that if an AI-generated review contains false information, delusional or inaccurate statements, or the review itself is of poor quality, the reviewer may be subject to disciplinary action for failing to comply with reciprocal review obligations. Penalties may include direct rejection of all posts that he or she participated in as an author, or limiting his or her future participation in conference posts.
Of course, only the first round of ICLR reviews is open to the public. The public discussion between the authors and reviewers will continue until 18 November, with the final decision on admission expected to be published on 16 December.
The link is as follows:
