Thoughts On the State of NLP

At EMNLP 2026 in Budapest, I was on a panel with Mor Geva, Lillian Lee, and Raymond Mooney about New Missions in NLP Research. The panel is on Underline, but if you cannot access it, here is a write-up of my main points. We all agreed beforehand that a panel where everyone agrees is boring. In that spirit, I scoped out some more provocative positions (though I was told later that many found them just plain reasonable). Anyway, here they are.

TL;DR: NLP’s problem is not everyone using Transformers, but confusing benchmark progress with scientific and sociotechnical advances.

I have spent all of my academic life in the *ACL community, but now I worry that we are losing our distinct point of view as a field. We have spent decades building “better” models by testing those models on data and getting numbers that went up, and largely ignored the people who would use these models. And for the longest time, nobody outside our community cared about those systems, data sets, or numbers. That all has radically changed: our technologies are being used by billions of people every day now. Consequently, the important bit is no longer building more capable models and making numbers go up, but understanding sociotechnical systems that serve the people who use them. NLP, however, seems to have missed that shift and is still stuck in bigger-number mode.

That singular focus on benchmarks and better numbers has direct ramifications for some of our core tasks: “Safe” and “aligned” are not intrinsic model properties but relationships between a model and its individual users. That renders most current approaches to solve these issues almost meaningless. By making a number go up, we can make ourselves feel good, but we are not achieving what we set out: A metric that becomes a target ceases to be a good metric, as Goodhart’s Law puts it. Benchmarks that do not contain users and recourse therefore only measure model behavior, not actual safety or alignment. In 2020, Ribeiro et al. wrote about using unit-test-like checks to go “beyond accuracy” (the title of the paper). We awarded it the Best Paper Award at ACL, and then promptly forgot about it. We should revisit that approach and need a further push to move from behavioral testing to evaluating systems in use.

I also want to push back against the notion that there could be a universal alignment that would somehow help safety for every user. We know that people differ across cultures and languages, and even within those. So what is perfectly aligned and safe for one user might not be for another. And the ones who are most affected are typically those who are least represented. Building more socially adaptable and personalizable models will go a long way toward addressing safety. Sociolinguistically and sociodemographically smart models are crucial for that. Instead, using a one-size-fits-all approach is like running crash tests with a seven-foot, 100 pound (or 2.1 m, 45 kg) dummy: maybe practical, but completely unrealistic. We should be careful though: using LLM-generated persona are caricatures, not realistic stand-ins for real people either.

We can and should develop smaller models, but that by itself cannot set the incentives for companies to use them. That is an economic and political issue. I don’t want to absolve us of responsibility, but we need to be realistic about our reach and impact. We can and should lobby for that, but it is not something we can solve within our field. Pretending otherwise offloads structural responsibility to the individual. Where we can help out is by developing and creating better data sets, especially for under-resourced language communities. Importantly, though, these groups are not mere data generation sources, but should be treated as stakeholders (shoutout to the work of Steven Bird). Crucially, we should not fall into the trap that formal linguistics fell into by pretending language is completely divorced from social, psychological, and other factors (which required decades of effort of sociolinguistics, psycholinguistics, etc. to undo). Simply throwing more text at a problem does not work. Treating the problem that way risks falling back into “bigger number = good” territory. Instead, we should include other modalities, including social behavior. These may look like separate problems, but they all stem from the same incentive structure: we reward measurable benchmark improvement more than understanding, social value, or real-world impact.

By ignoring the sociotechnical aspects of our work, we have not only abdicated the responsibility for these technologies, but also lost the authority of interpretation over them: The most important technical papers in NLP of the last decade have not been published in *ACL venues, but elsewhere. Increasingly, students prefer to send their work there. When the EU, US Congress, or other governing bodies seek expertise, our community is rarely the first port of call, instead they ask other experts or even companies. We should lobby more aggressively and make our points of view heard.

We instead often do worse versions of HCI, social science, or linguistics: without the necessary methodological training, theories, or standards. Instead, we should reach out to other fields and make sure we level up our methodological training: As Manoel Horta Ribeiro suggested in a blog post, we should all be more trained in methods like econometrics and approach papers more like economists (minus the publishing culture). The future of NLP needs more social science. They also have many of the big question worth answering, and we can contribute to doing that.

Instead, our current publishing culture of roughly bi-monthly deadlines encourages salami-sliced contributions (usually a temporary SOTA result) to have something ready for the next deadline. That leads to a “tragedy of the commons” situation where individuals can boost their h-index and citation count (at the cost of constantly feeling pressured to submit), but the credibility of the field as a whole suffers. Again, this is not something that individual researchers or groups can solve, but that needs to be addressed at an institutional level. We, as ACL, have that possibility. Instead of adding more deadlines and conferences, we should limit ourselves to two or even one annual deadline, and then hold two conferences in affordable and easily reachable places to maximize inclusion.

I am ok with all that meaning the coding and engineering part is outsourced to models. As the technical implementation becomes cheaper, the engineering aspect recedes a bit. However, other fields can now do that too. Those fields, in turn, might not need us anymore to do their work. As Pascale Fung said in her keynote: “What is NLP if everyone is doing NLP?” What is our contribution? This is where understanding, explanation, and interpretability become more important. Good ideas and understanding are still scarce. Let’s focus on those. I do not believe they will suffer by using Transformers to be implemented: the initial idea and prompt still need to come from a researcher. Knowing which questions are worth asking and which problems are important tol solve requires human assessment. We need not choose between researchers and LLMs, the two can co-exist in complementary roles. That complementarity (and the complexity of what it means to be human) are also the main thesis of my forthcoming book, “Not Like Us” (MIT Press, March 2027). The problem is not everyone using Transformers, but everyone still thinking bigger numbers mean better models or interesting science. SOTA is temporary and not a scientific goal or an advancement of understanding. The danger in using LLMs in our work is that everyone’s papers sound the same since they all have the same co-author (see the paper led by Joe Baumann).

Our field contains multitudes and, at its best, bridges rationalism and empiricism, understanding and engineering, linguistics and computer science. If we lean into this, we can produce exciting and meaningful research that is useful, actionable, and helps users, instead of just passing some benchmark to make a number go up.