dirk hovy

2017/09/19, 06:46AM
BLOG: Conference sizes in NLP are growing exponentially, which will likely affect how we review, organize, and experience conferences in the future. Some thoughts based on my observations at ACL and EMNLP.
more…

2016/10/22, 04:59AM
BLOG: I had some time and analyzed the US presidential debates from a quantitative point of view. Turns out the candidates differ even beyond their messages.
more…


“When you say you like me, you really like my pony. But when I say I like you, I really mean I like you.”
Bonaparte―Rave Rave Rave

blog

At very irregular intervals, you’ll find impressions, updates, and random thoughts about what I am doing. The blog does not claim to be complete, up-to-date, or meaningful. I tried to post all entries in both languages, and recently switched to English only, but there are some that are only available in one or the other language. In those cases, there was probably too little time, or I was just lazy and all that translating a hassle ;)

Some Thoughts on the Future of NLP Conferences

(2017/09/19, 06:46AM)
If you have been to ACL in Vancouver or followed the news on Twitter, you know that it was the biggest conference of its kind. And EMNLP, held in Copenhagen in September, could again claim the same for itself, too, with more than 1200 participants and more long paper submissions than ACL. Given the recent interest in all things NLP, this development is not too surprising, and if current trends hold, we should expect further growth and more record attendances to come on a regular basis.

Personally, I think that is a good thing: there are still plenty of open questions in NLP, and the field as a whole can only benefit if more people devote their thoughts to the questions in our field. However, it will (have to) affect the way we hold conferences.

While we are still a long way away from conference sizes as seen in medicine and economics, with up to 10,000 participants, we will likely soon regularly reach attendance of 2000 and more participants (as is already happening in the ML community). Even at this level, it is clear that our conference model needs to be open to growth.
As far as I can tell from my involvement in EMNLP, this growth will affect (at least) three aspects of conferences: reviewing, organization, and structure of the actual conference.

There has been a lively debate about the state of reviewing in the community (something I like a lot about NLP as a field: people are always willing to tinker with the status quo). Without weighing in on the arxiv debate (which deserves its own discussion), I think it is becoming clear that reviewing is a both a bottleneck and a control mechanism. It is getting ever harder to find reviewers for all submissions, and the quality of the reviews varies a lot. If we reach 2000, 3000, or even 5000 submissions (remember that there will always be way more submissions than papers), we will need enormous PCs to guarantee three reviews. Some people have entertained the idea of open reviewing, but I believe there is a danger that it becomes a popularity contest rather than a fair quality assessment. So far, reviewing is voluntary, but thankless, despite attempts at reviewing prizes. One incentive could be the requirement to review if you submit: since most papers have more than one author, this could go a long way to reducing the reviewer problem. It would not address quality, but it is hard to see how to tackle this better.

Even if we solve the reviewer problem, we are still left with the decision how to accept papers: right now, conferences aim for 20-25% accepted papers per area. To me, this is the crucial measure to control the ultimate conference size. If we keep the current ratio and submissions keep growing, so will conference sizes. The alternative is to set a size limit instead: the top N papers (by score, meta-review, and random tie break) across all areas get in, the rest is rejected. The problem with this approach is of course that it will drop acceptance rates precipitously, and make it so much harder for good work to get published, but it would allow us to have an upper bound on the size.

As for organization, I do enjoy the personal touch that the involvement of community members as general, local, publication, and other chairs brings to conferences. However, from my own experience I can say how hard it is to combine it with your day job as researcher, and once we pass the 2000 mark, it will become almost prohibitive. Given how tight a schedule many researchers already have, this does not sound like a feasible way forward. Certain positions should always be held by researchers, of course (program and are chairs, for example), but it should be possible to outsource some of the other positions.
A relatively straightforward solution to this problem would be another full-time position in the ACL exec, to support the people who are already there, and to professionalize certain responsibilities currently held by researchers (for example local, handbook, or publicity chair). The increase in salary costs for ACL would most likely be offset by the increased participant numbers.
This would help reduce variability: none of us are trained conference organizers, and since it is always somebody new picking it up as they go, outcomes vary quite a bit. The permanent members of ACL provide guidance and continuity, but while they are doing an outstanding job, they too feel the brunt of increasing conference sizes (probably even more than others). An additional position would help address this. Fewer organizers would help streamline communication.

Lastly, larger attendance numbers will change the actual structure of the conferences as we know them. We might have to envision a completely new model, where conferences become more of a discussion forum to exchange ideas than a presentation medium.
Increasingly, the parallel track model is reaching its limits (although parallel poster sessions seem to work well), and we will either have to introduce even more parallel sessions (unpopular, since attendees will have more conflicts of presentations scheduled at the same time), extend the conference duration (also unpopular, especially for members with family and small kids), or shorten talks. Personally, I am for a 5min talk limit: as it is, most talks can not convey all of the information contained in the paper anyway, so there is little reason for them to be so long. The best a talk can do is serve as appetizer for the paper, which you then want to read in your time. That, however, can be accomplished in 5min just as well as in 12 or 15. I think the exchange in QA sessions is great and should be kept as much as possible, but I am not sure it will be feasible. A better exchange is the direct chat during poster sessions, so increasing poster acceptances is an obvious solution (although maybe not an easy one). Personally, I increasingly like the focused topicality and exchange of workshops, but I am not sure it will be possible to scale them (even at an average workshop size of 50 participants, we would have to have dozens of workshops, which again raises the time vs. parallelity problem, not to mention space issues).

Either way, the growing field will present us with new challenges and affect the way we do conferences. However, I am confident that we will find a way as a community, and am more curious than concerned. The only thing that is for certain is that conferences as we know them will become a thing of the past.

Show all entries

Möchten Sie diese Website lieber auf deutsch lesen?

Valid XHTML 1.0 Transitional Valid CSS!