A promising approach to estimate the causal effects of peer review policies is to analyze data from publication venues that shift policies from single-blind to double-blind from one year to the next. However, in these settings the content of the manuscript is a confounding variable—each year has a different distribution of scientific content which may naturally affect the distribution of reviewer scores. To address this textual confounding, we extend variable ratio nearest neighbor matching to incorporate text embeddings. We compare this matching method to a widely-used causal method of stratified propensity score matching and a baseline of randomly selected matches. For our case study of the ICLR conference shifting from single- to double-blind review from 2017 to 2018, we find human judges prefer manuscript matches from our method in 70% of cases. While the unadjusted estimate of the average causal effect of reviewers’ scores is -0.25, our method shifts the estimate to -0.17, a slightly smaller difference between the outcomes of single- and double-blind policies. We hope this case study enables exploration of additional text-based causal estimation methods and domains in the future.
2022
DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Neha Kennard, Tim O’Gorman, Rajarshi Das, and 6 more authors
In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Jul 2022
At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined. We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors’ stance towards review arguments. Further, we annotate \textitevery review and rebuttal sentence. We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.
@inproceedings{kennard-etal-2022-disapere,title={{DISAPERE}: A Dataset for Discourse Structure in Peer Review Discussions},author={Kennard, Neha and O{'}Gorman, Tim and Das, Rajarshi and Sharma, Akshay and Bagchi, Chhandak and Clinton, Matthew and Yelugam, Pranay Kumar and Zamani, Hamed and McCallum, Andrew},booktitle={Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},month=jul,year={2022},address={Seattle, United States},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2022.naacl-main.89},doi={10.18653/v1/2022.naacl-main.89},pages={1234--1249},}
2017
To Plan or not to Plan? Discourse Planning in Slot-Value Informed Sequence to Sequence Models for Language Generation
Neha Nayak, Dilek Hakkani-Tür, Marilyn Walker, and 1 more author
@inproceedings{nayak17_interspeech,author={Nayak, Neha and Hakkani-Tür, Dilek and Walker, Marilyn and Heck, Larry},title={{To Plan or not to Plan? Discourse Planning in Slot-Value Informed Sequence to Sequence Models for Language Generation}},year={2017},booktitle={Proc. Interspeech 2017},pages={3339--3343},doi={10.21437/Interspeech.2017-1525},}
2016
Combining Natural Logic and Shallow Reasoning for Question Answering
Gabor Angeli, Neha Nayak, and Christopher D. Manning
In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Aug 2016
@inproceedings{angeli-etal-2016-combining,title={Combining Natural Logic and Shallow Reasoning for Question Answering},author={Angeli, Gabor and Nayak, Neha and Manning, Christopher D.},booktitle={Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},month=aug,year={2016},address={Berlin, Germany},publisher={Association for Computational Linguistics},url={https://aclanthology.org/P16-1042},doi={10.18653/v1/P16-1042},pages={442--452},}
Evaluating Word Embeddings Using a Representative Suite of Practical Tasks
Neha Nayak, Gabor Angeli, and Christopher D. Manning
In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP Aug 2016
@inproceedings{nayak-etal-2016-evaluating,title={Evaluating Word Embeddings Using a Representative Suite of Practical Tasks},author={Nayak, Neha and Angeli, Gabor and Manning, Christopher D.},booktitle={Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for {NLP}},month=aug,year={2016},address={Berlin, Germany},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W16-2504},doi={10.18653/v1/W16-2504},pages={19--23},}
reports and unpublished work
2018
Building a Conversational Agent Overnight with Dialogue Self-Play
Pararth Shah, Dilek Hakkani-Tür, Gökhan Tür, and 4 more authors
@article{DBLP:journals/corr/abs-1801-04871,author={Shah, Pararth and Hakkani{-}T{\"{u}}r, Dilek and T{\"{u}}r, G{\"{o}}khan and Rastogi, Abhinav and Bapna, Ankur and Nayak, Neha and Heck, Larry P.},title={Building a Conversational Agent Overnight with Dialogue Self-Play},journal={CoRR},volume={abs/1801.04871},year={2018},eprinttype={arXiv},eprint={1801.04871},timestamp={Mon, 13 Aug 2018 16:48:00 +0200},biburl={https://dblp.org/rec/journals/corr/abs-1801-04871.bib},bibsource={dblp computer science bibliography, https://dblp.org},}
@techreport{2014nayak-adjectives,title={A Dictionary of Nonsubsective Adjectives},author={Nayak, Neha and Kowarsky, Mark and Angeli, Gabor and Manning, Christopher D.},number={CSTR 2014-04},institution={Department of Computer Science, Stanford University},month=oct,year={2014},}