alt text 

Daniel Khashabi (دانیال خشابی)
Young Investigator
Allen Institute for Artificial Intelligence
2157 N Northlake Way Suite 110, Seattle, WA 98103


My research focuses on the computational foundations of intelligent behavior, through the lens of natural language. My goal is to explore the different ways in which we can develop and evaluate systems that understand and reason with (and about) natural language in different contexts. I mostly identify with natural language processing (ACL, NAACL, EMNLP) and artificial intelligence communities (AAAI, IJCAI) and, to some extent, with the machine learning community (NourIPS, ICML).

Here are several general themes I have pursued:

  • How can we build better and more useful models? We need models that have better generalization capabilities. AlphaGo may be the world champion at Go, but it doesn't know a damn about any other problem in the world! What can allow us to instill machines with better generalizability?
    AI systems should have the capacity to reason, in ways that are high-level and not attached to individual tasks. During my PhD, I worked towards this vision by creating symbolic systems that effectively utilize the knowledge of their context to solve a given problem (IJCAI 16, AAAI 18). More recently, I have pursued the generalizability objective in terms of neural models that can solve a wider range of problems (EMNLP 20).

  • What are more meaningful and effective ways to evaluate the progress of AI? We must actively rethink the evaluation of AI systems and move the goalposts according to the latest developments. I persistently pursue this perspective in order to build more comprehensive challenges for our latest technology. Instances of such vision are embodied in the design of our challenge datasets for language understanding (NAACL 18, EMNLP 19, TACL 21). Along the same goal, we recently released GENIE🧞, a human evaluation platform to fasciliatet comprehensive evaluation of text generation tasks.

  • Are system doing what they are supposed to do? The sad reality of our current AI technology is that our models, under the hood, often depend on shortcuts that are far from the human decision-making process. Part of my research is dedicated to the ways in which we can bring out their unreliable correlations that could lead to catastrophic errors (EMNLP 20).
    The tendency of our models to take advantage of problematic correlations could lead to algorithmic bias. No matter how small the implicit biases are in AI systems (say, a resume filtering system), in the long term they would contribute to inequitable outcomes particularly for the marginalized communities of society. This is an enormous challenge which requires the participation of policy-makers, social scientists, educators, and so on. Likewise, we, AI scientists, should be diligent about addressing problematic aspects of the same technologies we want to sell (EMNLP 20).

  • How can AI benefit the common good? Whether we like it or not, technology will play an increasing role in our daily lives. I tend to worry about the unintended consequences of technology in all areas of our lives. These include, but are not limited to, algorithmic biases, misinformation, echo chambers and our growing socio-political divisions.
    As a believer in technology, I like to see applications of AI that address the potential damages of new technologies. An instance of such mindset was embodied in our project that explored ways to organize ideas about controversial issues in ways that allows people to see alternative views and help them see alternative views to a discussion (NAACL 19).

One of the highlights of my research career was the collaboration with a team of brilliant researchers who had an ambitious goal of building a machine capable of answering elementary school science questions. Just a few years ago, the Aristo project reached its milestone by achieving over 90% in 8th grade science — a milestone for AI and NLP.

I was a doctoral student at Computer and Information Sciences at the University of Pennsylvania (2017-2019) and at the University of Illinois, Urbana-Champaign (2012-2016), under Prof. Dan Roth. I have a bachelor's degree from Amirkabir University of Technology, Tehran Polytechnic (2008-2012). I was greatly fortunate to work with Prof. Hamid Sheikhzadeh during my undergraduate studies.


Disclaimer: This material is presented to ensure timely dissemination of scholarly works. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms invoked by each author's copyright.

  • Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?
    Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sabharwal and Kai-Wei Chang
    ACL-IJCNLP, 2021
    Paper, Data & Code

  • Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions
    Swaroop Mishra, Daniel Khashabi, Chitta Baral and Hannaneh Hajishirzi
    arXiv, 2021
    Paper, Data, Slides

  • GooAQ: Open Question Answering with Diverse Answer Types
    Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi and Chris Callison-Burch
    arXiv, 2021
    Paper, Data

  • Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
    Tushar Khot, Daniel Khashabi, Kyle Richardson, Ashish Sabharwal and Peter Clark
    NAACL-HLT, 2021
    Paper, Resources, Demo, Slides

  • Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
    Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord and Peter Clark
    arXiv, 2021
    Paper, Data

  • GENIE: Leaderboard for Human-in-the-Loop Evaluation of Text Generation
    Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith and Daniel S. Weld
    arXiv, 2021
    Paper, Website, Media coverage: VentureBeat

  • ParsiNLU: A Suite of Language Understanding Challenges for Persian
    Daniel Khashabi, Arman Cohan, Siamak Shakeri, et al.
    arXiv, 2020
    Paper, Code & Data

  • Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth and Jonathan Berant
    TACL, 2021
    Paper, Dataset, Leaderboard, Slides

  • UnQovering Stereotypical Biases via Underspecified Questions
    Tao Li, Tushar Khot, Daniel Khashabi and Ashish Sabharwal
    EMNLP - Findings, 2020
    Paper, Code, Demo

  • UnifiedQA: Crossing Format Boundaries With a Single QA System
    Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark and Hannaneh Hajishirzi
    EMNLP - Findings, 2020
    Paper, Resources, Demo, Slides

  • Evaluating Models’ Local Decision Boundaries via Contrast Sets
    Matt Gardner et al.
    EMNLP - Findings, 2020
    Paper, Data

  • More Bang for Your Buck: Natural Perturbation for Robust Question Answering
    Daniel Khashabi, Tushar Khot and Ashish Sabharwal
    EMNLP, 2020
    Paper, Data, Slides, Talk

  • Commonsense Knowledge Discovery from Linguistic Graphs
    Hongming Zhang, Daniel Khashabi, Yangqiu Song and Dan Roth
    IJCAI, 2020
    Paper, Data

  • From ‘F’ to ‘A’ on the N.Y. Regents Science Exams: An Overview of the Aristo Project
    Peter Clark et al.
    AI Magazine, 2020
    Paper, Talk Media coverage: NYTimes, GeekWire, Vox, Forbes

  • Temporal Common Sense Acquisition with Minimal Supervision
    Ben Zhou, Qiang Ning Daniel Khashabi and Dan Roth
    ACL, 2020
    Paper, Code, ACL slides

  • Not All Claims are Created Equal: Choosing the Right Approach to Assess Your Hypotheses
    Erfan Sadeqi Azer, Daniel Khashabi, Ashish Sabharwal and Dan Roth
    ACL, 2020
    Paper, Software, Slides, Slides2, ACL slides

  • “Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding
    Ben Zhou, Daniel Khashabi, Qiang Ning and Dan Roth
    EMNLP, 2019
    Paper, Dataset, Leaderboard, Slides, Talk

  • PerspectroScope: A Window to the World of Diverse Perspectives
    Sihao Chen, Daniel Khashabi, Chris Callison-Burch and Dan Roth
    ACL - Demos, 2019
    Paper, Video, Code, Poster

  • Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
    Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch and Dan Roth
    NAACL, 2019
    Paper, Dataset, Code, Poster, Video

  • On the Possibilities and Limitations of Multi-hop Reasoning Under Linguistic Imperfections
    Daniel Khashabi, Erfan Sadeqi Azer, Tushar Khot, Ashish Sabharwal and Dan Roth
    arXiv, 2019

  • Zero-Shot Open Entity Typing as Type-Compatible Grounding
    Ben Zhou, Daniel Khashabi, Chen-Tse Tsai and Dan Roth
    EMNLP, 2018
    Paper, Code, Poster

  • Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
    Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay and Dan Roth
    NAACL, 2018
    Paper, Dataset, Poster

  • CogCompNLP: Your Swiss Army Knife for NLP
    Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos et al.
    LREC, 2018
    Paper, Code, Poster

  • Question Answering as Global Reasoning over Semantic Abstractions
    Daniel Khashabi, Tushar Khot, Ashish Sabharwal and Dan Roth
    AAAI, 2018
    Paper, Code, Slides-1, Slides-2

  • Relational Learning and Feature Extraction by Querying over Heterogeneous Information Networks
    Parisa Kordjamshidi, Sameer Singh, Daniel Khashabi, Christos Christodoulopoulos, Mark Summons, Saurabh Sinha and Dan Roth
    Seventh International Workshop on Statistical Relational AI (StarAI), 2017
    Paper, Code, Poster

  • Learning What is Essential in Questions
    Daniel Khashabi, Tushar Khot, Ashish Sabharwal and Dan Roth
    CoNLL, 2017
    Paper, Code, Poster, Spotlight

  • Better call Saul: Flexible Programming for Learning and Inference in NLP
    Parisa Kordjamshidi, Daniel Khashabi, Christos Christodoulopoulos, Bhargav Mangipudi, Sameer Singh and Dan Roth
    COLING, 2016
    Paper, Slides, Code

  • Question Answering via Integer Programming over Semi-Structured Knowledge
    Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni and Dan Roth
    IJCAI, 2016
    Paper, Code, Demo, UW talk, IJCAI talk, Poster

  • EDISON: Feature Extraction for NLP, Simplified
    Mark Sammons, Christos Christodoulopoulos, Parisa Kordjamshidi, Daniel Khashabi, Vivek Srikumar and Dan Roth
    LREC, 2016
    Paper, Poster

  • Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
    Peter Clark et al.
    AAAI, 2016

  • Online Learning with Adversarial Delays
    Kent Quanrud and Daniel Khashabi
    NourIPS, 2015
    Paper, Poster

  • Illinois-Profiler: Knowledge Schemas at Scale
    Zhiye Fei, Daniel Khashabi, Haoruo Peng, Hao Wu and Dan Roth
    IJCAI Workshop on Cognitive Knowledge Acquisition and Applications (Cognitum 2015)
    Paper, Slides, Poster

  • Solving Hard Co-reference Problems
    Haoruo Peng, Daniel Khashabi and Dan Roth
    NAACL, 2015
    Paper, Slides, Poster

  • Adaptive Tiled Neural Networks
    Mohammad Nokhbeh-Zaeem et al.
    in IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2011
    Paper, Code


  • Image demosaicing
    Sebastian Nowozin, Daniel Khashabi, Jeremy Jancsary, Bruce Lindbloom, Andrew Fitzgibbon, Microsoft Technology Licensing LLC
    United States patent US 9,344,690. 2016 May 17.


  • Reasoning-Driven Question-Answering for Natural Language Understanding
    PhD Thesis, University of Pennsylvania, 2019
    PDF, Slides

  • Analysis and Implementation of Bayesian Methods to Model Correlated Information (Multitask Learning)
    BSc Thesis, Tehran Polytechnic, 2012 (in Persian.)
    PDF, Slides

Select Talks


  • Guest Lecturer: CS446 (Machine Learning) @ UPenn, Spring 2018.

  • Guest Lecturer: CS446 (Machine Learning) @ UPenn, Fall 2018: Lecture 1 Lecture 2

  • Guest Lecturer: CS446 (Machine Learning) @ UIUC, Spring 2016: Lecture 1

  • Guest Lecturer: CS446 (Machine Learning) @ UIUC, Fall 2015: files Lecture 1 Lecture 2

  • Teaching Assistant: CS446: Machine Learning, Dan Roth, UIUC (Fall, 2015).

  • Teaching Assistant: CS473: Fundamental Algorithms, Jeff Erikson, UIUC (Fall, 2013).

  • Teaching Assistant: CS473: Fundamental Algorithms, Sariel Har-Peled and Alexandra Kolla, UIUC (Spring, 2013).

  • Teaching Assistant: “Probability and Statistics”, Instructor: Dr. Gholamreza Moradi, AUT (Spring, 2012).

  • Teaching Assistant: “Digital Signal Processing”, Instructor: Dr. Hamid Sheikhzadeh, AUT (Spring 2012).

  • Teaching Assistant: “C++ II : Numerical C++ Programming”, Instructor: Dr.Bahram Taheri, Joint program with University of Birmingham and AUT (Fall, 2011).

  • Teaching Assistant: “Introduction to Computers and Programming(C++)”, Instructor: Dr.Hassan Taheri (Spring, 2011).

Past Projects and Resources