Dr. Detective is an online game for annotating medical texts. The game is designed with the purpose of engaging medical experts into solving annotation tasks on medical case reports, tailored to capture disagreement between annotators. It incorporates incentives such as learning features, to motivate a continuous involvement of the expert crowd.
The game was designed to identify expressions valuable for training natural language processing (NLP) tools, and interpret their relation in the context of medical diagnosing. In this way, we can resolve the main problem in gathering ground truth from experts that the low inter-annotator agreement is typically caused by different interpretations of the text.
For further details on Dr. Detective game, please refer to "Dr. Detective": combining gamification techniques and crowdsourcing to create a gold standard in medical text. A snapshot of game statistics taken on June 2014 can be found here. Up-to-date game statistics can be found here.