Conference paper · WWW '25 · 2025

Alan's Speakeasy — An Ecosystem for the Evaluation of Conversational Agents

Luca Rossetto, Cristina Sarasua, Oana Inel, Juan Diego Bermeo, Pablo Sebastian Bolaños, Hugues Devimeux, Huiran Duan, Robin Hany, Athina Kyriakou, Dhivyabharathi Ramasamy, Varun Ghat Ravikumar, Konstantina Timoleon, Rosni Vasu, Ruijie Wang, Daan van der Weijden, Jan Willi, Abraham Bernstein

Companion Proceedings of the ACM Web Conference 2025 (WWW '25), 2025

PDF DOI Code Link

Abstract

Speakeasy is an ecosystem designed for evaluating and comparing conversational agents (bots). It provides a web-based platform that generates chatrooms where humans and conversational agents interact, enabling a Turing-test-like evaluation. Speakeasy also includes a set of artificial agents, called Assessor Bots, to assist in the development and evaluation of conversational bots, along with customizable feedback mechanisms and data-analysis tools.

Cite

@inproceedings{rossetto2025alan,
  title={Alan's Speakeasy--An Ecosystem for the Evaluation of Conversational Agents},
  author={Rossetto, Luca and Sarasua, Cristina and Inel, Oana and Bermeo, Juan Diego and Bola{\~n}os, Pablo Sebastian and Devimeux, Hugues and Duan, Huiran and Hany, Robin and Kyriakou, Athina and Ramasamy, Dhivyabharathi and Ravikumar, Varun Ghat and Timoleon, Konstantina and Vasu, Rosni and Wang, Ruijie and van der Weijden, Daan and Willi, Jan and Bernstein, Abraham},
  booktitle={Companion Proceedings of the ACM on Web Conference 2025},
  pages={2895--2898},
  year={2025},
  doi={10.1145/3701716.3715165}
}