Back to search

Georgia Institute of Technology

EvalAI: Evaluating AI systems at scale

Abstract

dc:description.abstract

Artificial Intelligence research has progressed tremendously in the last few years. There has been the introduction of several new multi-modal datasets and tasks due to which it is becoming much harder to compare new algorithms with existing ones. To solve this problem, this thesis introduces EvalAI, an open source platform for evaluating and comparing machine learning and artificial intelligence algorithms at scale. This platform is built to provide an open source, standardized, scalable solution for evaluating learned models using automatic metrics as well as with human-in-the-loop evaluation. By simplifying and standardizing the process of benchmarking, EvalAI seeks to lower the barrier to entry for participating in the global scientific effort to push the frontiers of machine learning and artificial intelligence, increasing the rate of measurable progress in these communities.

Degree

thesis:*
Level thesis:degree_level
Masters
Department dc:contributor.department
Computer Science
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2018

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Deshraj
Advisors dc:contributor.advisor
  • Parikh, Devi
  • Batra, Dhruv
Committee members dc:contributor.committeemember
  • Parikh, Devi
  • Lee, Stefan

Subjects

dc:subject × 10

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1853/60738

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Deshraj. EvalAI: Evaluating AI systems at scale. Masters thesis, Georgia Institute of Technology, 2018. http://hdl.handle.net/1853/60738