cloud-cv / evalai Goto Github PK

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

License: Other

Python 45.96% HTML 16.03% JavaScript 16.17% CSS 0.97% Shell 0.87% Dockerfile 0.17% TypeScript 13.61% SCSS 6.23%

ai machine-learning django angularjs python ai-challenges docker reproducible-research reproducibility evaluation

evalai's Introduction

EvalAI is an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence (AI) algorithms at scale.

In recent years, it has become increasingly difficult to compare an algorithm solving a given task with other existing approaches. These comparisons suffer from minor differences in algorithm implementation, use of non-standard dataset splits and different evaluation metrics. By providing a central leaderboard and submission interface, we make it easier for researchers to reproduce the results mentioned in the paper and perform reliable & accurate quantitative analysis. By providing swift and robust backends based on map-reduce frameworks that speed up evaluation on the fly, EvalAI aims to make it easier for researchers to reproduce results from technical papers and perform reliable and accurate analyses.

Features

Custom evaluation protocols and phases: We allow creation of an arbitrary number of evaluation phases and dataset splits, compatibility using any programming language, and organizing results in both public and private leaderboards.
Remote evaluation: Certain large-scale challenges need special compute capabilities for evaluation. If the challenge needs extra computational power, challenge organizers can easily add their own cluster of worker nodes to process participant submissions while we take care of hosting the challenge, handling user submissions, and maintaining the leaderboard.
Evaluation inside environments: EvalAI lets participants submit code for their agent in the form of docker images which are evaluated against test environments on the evaluation server. During evaluation, the worker fetches the image, test environment, and the model snapshot and spins up a new container to perform evaluation.
CLI support: evalai-cli is designed to extend the functionality of the EvalAI web application to your command line to make the platform more accessible and terminal-friendly.
Portability: EvalAI is designed with keeping in mind scalability and portability of such a system from the very inception of the idea. Most of the components rely heavily on open-source technologies – Docker, Django, Node.js, and PostgreSQL.
Faster evaluation: We warm-up the worker nodes at start-up by importing the challenge code and pre-loading the dataset in memory. We also split the dataset into small chunks that are simultaneously evaluated on multiple cores. These simple tricks result in faster evaluation and reduces the evaluation time by an order of magnitude in some cases.

Goal

Our ultimate goal is to build a centralized platform to host, participate and collaborate in AI challenges organized around the globe and we hope to help in benchmarking progress in AI.

Installation instructions

Setting up EvalAI on your local machine is really easy. You can setup EvalAI using docker: The steps are:

Install docker and docker-compose on your machine.

Get the source code on to your machine via git.

git clone https://github.com/Cloud-CV/EvalAI.git evalai && cd evalai

Build and run the Docker containers. This might take a while.
```
docker-compose up --build
```
That's it. Open web browser and hit the URL http://127.0.0.1:8888. Three users will be created by default which are listed below -

SUPERUSER- username: admin password: password
HOST USER- username: host password: password
PARTICIPANT USER- username: participant password: password

If you are facing any issue during installation, please see our common errors during installation page.

Citing EvalAI

If you are using EvalAI for hosting challenges, please cite the following technical report:

@article{EvalAI,
    title   =  {EvalAI: Towards Better Evaluation Systems for AI Agents},
    author  =  {Deshraj Yadav and Rishabh Jain and Harsh Agrawal and Prithvijit
                Chattopadhyay and Taranjeet Singh and Akash Jain and Shiv Baran
                Singh and Stefan Lee and Dhruv Batra},
    year    =  {2019},
    volume  =  arXiv:1902.03570
}

Team

EvalAI is currently maintained by Rishabh Jain, Gunjan Chhablani . A non-exhaustive list of other major contributors includes: Deshraj Yadav, Ram Ramrakhya,Akash Jain, Taranjeet Singh, Shiv Baran Singh, Harsh Agarwal, Prithvijit Chattopadhyay, Devi Parikh and Dhruv Batra.

Contribution guidelines

If you are interested in contributing to EvalAI, follow our contribution guidelines.

Contributors

evalai's People

Contributors

Stargazers

Watchers

Forkers

rishabhjain2018 gitter-badger taranjeet ankitjain28may neeraj12121 sakshi29 itaditya tejaskhot parismita rackstar17 shivanxi jayantsa rishav09 sagn1k virajprabhu sriharsha0806 gauthamzz karthik24iyer arun-jain yadavankit utkbansal igeeks jatinbalodhi96 curious72 aakashchan mayank-agarwal-96 aradhyamathur ashishbaghudana anshulbasia bhavesh-gupta aliraza3997 himaja20 kakshay21 varunagrawal ishapotnis1 prasadwajekar yogeshdarji artkorenev sawon1234 benjamesbabala guyandtheworld insomniac12 singlakdeepak mohitanand001 senorblasto rishabhja rishabhj1994 blitu12345 aayusharora the-dagger yell0wfl4sh hargovindarora s2606 modestlearner jcjohnson tengteng95 cometyang jemisa winnerineast hieuqtran veeramuthub sandeepnk mehrdadmzn codeaudit cclauss tony32769 guptasaloni ayukha reyadrahman liviust ghosthamlet 0x38 sudhanshuchopra saurabhranjan1806 gardlt tlugger aniq55 anirudhvenu sehgalayush1 markslingerland shantanujain zuzanatoldyova kiyominox thobhanifreddy amoshason matejmecka bentglasstube nitinprakash96 yashasingh prajwalgatti mohityadav23 geekyshacklebolt prakharchoudhary iteachmachines kartikeyash center1 kunalvishnoi 0xdaksh chamathabeysinghe shubhpatel108

evalai's Issues

Create endpoint to get details about a particular Challenge

Details about the endpoint:

Url: /api/challenges/challenge_host_team/<challenge_host_team_id>/challenge/<challenge_id>
Header: Authorization: Token
Method: GET
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Setup ModelAdmin class for each model in Team app

This class should be derived from TimeStampedAdmin of base app.

Notable attributes should be

list_display
search_fields
list_filter

Note that this issue depends on #68 which involves the creation of TimeStampedAdmin class.

Create endpoint for updating the challenge

Details about the endpoint:

Url: /api/challenges/<challenge_id>/
Header: Authorization: Token
Method: PUT/PATCH
Response:
- status on success: HTTP_200_OK
- status on exception: HTTP_406_NOT_ACCEPTABLE
- status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Create endpoint to get details about a particular ChallengeHost

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host/<challenge_host_id>
Header: Authorization: Token
Method: GET
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Setup ModelAdmin class for each model in Challenge app

This class should be derived from a TimeStampedAdmin of base app.

Notable attributes should be

list_display
search_fields
list_filter

Note that this issue depends on Issue 68 which involves the creation of TimeStampedAdmin class.

Add Footer module on homepage

For now just add Some Social sharing Icons and some dummy links like about us, contact us etc.

Create endpoint to get details about a particular ChallengeHostTeam

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/
Header: Authorization: Token
Method: GET
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Scroll to details section on home page.

There is an option to scroll up to second section in Landing page by clicking on "Get Insight". Need to implement smooth scroll which will scroll the page with in limit of atmax 2ms.

In case of query, leave your comment here.

Add url names in team apps

Right now the urls defined in team apps have no names. Naming a url actually gives the flexibility of changing the uri very easily if everywhere urls are referred by url tags.

Create endpoint for deleting a challenge

Details about the endpoint:

Url: /api/challenges/<challenge_id>/
Header: Authorization: Token
Method: DELETE
Response:
- status on success: HTTP_200_OK
- status on exception: HTTP_406_NOT_ACCEPTABLE
- status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Create endpoint to delete a team

Details about the endpoint:

Url: /api/teams/<team_id>/
Header: Authorization: Token
Method: DELETE
Response:
- status on success: HTTP_200_OK
- status on exception: HTTP_406_NOT_ACCEPTABLE
- status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Improve Gulp Watch task to handle remove, delete, and errors in files.

Need to update watch task of gulp so that it can easily get files update like delete and add.

Create endpoint to get details about a particular team

Details about the endpoint:

Url: /api/teams/<team_id>/
Header: Authorization: Token
Method: GET
Response:
- status on success: HTTP_200_OK
- status on exception: HTTP_406_NOT_ACCEPTABLE
- status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Add new Gulp task to build static files for prod.

Currently, Gulp is performing all tasks related to development. Need to make a new task so that it can minify all angular related and other static files (like CSS, Javascript) for production.

Task should minify all the files and can also do concatenation wherever needed.

Add Forgot password feature

Need to add a Password Change (forgotten password) feature. Refer API docs for change password endpoints

Create endpoint for creating a new Challenge

Details about the endpoint:

Url: /api/challenges/
Header: Authorization: Token
Method: POST
Response:
- status on success: HTTP_201_CREATED
- status on exception: HTTP_406_NOT_ACCEPTABLE
- status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Remove Remember Me option from login form.

Remove the Remember Me option and shift the Login button in place of it.

Setup logging in django

For more information, see this https://docs.djangoproject.com/en/1.10/topics/logging/

Move TimeStampedModel out of accounts app to base app

This will involve updating the import statements also.

For eg, import statements like
from accounts.models import TimeStampedModel

to something like

from base.models import TimeStampedModel

Add autofocus to login and signup page input box.

Autofocus username field in login and signup page.

Integrate Django REST Swagger for API docs

For reference, see this: https://django-rest-swagger.readthedocs.io/en/latest/

Create endpoint to delete a ChallengeHost

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host/<challenge_host_id>
Header: Authorization: Token
Method: DELETE
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Create endpoint for creating a ChallengeHostTeam

Details about the endpoint:

Url: /api/hosts/challenge_host_team
Header: Authorization: Token
Method: POST
Response:
status: HTTP_201_CREATED

Rename model ChallengeHostTeams to ChallengeHostTeam

model name should be singular.

Create a new app named as `base`

This app will contain all the models which will be used accross the project. Also it will be the placeholder for many mixins.

This app will not have an entry in urls as this will be the placeholder for many functions, mixins and views(using as a package)

Set configurations to store media files on S3

Create endpoint to delete a ChallengeHostTeam

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/
Header: Authorization: Token
Method: DELETE
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Create endpoint to get all the teams

Details about the endpoint:

Url: /api/teams/challenge/<challenge_id>
Header: Authorization: Token
Method: GET
Response:
- status: HTTP_200_OK

Also, please add the pagination for the same.

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Update status and permission fields in ChallengeHost model

Ideally status and permissions should be CharField instead of being foreign key fields to respective models. Reason for this being that at a time a particular challenge host will have a single status. So foreign key actually makes no sense here.

ChallengeHostPermission and ChallengeHostStatus should be removed.

Also while updating, make sure that string in each options(for status and permission) follows title case convention.

Setup token based authentication in DRF

Rename created and modified to created_at and modified_at

This actually makes it more clear, that we are talking about time. Concerned model is TimeStampedModel.

Setup ModelAdmin class for each model in Participant app

This class should be derived from a TimeStampedAdmin of base app.

Notable attributes should be

list_display
search_fields
list_filter

Note that this issue depends on Issue 68 which involves the creation of TimeStampedAdmin class.

Add content to home page

Need to complete the Homepage:

Main content that will come:

Top 5 challenges
Intro to EvalAI
How it works

Create endpoint to get list of all the challenges

Details about the endpoint:

Url: /api/challenges/
Header: None
Method: GET
Response:
- status on success: HTTP_200_OK
- status on exception: HTTP_406_NOT_ACCEPTABLE
- status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Remove challenge field from ChallengeHostTeam model

Challenge model has FK on ChallengeHostTeam as creator field.

ChallengeHostTeam model has FK on Challenge model as challenge field.

Ideally the foreign key(FK) should be on Challenge because a single team will be creating numerous challenges, not the other way around.

Create endpoint for creating a Team

Details about the endpoint:

Url: /api/teams/
Header: Authorization: Token
Method: POST
Response:
- status: HTTP_201_CREATED

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Create end point to get all the challenge host team

Details about the endpoint:

Url: /api/hosts/challenge_host_team
Header: Authorization: Token
Method: GET
status: HTTP_200_OK

Support for pagination should be there.

SMTP server not working for sending mails

When a user registers, then a mail needs to be sent to the user's email id.

The current error is:

SMTPServerDisconnected at /api/auth/registration/
please run connect() first

Create endpoint to update a particular ChallengeHostTeam details

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/
Header: Authorization: Token
Method: PUT/PATCH
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Create endpoint for creating a ChallengeHost

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host
Header: Authorization: Token
Method: POST
Response:
status: HTTP_201_CREATED

Fix choices option for status and permission field of ChallengeHost model.

choices are status and permissions are presently swapped. This needs to be updated.

Create end point to get all the challenge host of a particular challenge host team

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host
Header: Authorization: Token
Method: GET
Response:
status: HTTP_200_OK

Support for pagination should be there.

Typo in README

The line That's it, Now to connect to dev server at http://127.0.0.1:8000 needs to be replaced with That's it, Now to connect to dev server at http://127.0.0.1:8888

Setup ModelAdmin class for each model in Host app

This class should be derived from a TimeStampedAdmin of base app.

Notable attributes should be

list_display
search_fields
list_filter

Note that this issue depends on Issue 68 which involves the creation of TimeStampedAdmin class.

Create a ModelAdmin class in base app to exclude created and modified fields

The admin class should live in base apps. It will be derived from ModelAdmin class and will use exclude to exclude created and modified fields.

This is so that anyone adding an object from the admin panel should not be shown in these fields as these are handled internally by Django before saving an object.

Name of the class should be TimeStampedAdmin and it should live in base/admin.py

Add created_by in ChallengeHostTeam

With the existing design, it is actually very different to fetch ChallengeHostTeam objects of a particular user. One of the hacks that can be used to solve this are

fetch challenge host object of current user with permission level as admin and then query by it(select_related) to ChallengeHostTeam. But this is very much for a simple use case.

Adding this column will help us deal with the use case of fetching all the challenge host team object of a single user.

Create endpoint to update the team details

Details about the endpoint:

Url: /api/teams/<team_id>/
Header: Authorization: Token
Method: PUT/PATCH
Response:
- status on success: HTTP_200_OK
- status on exception: HTTP_406_NOT_ACCEPTABLE
- status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Rewrite TeamSerializer to contain context logic

Ideally TeamChallengeSerializer and TeamSerializer should be a single serializer.

One such example for reference can be ChallengeHostTeamSerializer in apps/hosts/serializers.py

Create endpoint to update a particular ChallengeHost details

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host/<challenge_host_id>
Header: Authorization: Token
Method: PUT/PATCH
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Broken link in readme.md

There is a link in readme.md line 84 to https://github.com/Cloud-CV/EvalAI/blob/development/CONTRIBUTING.md

cloud-cv / evalai Goto Github PK

evalai's Introduction

Features

Goal

Installation instructions

Citing EvalAI

Team

Contribution guidelines

Contributors

evalai's People

Contributors

Stargazers

Watchers

Forkers

evalai's Issues

Recommend Projects

Recommend Topics

Recommend Org