Giter Club home page Giter Club logo

chahub's Introduction

CodaLab logo Circle CI codecov

What is CodaLab?

CodaLab is an open-source web-based platform that enables researchers, developers, and data scientists to collaborate, with the goal of advancing research fields where machine learning and advanced computation is used. CodaLab helps to solve many common problems in the arena of data-oriented research through its online community where people can share worksheets and participate in competitions.

To see Codalab Competition's in action, visit codalab.lisn.fr.

Codabench, the next-gen of CodaLab Competitions, is out. Try it out!

Documentation

Community

The CodaLab community forum is hosted on Google Groups.

Quick installation (for Linux!)

To participate in competitions, or even organize your own competition, you don't need to install anything, you just need to sign in an instance of the platform (e.g. this one). If you wish to configure your own instance of CodaLab competitions, here are the instructions:

Install docker and add your user to the docker group, if you haven't already

$ wget -qO- https://get.docker.com/ | sh
$ sudo usermod -aG docker $USER

Clone this repo and get the default environment setup

$ git clone https://github.com/codalab/codalab-competitions
$ cd codalab-competitions
$ cp .env_sample .env
$ pip install docker-compose
$ docker-compose up -d

Now you should be able to access http://localhost/

More details on how to configure your own instance:

License

Copyright (c) 2013-2015, The Outercurve Foundation. Copyright (c) 2016-2021, Université Paris-Saclay. This software is released under the Apache License 2.0 (the "License"); you may not use the software except in compliance with the License.

The text of the Apache License 2.0 can be found online at: http://www.opensource.org/licenses/apache2.0.php

Cite CodaLab Competitions in your research

@article{codalab_competitions_JMLR,
  author  = {Adrien Pavao and Isabelle Guyon and Anne-Catherine Letournel and Dinh-Tuan Tran and Xavier Baro and Hugo Jair Escalante and Sergio Escalera and Tyler Thomas and Zhen Xu},
  title   = {CodaLab Competitions: An Open Source Platform to Organize Scientific Challenges},
  journal = {Journal of Machine Learning Research},
  year    = {2023},
  volume  = {24},
  number  = {198},
  pages   = {1--6},
  url     = {http://jmlr.org/papers/v24/21-1436.html}
}

chahub's People

Contributors

brendanmyers88 avatar ckcollab avatar dependabot[bot] avatar jimmykodes avatar logan-ruf avatar reprokillbot avatar tthomas63 avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

chahub's Issues

Producers, profiles, association discussion/ideas

We may want to attach competitions/such with users authed on ChaHub.....................

I am trying to add the idea of a Producer to ChaHub, which is any server (i.e. competitions) which produces data for ChaHub. What if we want to associate a user profile as the owner of this competition, how should we go about that?

Lay foundation

  • Be able to connect to OrcID or github or something like that
  • Be able to register an account manually
  • Create base schemas
  • Be able to search through data

Info to add to header


competitions
competition organizers
registered users
active users (made entries in the last 3-6 months or so)```

Is Active

Is active is returning more results with a query than without one.

Un-Published Competitions

Un-Published Competitions are currently pushed

image

We should:

  • double check the logic on the is_valid check
  • check edge cases:
    • What happens if someone publishes a competition, then un-publishes it. It will still have an entry but not a valid link

Next Deadline

Next Deadline is currently not working

  • Not displaying
  • Filter does not work

Add Logo/Footer/Etc

  • Add Chahub logo to upper right
  • Add codalab like footer
  • Add contact
  • Add link for Github issues
  • Verbosely highlight that Chahub is a service of/for Codalab

Allow Filtering by Producer

Allow search results to be filtered by producer:

IE: Only from codalab.lri.fr or competitions.codalab.org

Prize Amount

Add prize amount to the competition tile and add filtering

Have a one-off form for submitting competitions manually

Form should have these fields:

  1. title
  2. description
  3. url
  4. start
  5. end
  6. organizer email

Form should submit to an approval process, maybe emailing a few selected people, and then following a link "approves" the competition and adds it to ChaHub. This competition can be manually changed by administrators, or maybe through some admin interface in the future. The user will have to register before submitting the competition to be able to edit it in the future, or at least they will need to provide an email we can track and verify for editing permissions.

Auth/users

Design

What if I accidentally end up with 2 accounts on Chahub?

We should have a protocol for allowing you to completely delete old/unwanted accounts. All submissions would be marked as made by "missing user" at that point.

How can I reclaim my lost submissions/competitions from deleting an old account on Chahub?

We could have a protocol for re-assigning datapoints (competition, submission, task, solution, etc.) to a given user. This could be in User Management section of Chahub to "reassign" a user -- all of their datapoints would point to some new user and the old user would be removed.

What happens if you "login with Chahub" to an existing Chahub and already have an account on Competitions?

It will connect with OAuth your account on Competitions to Chahub if Chahub email matches an email from Competitions. It will not make a new account on Competitions. If your email on Competitions is different from your Chahub login, there is no way for Chahub to match you to that account on competitions.

We could have a button once logged in to "link with Chahub" your profile?

How does a site add a "sign in with Chahub" button

We have to register them in python-social-auth and give them a client secret/etc. to generate OAuth tokens for users

How will hall of fame work with users spread across so many servers?

Users will be uniquely identified by their email. Many emails may belong to one account, but the email must be unique across Chahub. When a submission is finished it is sent to Chahub and is associated with a user.

To even associate submissions with a User, how do we get Users?

Upon confirming your User account by email, the User will be sent to Chahub.

Race condition: what if someone signs up and makes a submission, but does not exist yet on Chahub yet?

Chahub will raise an error and return that it was not able to successfully add the User. The User model and submission will be marked for re-trying. Every hour or so the server attempts to re-submit failed sends, and should resend the Submission and User.

To make it not a permanent failure loop we will make sure USERS are sent/processed before SUBMISSIONS! It's not the biggest deal, because if we do retries properly User should get caught even if it arrives after, then the next send should contain the Submission that is now valid.

Spec

Chahub

  1. Setup django-oauth-toolkit again, if it's not still there
  2. Ability to register on Chahub
    1. Registration should require email activation, at least on prod
    2. Does reset password work?
  3. User model should be able to accommodate many emails
    1. Remove field email and add emails which can be a Postgres array of strings?
  4. User management:
    1. Ability to delete account
    2. Ability to assign a user to an existing user, overriding the old user with the new master user
    3. The above "Reassigning" should require a confirmation prompt where you type in the user's name you are deleting

Codalab v1.5

  1. Setup python-social-auth
    1. Make sure Chahub returns the information necessary to generate user: email, username
  2. Make a Chahub adapter
  3. Add a "login with Chahub" link in header
  4. Add a "login with Chahub" link in login form
  5. Make ClUser a ChahubMixin
  6. For existing users, we should have a "link with Chahub" button that automatically creates an account on Chahub with the same email and initializes a "password reset" process to enter in a new password... is this overkill?

Codalab v2

Still in dev, will implement these as necessary

Tile info

Platform name, Start, End, # of participants, next deadline date

image

Add to header new information

Add in header: the current number of:

  1. competition
  2. competition organizers
  3. registered users
  4. active users (made entries in the last 3-6 months or so)

Various Issues

  • Fuzziness too fuzzy? (Possibly boost the title field more)
  • Competition Tile organization/layout tweaks
  • Possibly strip html/js from description. (Check on v1.5 if you can imbed js in the description)

Clear old entries

Clear the database of old entries.

Competitions from before valid checks, is_active logic, and other changes may still be displaying

Chahub Wireframe Phase 2

  • Establish QA Review from Phase 1 Wireframe
  • Implement Necessary Changes
  • Ensure all pages have proper UX Flow

Widgets

Create embeddable widgets for v1.5 search: IE: Replace the current database query search with a widget that utilizes Chahub

Sorting By Participants

Sorting by participants currently is not taking all competitions into account. IE: AutoML has more participants than the default query sorted by participants.

Retries

When sending to chahub, set timestamp to None until we eventually get a valid response. We can re-try this with a celery task every ~5 minutes for each chahubmixin class that fails. We'll need to somehow set each of these classes in some global list to check against.

I.e. Competition(ChaHubMixin) and Submission(ChaHubMixin) might both need retries, so when we query we'll have to check like...

for model in ChaHubModels:
    needs_retry = model.objects.filter(chahub_needs_retry=True)
    for instance in needs_retry:
        # Saving forces chahub update
        instance.save()

Random logos

Instead of displaying the same default, generate a random logo like GitHub does for each profile. This will make it look better when scrolling and such through RAMP results

Crawler receiving many competitions, without having to locally store them

The crawlers will be simpler if they require no storage, we can leverage Chahub to remember which competitions have been seen or not. Instead of sending competitions one-by-one, we would work with 2 endpoints:

  1. competition-existence-check/ which takes a JSON list of competition REMOTE ids and returns the ids that do not yet exist on Chahub
  2. modify competition/ POST to accept a list.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.