The interface should allow to easily expire/delete all session HITs.

Instructions should walk through a full cycle:
1. Downloads and local installation
2. Uploading code to the web server
3. Uploading data to the web server
4. Running annotation
5. Downloading results
6. Parsing results
7. Approving

The UI should have a representation of the task queue. The user can
navigate between the tasks in his/her queue. They can go the the previous
task and they can skip the current task.

The task management component should keep a window over the queue.

Caching and autosubmit are described elsewhere.

Once a single polygon is finished, all the controls disappear.

Allow to give an icon URL instead of the name to represent objects in the
interface. The interface will show a button with a small image of, say, a
cup. The text won't be present.

Allow to specify long text in the checkboxes.

Objects count. Image counts per category.

The annotations should be saved to the server frequently to avoid data
loss. The UI should support auto-save (saving the complete annotation at
frequent intervals), save button (auto-save upon request).

"Saving in progress" indicator will show that the UI is currently saving
the data to the server. 

"Saving due" indicator will show that the data has been modified since the
last auto-save.

- Prevent repeated submission – on a dataset by dataset basis
- Specified by number N in time T
- Administrator can allow extra for individuals i.e. for a period dT the
user will be allowed M extra submissions
- The exception must be removable/editable at any time

The interface should support the display of a large number of objects. The
 names and the attributes of the objects should be visible and easy to read.

Options for display:
 # Global variable for a level of detail
 # Per-category variable for a level of detail (when we focus on a single
class, all others fade away)
 # Focus on specific object instance
 # Iconic representation of object categories and object attributes
 # Unique (pre-selected or random) color per object category. 
 # Automatically selected random icons for object attributes.

Hovering over the object can highlight it's name and attributes.

The user gives a URL and an MD5 sum. The server registers the submission
and downloads it asynchronously. The problem is reported if MD5 sums don't
match.

The system should support announcements. When important changes to the
annotation protocol are made, the organizers will activate the
announcement. Every annotator should see a prominent message (at least
once), so that they know important changes have been made.

Compute MD5 sum and stick it into report. This will show the user, what
they have uploaded.

What steps will reproduce the problem?
1. please lookup the hands-1 HIT in MTurk for the problem

Upload a single image to a session.

Upload a tar with image to a session.

The current interface is modal. The user is locked into the task until it's
completed. Non-modal interface presents a perception that the user can
start any task from any point in time. 

The upgrade to non-modal interface will include:
 1. Top level controls are visible at all times. 
 1. The user may start a different task or switch between tasks with a
single click. 
 1. When the person changes the object type, the tool will change the
object category of the current object (optional).

When selecting the object, they should be ordered small to large. Small
objects should receive the focus first and then pass the focus to the
larger objects.

The tool will have a prompt for the image URL, task XML and annotation XML.
It will then load them, render the GUI and show the task.

Update and cleanup the example.

Send mail to specified address(es) when users register.

The size of the interface depends on the size of the screen.

The html page should provide a slider to change the size of the UI. It will
use cookies to store the latest size, so that when the page is reloaded it
doesn't change the pre-selected size of the control.

Object resizing works poorly.

This will decouple from the bounding box.

The author of the annotations should be able to completely remove the
annotations from the system.

Show a video, have a timeline of events.

Add events from the dictionary. Edit event boundaries.

Release the tool that supports:
1. 1-layer annotation
2. multiple-layer annotations
3. segmentation 

The tool will support input and display (no editing).

We need to upload existing annotations in the Pascal VOC format to the server.

The most plausible approach is to:
 1. create a fixed annotation task that captures all data in the VOC format;  
 1. convert VOC data to the representation consistent with (1)
 1. upload the created annotations to the server.

We'll have to authenticate (check the password) and authorize (check group
membership) the upload.

Keep track of whether the worker seen the instructions. Show a div if they
haven't.

Ability to download an Excel file with all submissions.

We need to keep the source and licenses for the images.

The public/non-public separation will allow to hide the annotations (e.g.
pascal ground truth) or keep uploads private.

This will allow to decompress the segmentation on the server w/o the
enclosing annotation.

Bounding boxes can be started from any corner and finished at any corner.

The author of the annotations should be able to completely remove the
annotations from the system.

Evaluation code issue – where too many/not enough results were submitted –
but should give warning. Upgrade to newer Mark's code?

The codebase has diverged. A merge is necessary.

The task is to create an image with known drawings (line, circles, elipses,
etc) and mark it with the annotation tool. The test is how precise the
annotations match the correct data.

The tool should support editing mode - open existing annotation, make
changes, save the result. The result should be either saved with the same
name or under different name.

Bounding boxes should allow adjustment when in the editing mode.

The server instructions should explain:
1. required software
2. installation instructions
3. basic use cases

When the data is in the edit mode, the polygon control points should be
movable.

The interface allows to create single pixel objects and it's impossible to
remove them nor to change their size.

Each task in the queue should have a priority. The next task is assigned to
a worker based on the priority of that task.

The priority mechanism will allow the organizers to speed-up the annotation
of some images and delay (or cancel) the annotation of other images.

The interface must check version of the tool required in the task
definition. If the version doesn't match, a warning is issued with
instructions how to upgrade.

Precision/recall curves – should be downloadable in some suitable format
e.g. csv – number of points must be configurable

The user interface should cache the upcoming task images to speed up the
annotation process and increase task interactivity.

The cache will keep 3-5 upcoming tasks and choose the one with highest
priority to show next.

The annotation system can utilize some external chat service (e.g.
http://tinychat.com/) for quick Q.&A. 

Submission registration will allow to input the data once and change it as
necessary.

Upload will be a simple reference to the submission.

1. Create annotation tasks
2. Select the next task for a particular user
3. Lock task to the user (with timeout)
4. Track task progress

Show multiple annotations on a single page.

Submit each annotation separately.

When all annotations are done, submit the whole page.