autolab / tango Goto Github PK

Initially mentioned here: autolab/Autolab#169

If the VMs on a backend machine become wedged, Tango stalls indefinitely during the retry waiting for destroyVM to finish. Instead, Tango should timeout on waitVM, create a new instance, and retry waitVM on that instance. Eventually, the wedged machine will fill up with VMs, but this will allow Tango to degrade gracefully while we are waiting to restart the backend.

To help reduce jitter on performance-sensitive assignments like malloc, we need a new docker vmms that will allow us to specifify a list of physical machines, and that schedules at most one job per physical machine.

The entire directory needs to be re-spaced.

The 213 datalab is taking about 7-8 seconds to run, with up to 3-4 seconds required for the scp-based copyin step, which should be almost instantaneous. In the old system, datalab took 2-3 seconds. In the past, this kind of slow-down has been due to authentication issues that required the ssh client to retry using different authentication schemes.

It would be easy to diagnose if there were a way to tell Tango to call ssh with the verbose option (-vvvv) and then dump the ssh output to the tango log.

We need to update Tango and the command line support to support the ability to delete any job in the job queue.

Hi there,

I came across an issue where if I deleted an assessment in Autolab and reuploaded it, Tango would grade the latest file submitted to the previously deleted assessment when a new file was submitted to the reuploaded assessment (even if the files differ). Since there was an update in commit 050e5fc which had to do with MD5 checking, I updated Tango to see if this solved my issue. Unfortunately, this lead me to this error in Autolab:

After doing some digging in the source, it seems like the output from Tango's open() function in tangoREST.py is incompatible with what Autolab expects since the commit mentioned above. As you can see from the screenshot, the error finally occurs in tango_upload in Autolab's autograde.rb, but is due to the existing_files variable which is obtained from a TangoClient.open(..) call.

Thanks for looking into this.

This would be nice for devs and anyone who wants to use Redis.

Job2 should timeout and runJob should return -1. Instead, the return code is 3, which corresponds to an OS error and results in a catastrophic retry.

Make the response interesting.

It'd be great if developers connecting to Tango could understand the error codes returned from Tango. (e.g. -2)

If volumes directory is not created and config is *, then it should be automatically created.

There is a corner case (experienced by the 15-381) TA that causes Tango to improperly use a previously cached version of a submission file.

In the feedback that Tango returns to the client, it should be using local times rather than UTC. For example, I submitted this job at 11:07am EST:

Autograder [Thu Jan 15 16:07:46 2015]: Received...
Autograder [Thu Jan 15 16:07:50 2015]: Success: Autodriver returned normally
Autograder [Thu Jan 15 16:07:50 2015]: Here is the output from the autograder:

I have encountered a couple of situations in testing where the preallocator's machines datastructure got out of sync. There were machine ids in the list that no longer had an entry in the relevant queue, which prevented any jobs from being scheduled on that machine. If all machines get in that state, job scheduling would halt. #77 fixes some of these, but it probably makes sense to either reset the preallocator state or validate it when tango is reset.

If the user specifies a non-existing VM image, Tango should return immediately with a validation error. It doesn't. Instead, Tango runs waitVM three times in an unsuccessful attempt to start a non-existing VM image.

Tango should be doing extensive validation of job requests, which it apparently isn't.

We need the ability to delete a job from the job queue, stop its worker thread, and free its instance.

Implement a "local-docker" option that autogrades on the localhost in a docker container.

Currently, jobs are removed from the liveJobs queue to the deadJobs queue. If the JobManager goes down before a job is added to the deadJob queue, the job will be lost.
Possible solution: only remove from liveJobs queue when adding to deadJobs queue succeeds with no error.

I'm going to work on the Ruby API client and put the code into autolab front-end repository.

Instrument tango so that it can be configured to dump ssh verbose output to the logs. This would be a huge help in debugging ssh/scp issues.

This issue makes the command-line client unusable for prealloc.

It's baaaaack! When I autograded this program the local time was 6:54, but Tango is reporting UTC instead:

_begin_
Autograder [Mon Aug 24 22:54:03 2015]: Received job [email protected]:82
Autograder [Mon Aug 24 22:54:11 2015]: Success: Autodriver returned normally

Autograder [Mon Aug 24 22:54:11 2015]: Here is the output from the autograder:

Autodriver: Job exited with status 0
...
_end_

However, the times in the job trace are correctly reported using local time:
_begin_
Runtime Trace
2015-08-24 18:54:03 | Added job [email protected]:82 to queue
2015-08-24 18:54:03 | Dispatched job [email protected]:82 [try 0]
2015-08-24 18:54:03 | Assigned job [email protected]:82 existing VM
...
_end_

When there are no VM pools (for example, when Tango has just started). The /pool endpoint returns {"pools": {}, "statusMsg": "Pool not found", "statusId": -1} complaining that the image name is invalid. This causes unexpected behavior on the frontend Autolab job status page.

The 213 datalab is taking about 7-8 seconds to run, with up to 3-4 seconds required for the scp-based copyin step, which should be almost instantaneous. In the old system, datalab took 2-3 seconds. In the past, this kind of slow-down has been due to authentication issues that required the ssh client to retry using different authentication schemes.

It would be easy to diagnose if there were a way to tell Tango to call ssh with the verbose option (-vvvv) and then dump the ssh output to the tango log.

The times in the feedback file returned by Tango are in UTC instead of local time. This was working a week ago and seems to be a regression caused by having multiple Tango repos.

Jobs can pick which VMMS they want to run on. Some can run on Tashi and some on Distributed Docker.

Jobs should roughly be dispatched in the order which they came. Exceptions are only when some jobs are not schedulable.

When I hit /info endpoint in the API, the elapsed_secs variable is shown as the current epoch time (e.g. 1429372581) instead of the actual number of seconds elapsed.

May you forgot to save the time when Tango was started?

I'm not entirely sure about the Preallocator implementation so I wasn't able to do it, but we definitely need this.

I already created a boilerplate at tests/testPreallocator.py

Any TangoJob must have a Makefile since the autodriver runs make in order to start running a job. validateJob should ensure that a Makefile is part of the input files. If not, it should reject the job.

This will return a list of images that the VMM can boot its VMs with.

Call to /prealloc simply freezes

We want a more interesting message to be printed when we hit the / route of RESTful Tango (i.e. when we are testing whether RESTful Tango is up).

The permission bits of files are lost as they are uploaded to Tango (by /upload). As a result, certain scripts that need +x to run, cannot run. We should either find a way to preserve these bits during upload or tell front-end to not rely on those permission bits and do bash hello.sh in their Makefile instead of ./hello.sh.

The tab spacings are well weird on greatwhite, github and my local machine. Let's make this more consistent, somehow.

We need to confirm that the Redis version is working.

With the exception of proxylab, 213 uses legacy .rb files that overload functions like autogradeInputFiles and parseAutoresult. None of the labs are autograding. Tango returns a status code of -3 to the front-end with no other feedback.

Kosbie uses similar legacy .rb files, so he's going to be running into this issue as well.

autodriver.c has a hard coded path to /bin/find, which causes it to break on ubuntu systems, which have find in /usr/bin/find.

The JSON output of getInfo and getPool uses JSON Arrays of ["a=X", "b=Y"] instead of Objects {"a":X, "b": Y}. This makes them non trivial to parse, since the client has to dive into the string rather then letting a JSON library do all the work for it.

Is there a reason for doing things this way?

Write a "distributed-docker" module that allows autograding on a set of distributed machines, one job at a time on each machine, in a Docker container.

Essentially, this section: https://github.com/autolab/Tango/wiki/Setting-up-Tango-server-and-VMs#test

Write a "distributed" option that allows autograding on a set of distributed hosts.

The localSSH VMMS is basically deprecated in favor of localDocker, so we should remove it