neurodata / ndstore Goto Github PK

Two errors:

1. Looks like resolution isn't set in the right place:

1192           offset = proj.datasetcfg.offset[resolution]
1193           corner = (h5f['XYZOFFSET'][0]-offset[0],h5f['XYZOFFSET'][1]-offset[1],h5f['     XYZOFFSET'][2]-offset[2])
1194           dim = h5f['CUTOUTSIZE'][:]
1195           resolution = h5f['RESOLUTION'][0]

Moving the resolution (line 1195) up to line 1191 seemed to fix that problem (did not commit). But then I got this error:

Traceback (most recent call last):
  File "/Users/abaden/Projects/open-connectome/django/ocpca/views.py", line 139, in queryObjects
    return django.http.HttpResponse(ocpcarest.queryAnnoObjects(webargs,request.body), content_type="product/hdf5")
  File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 1197, in queryAnnoObjects
    if not proj.datasetcfg.checkCube( resolution, corner[0], corner[0]+dim[0], corner[1], corner[1]+dim[1], corner[2], corner[2]+dim[2] ):
TypeError: checkCube() takes at most 5 arguments (8 given)

I think Kunal already fixed this in a previous query?

When calling /ocp/ca/kharris15apical/annos/id/3/582/530/88/ I get the following:

[09/Jun/2015 18:46:24] "GET //ocp/ca/kharris15apical/annos/id/3/582/530/88/ HTTP/1.1" 404 3431
[09/Jun/2015 18:46:28] ERROR [ocp:views:86] Unknown exception in getCutout.
Traceback (most recent call last):
  File "/Users/abaden/Projects/open-connectome/django/ocpca/views.py", line 62, in cutout
    return django.http.HttpResponse(ocpcarest.getCutout(webargs))
  File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 547, in getCutout
    return selectService ( service, webargs, proj, db )
  File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 388, in selectService
    return annId ( webargs, proj, db )
  File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 346, in annId
    (resolution, voxel) = restargs.voxel ( imageargs, proj.datasetcfg )
  File "/Users/abaden/Projects/open-connectome/util/restargs.py", line 147, in voxel
    if not ( datasetcfg.checkCube ( resolution, [x,y,z], [x+1,y+1,z+1] )):
TypeError: checkCube() takes exactly 5 arguments (4 given)
[09/Jun/2015 18:46:28] ERROR [django.request:base:231] Internal Server Error: /ocp/ca/kharris15apical/annos/id/3/582/530/88/
Traceback (most recent call last):
  File "/Users/abaden/.virtualenvs/ocp/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/abaden/Projects/open-connectome/django/ocpca/views.py", line 87, in cutout
    raise OCPCAError("Unknow exception in getCutout")
OCPCAError: 'Unknow exception in getCutout'
[09/Jun/2015 18:46:28] "GET /ocp/ca/kharris15apical/annos/id/3/582/530/88/ HTTP/1.1" 500 62955

Not sure what exactly is going on. Feel free to test on bv1.

I have created a local django server (open-connectome) and the server passes all tests (from py.test). However when attempting to log into the server (/accounts/login/) I am getting a "CSRF verification failed" error.

i am thinking it would be great to automatically generate a table from projinfo (for public projects) that had fields like:

• dataset name
• species
• xyz resolution
• xyz pixels
• image modality
• data type

@kunallillaney @randalburns @alexbaden @WillGray how hard is that to make?

Allows a user to link to existing channel tables instead of creating channel tables.

Right now it produces DATATYPE, which contains one of {image, annotation, probmap, etc}. I'd like to rename that DBTYPE and add DATATYPE, which contains {uint8, uint16, uint32, etc}. We could probably engineer a solution to pull the DATATYPE from the CUTOUT, but this is a more general way to do it.

Format should be token/channel/plane/cutout

Fixed in d4e015e

It seems like it should still delete the project. Perhaps the user already deleted the database?

for each project, i was thinking that perhaps we can have a field called 'parent',
which either points to another ocp token, or is root.
in particular, if the project is an annotation project, then its parent is the image data.
similarly, for a color corrected project.

for the raw images, its field is root.
this makes keeping track of things way easier, and doesn't require a full LIMS.

@WillGray @randalburns @kunallillaney @alexbaden - thoughts?

Center of map XY Coordinates (& ability to change them)
Ability to specify and navigate to a zindex

x and y tiles were swapped in the URL arguments. Works now!

See fed4e6c for the fix.

Based on Catmaid? @jovo any ideas?

http://w.ocp.me/images has a bunch of fields that i'd like in ocp.
i'm sure @kunallillaney working with the HBP people knows of others that they need/want.

I have implemented the isotropic scaling hierarchy. This means changing the ocpconfig datasets and projects table. I need the current version pushed ASAP, so that I can integrate these changes. I don't need them pushed to the master branch. But, I do need them.

Fix the the activation email module. The current setup is for django1.6 and broken because it is hard-coded to send all emails with link for dsp061.pha.jhu.edu. I think django1.7 does it differently(https://docs.djangoproject.com/en/dev/topics/email/)
Delete channel does not remove the tables. I have added a new method in OCPCADB called deleteOCPCAChannel() which needs to be called for channel deletes. I am calling this from views.py but seems to throw some error. ( the function in itself is tested and works fine)
Login Protect /channel and /token pages.
Non-owners of data public dataset should not be able to update it.
Non public datasets are not available to super-users.
Lock the datatype for a channel type, image can only be uint8/uint16, annotation can be uint32/uint64 and so on.
Lock the propagation state to either propagated or non-propagated. It is set to propagating at start then it will never leave that state via the service.
If a user creates a project and set a default token, then the public value of token should be the same as that of the project, now all default tokens are public.
If a project is a public should all it's tokens be public? If no then we need to restrict this, issue 1 and this are related and have the same fix.
Ensure that there exits exactly one default channel.
Create a "link to existing channel" option in the GUI which does not create the channel tables.

Details based on our discussion are here:

http://w.ocp.me/private:graph_construction

Open to other ideas - basic functionality (v1) seems sufficient for now, although propagate would also be nice.

Page: http://www.openconnectomeproject.org/#!male-c-elegans/c13v6

Broken link: http://openconnecto.me/data/public/.JarrellEmmons12.pdf
Correct link: http://openconnecto.me/data/public/articles/JarrellEmmons12.pdf

See ocpcarest line 1444. Looks like it's trying to pull getPropagate values from the project instead of the channel.

document how to use thunder+ocp

create file names for HDF5 downloads

@alexbaden @randalburns @kunallillaney
they want to be able to build a plugin to pull data from OCP.
unclear which spec they want to pull down, they want us to export a vaa3d file.
they will send us the details.

Look at a new window function. The file is ocpca/windowcutout.py

@randalburns @kunallillaney @WillGray @gkiar
http://research.mssm.edu/cnic/swc.html
is the format that vaa3d and janelia are now using to store skeletons

for example, 'default-channel' is not ok!

What's our strategy for users to get tokens/projects?

Do we want to document the web interface and allow anyone to make an account and request? Or do you prefer we point them to a person or a support page somewhere for now?

Definitely getting to the point that lots of people need to make these. I'd love it if we could do this in the API - but would need some sort of user auth.

The file appears to post fine, but when I download the cutout in HDF5 format using the same URL, I get a file of all "0s".

Groups, DATATYPES, DATASETS. Encompasses some other issues ( #38 , #35 ) here. Talk this week?

It doesn't seem like it would be too hard... just add a check to see if the h5py.get call returns None, and then throw an error. I'm happy to do it. @randalburns @kunallillaney, what say you?

http://localhost:8000/ocp/ca/apiUnitTests/apiUnitTestKasthuri/xy/1/1200,1400/2300,2500/557/filter/3653/

yields the following message

[23/Jun/2015 20:40:26] WARNING [ocp:ocpcarest:203] Incorrect arguments for imgSlice apiUnitTestKasthuri/xy/1/1200,1400/2300,2500/557/filter/3653/. 'NoneType' object has no attribute 'groups'

OCP problem or new syntax?

The default channel bug is not yet resolved. When creating a channel, if we select the default channel button, can lead to multiple default channels.
Similarly when deleting a default channel, it does not convert an existing channel to a default channel leading to a state where there might be channels but no default channel.

Randal, We need a page/interface via which a user can create channels while creating a project.

it would be great to have a service such that when we upload a brain mask,
server side that mask is propagated both up & down zoom levels,
and applied, which automatically builds a new image token that is the same as the original one but appended with '_mask', or some such

@randalburns @kunallillaney: Did interface for probability overlays change? They quit working for me (actually about a month ago...)

Doesn't work (prob32)
http://openconnecto.me/ocp/overlay/0.7/kasthuri11_ac3_membranes/xy/1/5472,6496/8712,9736/1005/

Does work (anno32)
http://openconnecto.me/ocp/overlay/0.7/kasthuri11_ac3_synapseTruth/xy/1/5472,6496/8712,9736/1005/

----- 500 error, not particularly informative to me -----

HTTPError at /overlay/0.7/kasthuri11_ac3_membranes/xy/1/5472,6496/8712,9736/1005/
HTTP Error 500: INTERNAL SERVER ERROR
Request Method: GET
Request URL: http://openconnecto.me/ocp/overlay/0.7/kasthuri11_ac3_membranes/xy/1/5472,6496/8712,9736/1005/
Django Version: 1.6.1
Exception Type: HTTPError
Exception Value:
HTTP Error 500: INTERNAL SERVER ERROR
Exception Location: /usr/local/lib/python2.7/urllib2.py in http_error_default, line 527
Python Executable: /usr/share/ocp/virtualenvs/ocp/bin/uwsgi
Python Version: 2.7.3
Python Path:
['.',
'',
'/usr/share/ocp/virtualenvs/ocp/lib/python27.zip',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/plat-linux2',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/lib-tk',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/lib-old',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7',
'/usr/local/lib/python2.7/plat-linux2',
'/usr/local/lib/python2.7/lib-tk',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/site-packages',
'/usr/share/ocp/open-connectome/django',
'/usr/share/ocp/open-connectome/util',
'/usr/share/ocp/open-connectome/ocpca',
'/usr/share/ocp/open-connectome/ocplib']
Server time: Tue, 28 Apr 2015 00:17:16 -0400

when creating a new database, syncdb needs to be run twice to create the database. why?

I don't know enough about how the probabilitymap stuff was written to make any headway on this...

[16/May/2015 19:23:41] ERROR [ocp:views:86] Unknown exception in getCutout.
Traceback (most recent call last):
  File "./ocpca/views.py", line 71, in cutout
    django.http.HttpResponse(ocpcarest.putCutout(webargs,request.body))
  File "/var/www/ocp/open-connectome/ocpca/ocpcarest.py", line 569, in putCutout
    return selectPost ( rangeargs, proj, db, postdata )
  File "/var/www/ocp/open-connectome/ocpca/ocpcarest.py", line 488, in selectPost
    if voxarray.dtype == ocpcaproj.OCP_dtypetonp[ch.getDataType()]:
KeyError: u'float32'

Looks like OCP_dtypetonp wasn't updated for the new schema?

OCP_dtypetonp = {'uint8':np.uint8,'uint16':np.uint16,'uint32':np.uint32,'rgb32':np.uint32,'rgb64':np.uint64,'probability':np.float32

Or I'm just confused...

Backend + API - this may be equivalent to 8-bit image upload, or nearly so...

Many times we don't need 32-bit precision on probabilities; as datasets scale, we could significantly cut down on data being transmitted/written. Two near term use cases (cell detection and large scale synapses)

Two test fail:

Shave test - Cannot post empty cubes to the database(all zeros)

Second test - Changing the HDF5 file format for posting data.