support of more fancy MATLAB objects about datajoint-matlab HOT 7 CLOSED

Hey @zym1010 ,

DataJoint is not necessarily coupled to Matlab. We also provide a Python version.

We agree that storing large files in the database usually doesn't make sense. Normally, the data after the first preprocessing step is stored in the database. For example, you could spike sort and extract spike times and store those in the database. The datajoint schemas of type Imported is what you would use in that case.

DataJoint does not help you copying the data via scp.

I have no idea how mym deals with more fancy Matlab objects (I am mainly working with the python version). Generally we don't recommend storing those kind of objects as blobs in the database. The reason is that you loose the power of relational queries on those attributes. What I mean is the following: Say you have a table with experimental data. Then you can quickly retrieve values for a particular animal and session like this

mytable & 'animal_id=8623' & 'session=2'

With blob fields you cannot do this because the comparisons are actually done on the mysql side.

I never worked with tables, but it seems that you could easily make them into a proper database table (via DataJoint). Writing a conversion function is probably not hard. That way you would have more flexible access to your data.

Hope that helps.

from datajoint-matlab.

@fabiansinz Thanks for your quick reply!

DataJoint is not necessarily coupled to Matlab. We also provide a Python version.

By tightly coupled I mean many blobs are saved in a matlab-specific way (or, more correctly, I guess mym way).

We agree that storing large files in the database usually doesn't make sense. Normally, the data after the first preprocessing step is stored in the database. For example, you could spike sort and extract spike times and store those in the database. The datajoint schemas of type Imported is what you would use in that case.

From what I see in the codebase of datajoint, in Import, you can read (probably also write) some large files from some local folders, which are probably mapped from some remote servers. Is that right?

from datajoint-matlab.

You are correct. We store data how mym stores it. You can load the same arrays with python though.

I am not sure which code base you are referring to (DataJoint or the example schemas). Import is just a particular way of designing a table. It requires you to have a makeTuples function that is called by populate. It is "just" semantics to indicate that this table imports data. The table name in the database takes that into account to make backup schedules a bit easier.

We personally use network drives with the experimental data on the machines that import/process the data and store it into the database.

from datajoint-matlab.

I am not sure which code base you are referring to (DataJoint or the example schemas). Import is just a particular way of designing a table. It requires you to have a makeTuples function that is called by populate. It is "just" semantics to indicate that this table imports data. The table name in the database takes that into account to make backup schedules a bit easier.

Thanks for clarification. Actually I don't know the difference between computed and import, since they all support populate. So that's just a semantic difference?

from datajoint-matlab.

Yes, the difference is semantic as well and mainly translates into different database tables naming conventions. The idea is that Computed tables are less important than Imported which are less important than Manual tables. The former two can basically always be recomputed while Manual data is lost. Therefore, backup should focus on manual tables (i.e. daily) and less on computed (e.g. weekly). The naming schemes make it easy to automatically detect important tables with a script in the database.

from datajoint-matlab.

Imported tables require access to data in the database and also in external files while populating.

Computed tables only require access to data already in the database.

FYI: a few labs are adopting DataJoint, so we will make a series of video tutorials and put them on YouTube. Watch this space.

from datajoint-matlab.

BTW: You can store any fancy MATLAB object in the database including class objects. However, more complex objects may not be loaded in Python. We generally stick with simpler things like n-dimensional arrays.

from datajoint-matlab.