API change Staging for generators is also a simple task about adaptivemd HOT 5 OPEN

I'm in favour of option 1. Some system-specific files are always needed, but its not a big overhead to copy them (they are usually small compared to the results files). This makes the writing and debugging of MD-code specific scripts easier. Its not necessary to store these files in the end, so its always just a temporary redundancy of information.

from adaptivemd.

Yes and no. Imagine that there are 1000 workers on Titan running and every time they start a trajectory they request about 15MB from the DB each. If the DB is running here at FU (which is possible) then you need to transfer 15GB before you can even start. I guess mechanism 2 will just minimise transfer to the cluster. Well, 15MB is on the very large end, but still.

I agree, 1. is much simpler and would be one transfer per job. Currently it is 1 transfer per worker and option 2 would be 1 transfer per project.

from adaptivemd.

I think we will have to test this on different remote resources in order to know whether this is a problem. However, in general I think there will be other more important problems to this, e.g. the continuous access to the database due to restrictive access policies for some clusters.

If this is a problem what could also help is to have a directory on the remote cluster from where workers can copy system files. This is probably the most basic functionality of a staging area. For this functionality the only change is that workers need the option to get the path to that directory from the DB and copy all files from this directory to their working directory.

from adaptivemd.

Well, the staging option already works and was there before the DB one. It is part of the original RP implementation and to be compatible we might still need this.

My main point was the question, if we want to support a special "Now move all the staging files" function and mechanism or do we just implement this as a simple task, that will only copy the files, but not run the code.

I guess if I would have changed that you would not have noticed. Only that there is one more task in the list.

from adaptivemd.

I think a simple task is sufficient. Maybe just add an example of how this can be done.

from adaptivemd.