Error for unpick-able instruments about qcodes HOT 4 CLOSED

Re: getting more information on what caused the pickling error - Just looking at the traceback, I don't think I'm going to be able to do much. I could wrap p.start() in a try/except just to add "this probably means something in one of your instruments is not picklable" to the error, but the error itself doesn't give me any more context to work with. Beyond that the only thing I can think of is to try and reconstruct what multiprocessing does when it collects and pickles objects for forking, and step through it to pinpoint the error. That sounds like a major headache...

Re: still running in the background - fundamentally, if you want to run this in the background, everything you will interact with in the loop needs to be somewhere outside the main process. Which means that either it's picklable so it can be sent to a separate process, or it needs to have been created in a separate process in the first place and no matter which process wants to talk to it, you do so by sending messages to it in its own process.

The latter is what I've settled on doing even for simple models, because this also means the model never gets copied, so it can have state that every caller in any process will agree about. My MockModel is probably a bit overkill for what you want to do (it was meant to mimic real instrument communication channels, so it's separate from the MockInstruments and has a stringified interface to mimic network communication channels), though it would probably work. You can see how that works now with the example system here.

Simpler, and probably better for this purpose, is to just make a single Instrument that contains the Kwant model and interacts with all of its degrees of freedom as Parameters. The key (in the branch I'm using to fix #53 ) is to do everything unpicklable (ie creating the Kwant system) in the server process, by putting it in the on_connect method, which is decorated with @ask_server (or @write_server, since this doesn't return anything, but @ask_server is safer because the main process blocks until on_connect finishes and can therefore catch any errors that happen during this setup.)

Check out how VisaInstrument does this - it stores the information it needs to create the connection in the instrument prior to starting the server (which happens in super().__init__, which we have to put at the END of __init__ rather than the beginning where it normally goes) and then all the unpicklable stuff (involving self.visa_handle) happens on the server.

Will that work for Kwant systems? You may also find the getattr and setattr methods useful - they work exactly like the getattr and setattr built-in functions if you supply a string for attr as usual, but a) they proxy these calls to the server, and b) they accept sequences for attr as well, which let you dive into nested dictionaries for flexible access of state variables.

Sorry for being so slow to respond to your issues... getting the basics of the new InstrumentServer setup working was actually pretty easy, but it's taking a long time to tweak it to be (I think) easy for users to work with, and to test it thoroughly so we don't hit any big snags like this again!

from qcodes.

I think this solution might work quite well for numerical simulations.
Normally you don't want the simulation to be copied or anything, because they can be very memory intensive. Hence this solution doesn't look to me as a big change to the simulation I wrote.

If I understand correctly, I need the inst-process branch in order for this to work.
Would it be a good idea to start using this branch already, or shall I wait until it is merged into the master branch?

from qcodes.

@damazter @alexcjohnson should we close and move forward with this ?

from qcodes.

Yes, this issue is obsolete - must have gotten missed in the great purge :)

from qcodes.