Memory Management about angr-doc HOT 9 CLOSED

On a similar note, when using Explorer, I can ctrl-C out mid-operation and it will have stored the current progress. However, I'm trying to use path groups instead and it appears that ctrl-c'ing out of a path group will throw away all progress.

from angr-doc.

I'm not the most appropriate person to answer this issue. Just my two cents:

• Angr does not utilize more than one CPU core right now. So it cannot chew up all CPU resources if you have more than one core.
• You can set a heap limit for your process in Python using resource.setrlimit(). Reference: http://stackoverflow.com/questions/2308091/how-to-limit-python-heap-size

from angr-doc.

Most memory issues in angr come from either solving constraints in Z3 or having too many paths (or states) in memory. For Z3, I don't think there is anyway to make it perform better (memory-wise). Regarding the latter one, you can manually pickle Path, and then load them back when your memory resource allows. I think angr already provides support for that - hopefully @zardus can give you more information about it.

EDIT: freaking typos...

from angr-doc.

By the way, it'll be really helpful for us and other readers if you can post your next angr-related issue in angr issues instead of angr-doc :-)

from angr-doc.

Gotcha. Yep, totally understand on the resource limit/ulimit/user changes that can be made. The thing that crunches the machine to a half the most is the memory utilization, which would be best managed by the app, tho my linux box does a great job of killing ipython when it's upset. The other challenge with the user controlling the memory use (not claiming the user doesn't have a part in this) is not knowing before you hit go just how much of a toll the search is going to take on your machine.

My guess is with some creative memory management you could take a problem that, at the moment causes Linux to kill ipython due to over consumption of the memory, and allow it to run with the understanding that it will take longer (thinking swap file/paging or something).

from angr-doc.

The PathGroups issue (interrupting it doesn't save the current status) is resolved by instantiating the PathGroup in mutable mode (pg = project.factory.path_group(immutable=False)). This will become the default in the near future, but in the meantime, you can use this to have interruptable path groups.

@ltfish's proposed solution of setting the rlimit to control memory utilization is really the correct option. This will cause Python to raise a MemoryError when the memory limit is met and will allow you to clean up, clear some memory, and resume your analysis. This is the best way to currently do this.

For pickling data out to disk, please keep in mind that to avoid deduplication of data, angr uses a storage layer (called ANA) on top of pickle. By default, it stores intermediate data in memory, which obviously makes it hard to reduce memory usage by pickling stuff out to disk. To remedy this, do:

import ana
ana.set_dl(pickle_dir="/path/to/some/fast/storage")

path_group.split(from_stash="active", to_stash="stashed", limit=8)

import pickle
with open("/path/to/some/fast/storage/stashed_paths.p", "w") as outfile:
    pickle.dump(path_group.stashed)
    path_group.drop(stash="stashed")

That'll pickle out those paths to disk, and you can retrieve them afterwards.

Doing this automatically, depending on memory constraints, would be great. However, our time is crazy limited, so rather than sink time into this, we've gotten 100GB machines to run our heavy analyses on. If you're inclined to implement some good as-needed spilling of objects to disk, we'd really appreciate a pull request! Other than that, it's unlike that it'll be done anytime soon :-(

from angr-doc.

Appreciate the response. I did not know about ana and will look into it. Recently I've been trying to understand the sim procedures enough that I could start building more of the common ones and help develop the project. I'll absolutely see if I can get the ana just-in-time dump stuff to work in an automated way.

Keep up the great work!

from angr-doc.

So I've had a chance to play with this a little lately but have been unable to get your ana example to work. I've tried setting the pickle dir during the ipython session (before and after running an analysis). I've also tried adding that "ana.set_dl" command to every library that I found "import ana" in. I have yet to get any pickle objects appear in my temporary directory, even though the ._store_type shows it should be using pickle objects.

Any idea why this is happening? Is there a correct place to put that ana command that will cause it to utilize the pickle file?

from angr-doc.

Could you move this to the ana (github.com/zardus/ana) repo? Having everything in one thread is a bit confusing.

from angr-doc.