implement chunking to reduce memory overhead about bca4abm HOT 2 CLOSED

I am not sure we are going to get significant memory savings from chunking the calls to assign_variables in (e.g.) person_trips processor without some major re-architecture. Because the resulting assigned_columns are simply added to the trips table after being computed, so there needs to be room for them unless we are willing to forgo saving disaggregate trip or person-level benefit results . But if we are willing to forgo saving disaggregate results, chunking could substantially reduce total memory overhead.

On the other hand, however, I notice that when Binny ran out of memory, it was in when grouping results in person_trips.py calling add_grouped_results. I think we can easily reduce the memory footprint of that call substantially.

from bca4abm.

As we discussed, our plan is to continue to read the entire initial table into memory and then to chunk/batch the additional field calculations, which also means not saving the additional fields. We don't write out the additional fields anyway, so not saving them should be fine.

from bca4abm.