rsginc / daysim Goto Github PK
View Code? Open in Web Editor NEWDaySim Activity-Based Model
Home Page: https://github.com/RSGInc/DaySim/wiki
License: Other
DaySim Activity-Based Model
Home Page: https://github.com/RSGInc/DaySim/wiki
License: Other
All the skim file readers were using short.maxint as a maximum storage value instead of ushort.maxint. I changed all readers to truncate at ushort.maxint -1 (the - 1 just to be safe). This will allow values to be twice as high before truncating (around 650 minutes, miles or dollars, instead of around 325)
SVN revisions 2582 through 2585
I notice that during Engine.BeginCalculateAggregateLogsums(randomUtility) the automatic Garbage Collector dramatically reduces the amount of memory being used (by at least 50%) and it then stays low. I experimented by inserting GC.Collect() immediately after Engine.BeginLoadRoster() and saw that this forced an equivalent memory reduction at that (earlier) point in the process. Apparently there is something about the way that Engine.BeginLoadRoster is coded that causes memory to be hogged and retained until the entire roster is loaded and the corresponding impedance matrices are all created. This causes performance problems in cases where the ultimate size of the matrices is smaller than the available memory but Engine.BeginLoadRoster hogs more than the available memory. Paging grinds the process to a near standstill. Looked at another way, it requires more memory than should be necesary to support large rosters. I wonder if there is a way to recode Engine.BeginLoadRoster so that it frees memory up if possible after processing each entry.
Open matrix (OMX) is an HDF5-based matrix container for travel modeling. We need to add support for it to Daysim. Currently Daysim reads and writes HDF5-based matrices for SFCTA and PSRC. OMX also uses HDF5 as its container, but it uses it differently than SFCTA and PSRC. The OMX specification is here. PSRC wrote some C# for OMX I/O as well that we may be able to start from or simply use. We need to add OMX reader and writer classes as well as support to the impedance roster for OMX.
@mabcal to identify some examples of good and bad practices in order to better define this task
There is an issue with TransCAD matrix reader as described below.
If skims are in TransCAD format, an loading error might occur in following situations:
Not sure whether this situation would work or not:
Matrix is saved in TransCAD 6; DaySim is built using TransCAD 6 DLL but local machine only has version 7 license;
Adds three new configuration parameters
IntrazonalWalkMinutesPerMile_OverrideSkims
IntrazonaBikeMinutesPerMile_OverrideSkims
IntrazonalAutoMinutesPerMile_OverrideSkims
If they are set >0, are used instead of skims to determine the speed to apply to the short-distance adjustment for intra-zonals in ImpedanceRoster.SkimValue. Use when the intrazonal values for skims are not meaningful.
Need to revise the transit path utility to:
Will add Global.Configuration.Nashville == true
Nashville not working with recent changes, but issue due to something else.
This configuration switch had been moved to Settings by John Mulholland, but we were still thinking that it was operational in configuration - it makes more sense as a configuration setting, so it was moved back.
Will create a separate issue on flagging invalid Configuration lines, which would have alerted us to this issue much sooner.
We don't know what version / release each model is using. I suggest we create a Daysim.exe -version entry point that prints to the console the version / build date.
There seems to be an error in “OtherTourDestinationCoefficients_nashville-v1.8.F12”, coefficient 24 is coded as 924 (row 27). I guess DaySim might ignore this coefficient if coded as 924.
Daysim performance seems to max out at 8 threads....can we do better than that? We should profile the code and see where the bottlenecks are.
People who are self-employed and work at home are generally counted in the employment data, so their work location should be counted and compared against the employment data in shadow pricing. Without this change, the current code tends to drive the work at home share towards 0 when there are more jobs than workers in the region.
Suzanne from PSRC had put in a a postbuild app that would write the SVN revision number to the log print file. This does not work for Git, so it was deleted from the post-build configuration
This is called in 208 places. Haven't don't profiling but suspect it is called very often. The work is divvied into many batches, each run in a separate thread. As each thread does work it stores results. These results for each thread are stored separately in an array indexed by the batch id. The code to get the batch id from the thread is a linear search. This code is also called to look up information such as ModelIsInEstimationMode and GetChoiceProbabilityCalculator. It is true that the number of threads is small but there is no reason to do this -- each thread could simply store its associated batch id using Thread.SetData. Alternatively the name of each thread could be specified at creation. Even better the ChoiceModelHelper which is so often retrieved could be stored in the thread local data.
I do not know the limits to how much memory can be stored in each thread. It could be that each thread should just store all of its own data.
This was due to edits made to get it to also read SFCTA's HDF5 files.
Temporary fix was made so it will work for both.
Still need to make a more elegant fix to get rid of PSRC switch. (Idea is to use two separate file type labels in the Roster and use that instead of PSRC switch).
Changed to work like the other readers.
Need to add work and other tour mode choice constants for person type university student in Nashville. This will help correct large differences in tour mode results for university students in the Vanderbilt area. School/univ tour mode choice already has person type segmentation.
This can be especially useful for testing.
Add documentation to this Daysim project site on how to add a new model and a new test to our testing system. Currently we have Nashville as our test system with different bat files for running different tests.
We agreed our formal test system would look like:
Added extra additive distance terms for over 20, over 30 and over 40 miles in the WorkLocationModel (new parameters 46,47,48) and a walk mode calibration constant (parameter 70) in all the TourMode models (this was the base mode in estimation, so no parameter was estimated).
If these parameters are not in the F12 file, DaySim will just ignore the variables, so it is backwards compatible.
The observation:
My guess: When using "transcad" format, the TAZ index from matrix is used as internal zone number while they should be treated as TAZ. In household 200, it is supposed to find aggegate logsum for TAZ 889, or internal zone 690. If program view the skim index as internal number, it would search for zone 690 aggregate logsum. But there is no TAZ 690 then no corresponding aggregate logsum.
This was due to edits made to get it to also read SFCTA's HDF5 files.
Temporary fix was made so it will work for both.
Still need to make a more elegant fix to get rid of PSRC switch. (Idea is to use two separate file type labels in the Roster and use that instead of PSRC switch).
The DORP (driver or passenger) variable on the output trip file is used for this, since it is otherwise always 0 for transit trips, and there are no empty output fields available on the trip file, so this provides the information without changing the file format. The value is rounded to integer minutes of walk time, access and egress combined.
When making this change, I discovered an error in the path calculations for the new stop area version of the walk-to-transit path type model. It was not multiplying the access/egress walk times by 2 when run at the tour level for round-trip LOS, so that has been fixed. This does not affect cases where TAZ-based transit matrices are used - only DVRPC.
in order to diff output files from two identical runs to ensure there are no differences, we need to first sort output files by ID before writing them out.
Daysim is looking up the skims incorrectly for inbound versus outbound. They are transposed in some cases. We are investigating for Nashville.
Was converted from minutes after 3 am to minutes after midnight twice
Bug found by John B
Daysim.ChoiceModels.ChoiceModelUtility.DrawRandomTourTimePeriods, which is used to guess a tour arrival and departure time for WorkTourDestinationChoice, never returns PM peak for tour destination departure time. Since Commuter Rail only runs outbound in the PM peakin Nashville, we end up with no drive commuter rail trips. I traced (see attached) the work tour destination arrival time and work tour destination departure times, and we always only get 8pm or later for departure time. It looks like arrival times are ok though. This will impact every model to some degree, especially where there is different service by time-of-day, like commuter rail, BRT, etc.
out.zip
Revisoin 2580 in SVN
Get DVRPC testbed setup
The code to read the configuration file should pause and print an error if it reads a configuration line that is not valid (is not contained in Configuration.cs).
We still had the DestinationScale line in assuming that it was being used correctly, but it had been moved from Configuration to Settings, and there was never a warning or error to alert us that the DestinatlonScale line in the configuration file was no longer valid.
If a location is not specified on the command line the log is stored in the same directory as the .exe
I think we will/should be moving away from executables being kept in the model directories and therefore a better system would be for it to be stored in the same folder as the other default items such as the output folder are stored. I believe this is the same directory as the configuration file.
This came up for me as a developer because when I ran from inside Visual Studio the log file was difficult to find in subdirectory: bin\x64\Debug
Was revision 2566 in SVN
@mabcal requested
Instead of using R scripts and F12 files, put shadow pricing code into Daysim
Changed it to work like the new code for the stop area version - uses transit travel time stored at the tour level. Needs to be tested on Nashville outputs.
This was done in a hurry for the Chattanooga project, and uses external R scripts to calculate the shadow prices and put them into the F12 files. Will create another issue to do this more elegantly within DaySim in the future.
See RSGInc/DaysimDataTools#8
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.