Giter Club home page Giter Club logo

travel-model-two's Introduction

travel-model-two

Development plan:

  • TM2.0: Initial TM2 with Cube, CTRAMP core, 3-zone system. In use by TAM.
  • TM2.1: TM2 with transit CCR (capacity, crowding, reliability) implemented in Emme (uses Cube and Emme), CTRAMP core. Anticipated release: Fall/Winter 2021. This work is being performed in the branch develop
  • TM2.2: TM2 with Emme only (not Cube). CTRAMP core. This work is being performed in the tm2py repository
  • TM2.3: TM2 with Emme only and ActivitySim core. This work is also in progress in ActivitySim / BayDAG

travel-model-two's People

Contributors

binnympaul avatar davidory avatar dhensle avatar dorawu avatar flaviatsang avatar jfdman avatar khademul avatar lmz avatar rosellapicado avatar shimonisrael avatar sidharthanr avatar ychtw avatar yuqiww avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

travel-model-two's Issues

Data Dictionaries

How should we handle the data dictionaries, eg MAZ? In-line in guide with an anchor link? Separate markdown page?

Hard-coded output matrix type and incomplete MatrixType definition

Describe the bug

Failed to write out trip table matrices from CTRAMP, using the latest mtctm2.jar and common-base.jar.

08-Jun-2021 10:55:16:440, ERROR, InvocationTargetException exception making RMI method call: //172.22.78.49:1191/com.pb.mtctm2.abm.ctramp.MatrixDataServer.writeMatrixFile().
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$240(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$1/1876363557.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(Unknown Source)
	at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source)
	at sun.rmi.server.UnicastRef.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote_Stub.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at com.pb.mtctm2.abm.ctramp.UtilRmi.method(UtilRmi.java:123)
	at com.pb.mtctm2.abm.ctramp.MatrixDataServerRmi.writeMatrixFile(MatrixDataServerRmi.java:41)
	at com.pb.mtctm2.abm.application.MTCTM2TripTables.writeMatricesToFile(MTCTM2TripTables.java:546)
	at com.pb.mtctm2.abm.application.MTCTM2TripTables.writeTrips(MTCTM2TripTables.java:503)
	at com.pb.mtctm2.abm.application.MTCTM2TripTables.createTripTables(MTCTM2TripTables.java:282)
	at com.pb.mtctm2.abm.application.MTCTM2TripTables.main(MTCTM2TripTables.java:717)
Caused by: java.lang.RuntimeException: Could not determine type of matrix file, D:\MTC\travel-model-two\ctramp_output\auto_EA_SOV_GP_EA.mat
	at com.pb.common.matrix.MatrixReader.determineMatrixType(MatrixReader.java:193)
	at com.pb.common.matrix.MatrixWriter.createWriter(MatrixWriter.java:115)
	at com.pb.mtctm2.abm.ctramp.MatrixDataServer.writeMatrixFile(MatrixDataServer.java:113)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$240(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$1/1876363557.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

To Reproduce

Steps to reproduce the behavior:
Run CTRAMP/runtime/runMTCTM2ABM.cmd

rem ### create demand matrices in Cube matrix format - must restart mtx manager before running?
java -Xmx200g -cp "%CLASSPATH%" -Dproject.folder=%PROJECT_DIRECTORY% com.pb.mtctm2.abm.application.MTCTM2TripTables mtctm2 -iteration %iteration% -sampleRate %sampleRate%

Triggering line of code

  1. output matrix file extension is hard-coded as "mat". Matrix writer uses filename extension to determine matrix type:

public void writeMatricesToFile(String fileName, Matrix[] m){
//todo - currently write one matrix to each file since there is a problem
//calling RemoteMatrixDataServer.writeMatrices from RMI if the matrix array has more than 1 matrix
for (int i=0; i<m.length; i++) {
String matFileName = fileName + "_" + m[i].getName() + "." + matrixFileExtension;
Matrix[] temp = new Matrix[1];
temp[0] = m[i];
logger.info( m[i].getName() + " has " + m[i].getRowCount() + " rows, " + m[i].getColumnCount() + " cols, and a total of " + m[i].getSum() );
logger.info("Writing " + matFileName);
ms.writeMatrixFile(matFileName, temp);
}
}

  1. ".mat" is not defined in the common-base DetermineMatrixType method

public static MatrixType determineMatrixType(File file) {
MatrixType type;
String fileName = file.getName();
if (fileName.indexOf(".bin") > 0 || fileName.indexOf(".BIN") > 0 ||
fileName.indexOf(".binary") > 0 || fileName.indexOf(".BINARY") > 0)
{
type = MatrixType.BINARY;
}
else if (fileName.indexOf(".zip") > 0 || fileName.indexOf(".ZIP") > 0
|| fileName.indexOf(".zmx") > 0
|| fileName.indexOf(".compressed") > 0
|| fileName.indexOf(".COMPRESSED") > 0) {
type = MatrixType.ZIP;
}
else if (fileName.indexOf(".csv") > 0 || fileName.indexOf(".CSV") > 0) {
type = MatrixType.CSV;
}
else if (fileName.indexOf(".emme2") > 0 || fileName.indexOf(".EMME2") > 0 ||
fileName.indexOf(".e2ban") > 0 || fileName.indexOf(".E2BAN") > 0 ||
fileName.indexOf(".em2") > 0 || fileName.indexOf(".EM2") > 0 ||
fileName.indexOf("emme2ban") >= 0 || fileName.indexOf("EMME2BAN") >= 0 ||
fileName.equalsIgnoreCase("emme2ban") || fileName.equalsIgnoreCase("emmebank"))
{
type = MatrixType.EMME2;
}
else if (fileName.indexOf(".tpp") > 0 || fileName.indexOf(".TPP") > 0 ||
fileName.indexOf(".tpplus") > 0 || fileName.indexOf(".TPPLUS") > 0)
{
type = MatrixType.TPPLUS;
}
else if (fileName.indexOf(".mtx") > 0 || fileName.indexOf(".MTX") > 0 ||
fileName.indexOf(".transcad") > 0 || fileName.indexOf(".TRANSCAD") > 0)
{
type = MatrixType.TRANSCAD;
}
else if (fileName.indexOf(".omx") > 0 || fileName.indexOf(".OMX") > 0) {
type = MatrixType.OMX;
}
else {
throw new RuntimeException("Could not determine type of matrix file, " + file.getAbsolutePath());
}
return type;
}

Thoughts on resolution

  1. Remove the hard-coded matrix file extension, replace it with reading from properties file
  2. Add if condition for ".mat" in DetermineMatrixType()

Environment

windows 10
java 1.8.0_31

fyi @jfdman @lmz @DavidOry

Jar files compilation setup

The existing setup for compiling jar files includes dependencies in mtctm2.jar, common-base.jar, and jar files in the runtime\lib directory. Possibility to look into gradle builds.

HighwayAssign.job dividing SR2 and SR3 demand by cost share factors

Why does the demand need to be divided by the cost share factors (1.75 for SR2, 2.5 for SR3)? @dhensle

; shared ride 2, no value toll, exclude sr2 value toll and HOV 3+ facilities, internal demand plus intra-regional demand plus air passenger demand
pathload path = lw.gencost_s2, excludegrp = 12, 3, vol[2] = (mi.1.SR2_GP_@token_period@ + mi.1.SR2_HOV_@token_period@) / sr2costShare + mi.2.sr2 + mi.4.sr2
; shared ride 3+, no value toll, exclude sr3 value toll facilities, internal demand plus intra-regional demand plus air passenger demand
pathload path = lw.gencost_s3, excludegrp = 13, vol[3] = (mi.1.SR3_GP_@token_period@ + mi.1.SR3_HOV_@token_period@) / sr3costShare + mi.2.sr3 + mi.4.sr3

Generate Non-institutional Group Quarters

Generate non-institutional group quarters estimates at the MAZ level for 2000, 2005, 2010, and 2015. 2000 and 2010 will be based on Census. 2005 will be interpolated, and 2015 will be extrapolated. GQ estimates to be classified by military, university, or other.

Highway reliability skim - link attribute `RELBTY_VAR` not being created

In "HighwaySkims.job", lw.RELBTY_VAR on the path is being skimmed and stored in matrices,

mw[6] = pathtrace(lw.RELBTY_VAR)

but I could not find the creation of this link variable RELBTY_VAR. The closest I can find is RELBTY_SD which is generated in the "AverageNetworkVolumes.job", but RELBTY_SD does not get used in skimming.

; calculate total reliability
; zero reliability sd on managed lanes and tolled facilities
if (MANAGED == 1 | TOLLBOOTH > 0)
RELBTY_SD = 0
else
RELBTY_SD = (VC_REL_FAC + li.2.STREL) * li.2.DISTANCE * li.2.CTIM
endif

Missing the creation of RELBTY_VAR, I don't think the develop branch would run successfully.

Memory allocation and Java crashes

My apologies if this isn't correct place to ask this question. I couldn't find a user forum for TM2. Is there one?

I have a question about frequent "Java stopped working" crashes on TM2 model runs. They seems to occur during the second or third iteration of CTRAMP. Below are details of the model run and computing environment:

  • Select county 3 (Santa Clara)

  • Sample rate 1.0 for all TAZs

  • Windows Server 2016 Virtual Machine

  • Intel Xeon Gold 6134 CPU @ 3.20 GHz with 32 (virtual) cores

  • 512 GB RAM

Below is the event log message at the time of crash:

02-Apr-2021 09:29:34:640, ERROR, Exception exception making RMI method call: //10.1.0.80:1191/com.pb.mtctm2.abm.ctramp.MatrixDataServer.writeMatrixFile().
java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
java.net.SocketException: Connection reset
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:254)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:163)
at gnu.cajo.invoke.Remote_Stub.invoke(Unknown Source)
at gnu.cajo.invoke.Remote.invoke(Unknown Source)
at com.pb.mtctm2.abm.ctramp.UtilRmi.method(UtilRmi.java:123)
at com.pb.mtctm2.abm.ctramp.MatrixDataServerRmi.writeMatrixFile(MatrixDataServerRmi.java:41)
at com.pb.mtctm2.abm.application.MTCTM2TripTables.writeMatricesToFile(MTCTM2TripTables.java:514)
at com.pb.mtctm2.abm.application.MTCTM2TripTables.writeTrips(MTCTM2TripTables.java:495)
at com.pb.mtctm2.abm.application.MTCTM2TripTables.createTripTables(MTCTM2TripTables.java:275)
at com.pb.mtctm2.abm.application.MTCTM2TripTables.main(MTCTM2TripTables.java:687)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:240)
... 9 more

Is this some sort of memory/resource allocation issue with Java? The CTRAMP parameters are configured as follows. I listed ones that I thought were relevant.

runDriver.cmd
java -server -Xmx256m -cp "%CLASSPATH%" -Dlog4j.configuration=log4j-driver.properties -Djppf.config=jppf-driver.properties org.jppf.server.DriverLauncher

runMTCTM2ABM.cmd
java -server -Xmx130g -cp "%CLASSPATH%" -Dlog4j.configuration=log4j.xml -Dproject.folder=%PROJECT_DIRECTORY% -Djppf.config=jppf-clientLocal.properties com.pb.mtctm2.abm.application.MTCTM2TourBasedModel mtctm2 -iteration %iteration% -sampleRate %sampleRate% -sampleSeed 0

java -Xmx480g -cp "%CLASSPATH%" -Dproject.folder=%PROJECT_DIRECTORY% com.pb.mtctm2.abm.application.MTCTM2TripTables mtctm2 -iteration %iteration% -sampleRate %sampleRate%

runMtxMgr.cmd
START "Matrix Manager" %JAVA_PATH%\bin\java -Dname=p%HOST_MATRIX_PORT% -Xmx480g -cp "%CLASSPATH%" -Dlog4j.configuration=log4j_mtx.xml com.pb.mtctm2.abm.ctramp.MatrixDataServer -hostname %HOST_IP_ADDRESS% -port %HOST_MATRIX_PORT% -label "MTCTM2 Matrix Server"

runHhMgr.cmd
START "Household Manager" %JAVA_PATH%\bin\java -server -Xmx32g -cp "%CLASSPATH%" -Dlog4j.configuration=log4j_hh.xml com.pb.mtctm2.abm.application.SandagHouseholdDataManager2 -hostname %HOST_IP_ADDRESS% -port %HOST_PORT%

jppf-clientLocal.properties
jppf.local.execution.threads = 26

mtctm2.properties
distributed.task.packet.size = 500

County group three maz trip table matrices missing a row/column

When reviewing MAZ assigned demand, Kevin noticed this: The final set 3 group has one less row/column than there are MAZs in those four counties (8070 instead of 8071 in Solano, Napa, Sonoma and Marin).

cause: a bug in code that's skipping the last MAZ (id = 39726)

Feature: Improve Telecommute Representation

A short-coming of the original TM1 that persists, to some degree, in TM2 is the representation of working from home. In TM1, there was no explicit representation of workers working from home. The procedure went as follows:

  1. Workers are generated in the synthetic population.
  2. Every worker is assigned a "usual work location".
  3. Each worker is assigned a daily activity pattern, either "mandatory", "non-mandatory", or "home".
  4. Workers assigned a "mandatory" pattern generate at least one work tour to their usual work location.

To simulate workers who worked full time at home (i.e., did not have a usual work place) as well as telecommuters, MTC manipulated the coordinated daily activity pattern model (CDAP) to increase, over time, the number of workers engaged in a "non-mandatory" or "home" pattern, therefore reducing the quantity of work travel. But the model treated those working from home full-time and those telecommuting in an identical manner as workers who stayed home due to an illness.

With TM1.6, MTC modified these procedures as follows:

  1. Workers are generated in the synthetic population.
  2. Every worker is assigned a "usual work location".
  3. A new model predicted whether a worker was likely to telecommute based on the mix of employment by industry at the work location.
  4. Each worker is assigned a daily activity pattern, either "mandatory", "non-mandatory", or "home". Workers who are predicted to telecommute in Step 3 are assigned either a "non-mandatory" or "home" pattern.

This approach had the following benefits:

  • Telecommuters can be explicitly identified in the simulation results.
  • Telecommuting is a function of employment industry.

TM2 is similar to TM1, with the notable addition of an explicit work from home model, which identifies simulated workers that never work outside the home. The steps in TM2 are therefore:

  1. Workers are generated in the synthetic population.
  2. Some number of workers are assigned to work from home, i.e., never travel to a usual work location outside the home.
  3. Workers not assigned to work from home in Step 2 are assigned a "usual work location".
  4. Each worker is assigned a daily activity pattern, either "mandatory", "non-mandatory", or "home".
  5. Workers assigned a "mandatory" pattern generate at least one work tour to their usual work location.

While this approach improves upon the TM1 formulation by explicitly identifying those who work only from home, it still treats telecommuters in an identical fashion as workers staying home due to illness.

For TM2.2, we will modify the Java code to combine the current TM2 approach with an improved version of the TM1.6 approach. The model steps will be as follows:

  1. Workers are generated in the synthetic population.
  2. Some number of workers are assigned to work from home, i.e., never travel to a usual work location outside the home.
  3. Workers not assigned to work from home in Step 2 are assigned a "usual work location".
  4. Each worker is assigned a daily activity pattern, either "mandatory", "non-mandatory", or "home".
  5. Each worker assigned a "mandatory" pattern will be assigned a telecommuting status in a new "explicit telecommute" model. This model will identify whether or not the worker is telecommuting on the simulation day and will be taken from the TM1.6 implementation.
  6. Explicit telecommuters, as identified in Step 5, will have their daily activity pattern updated to either "non-mandatory" or "home", based on a fixed probability distribution.

The advantages of this approach relative to TM2 are the same as in TM1.6, specifically:

  • Telecommuters can be explicitly identified in the simulation results.
  • Telecommuting is a function of employment industry.

The minor change in this approach relative to TM1.6 is that the explicit telecommuting model comes after CDAP, not before. We therefore benefit from the coordination across household members that CDAP provides in determining who will work on the simulation day. For workers subsequently identified as telecommuters by the explicit telecommuting model, the mandatory pattern is updated to reflect the lack of a work tour.

cc: @lmz, @gregerhardt, @FlaviaTsang, @AshishKuls, @vivverma9

Updates to Intermediate Stop Location Choice Algorithm for CT-RAMP without TAPs

Existing implemented algorithm (with TAPs) to find the stop location alternatives on a Transit Tour evaluates:

For an Outbound Stop on Walk to Transit Tour:
Can you walk from Origin MGRA to Stop MGRA and then can you walk from Stop MGRA to Boarding TAP? or Can you walk from Alighting TAP to Stop MGRA and then can you walk from Stop MGRA to Destination MGRA? If yes, that stop MGRA is a potential stop location.

image

For an Outbound Stop on Drive to Transit Tour:
Can you Drive from Stop MGRA to Boarding TAP? Or Can you walk from Alighting TAP to Stop MGRA and then can you walk from Stop MGRA to Destination MGRA? If yes, that stop MGRA is a potential stop location.

image

There is similar logic's for Inbound Stop.

This algorithm needs to be revised in the CT-RAMP version without TAPs. Ideally, the logic to find stop location alternatives for transit tour would be to check - is there a transit path from origin to stop and is there a transit path from stop to destination. However, this would mean code will have to look up the transit skims from origin to all zones (alternatives) and from all zones to destination, for every stop for every tour, which is doable but computationally expensive.

We have currently implemented a simplified algorithm that is comparable to the existing algorithm and is computationally feasible.

For an Outbound Stop on Walk to Transit Tour:
If the Origin MGRA and Stop MGRA are in the same TAZ or if the Stop MGRA and Destination MGRA are in the same TAZ, then the Stop MGRA is a potential stop location (this works because we know that there is a transit path from Origin TAZ to Destination TAZ for TAZ-based transit skims).

image

For an Outbound Stop on Drive to Transit Tour:
If the Stop MGRA and Destination MGRA are in the same TAZ, then the Stop MGRA is a potential stop location. Note that the updated algorithm only allows the stop choice on the walk leg of the drive to transit tour. This is because we don't know which parking lot is going to get chosen for the drive leg and therefore making it impossible to identify a intermediate stop on the drive leg.

image

Team acknowledges that neither of existing or revised algorithms are correct way of finding the intermediate stop location choice and team will revisit this at some point and see if we would like to improve this in future.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.