lacmta / metro-api-v2 Goto Github PK

Would be nice to tell ITS to point to one custom domain rather than various lightsail servers when we change.

We need to make sure the custom domain SSL certificate is automatically renewed.

Remove this legacy endpoint:

/{agency_id}/stop_times/{trip_id}

Update the AWS CLI command to include the new data-loading-service container.

current_status values should reflect the VehicleStopStatus Enum with the values:

• INCOMING_AT
• STOPPED_AT
• IN_TRANSIT_TO

Originally posted by @matikin9 in #69 (comment)

Prerequisites:

• #24
• #1

Standards

• https://amdigital-co-uk.github.io/6.-Engineering/Quality-Standards/Endpoint-Naming-Conventions/

Examples

CTA

• gettime
• getdirections
• getstops
• getservicebulletins
• getpredictions
• getvehicles

Source: https://www.transitchicago.com/assets/1/6/cta_Bus_Tracker_API_Developer_Guide_and_Documentation_20160929.pdf

MBTA is:

• Transitioning to running containers on AWS Fargate.
• Running anywhere from 3-5 servers (not containers) at a given time, and serving ~500 million API requests per month.
• Polling the TripUpdates feed every 10 seconds, the VehiclePositions every second, and Alerts every minute.

We've set up a Postgres DB already, but we may want to go with SQLite and do in-memory querying. SQLite is a file-based DB that we can connect to and run queries on. Data is saved and persistent.

On the Lightsail container service that currently runs our API, we would modify the scheduled process to pull data from Swiftly and write it to a local SQLite DB file on the container. The SQLite file would then be accessed when needed.

• what DB option to go with?
• define tables for the data

[email protected]?

This is a dependency for:

• #21

Things to figure out:

• #151
• How are we going to come up with a list of stops, and can we do this in phases?
  • Start with the trip with the most number of stops. maybe using the trip_id with the highest stop_sequence value.
  • Create a table of unique stop patterns for each route_id? Would that just be the shape_id?

Specs:

• Takes in a route_id for rail, example: 801 or route_code for bus. need to work this out more.
• Endpoint response should be:
  • an array of objects, one for each stop along the route
  • each object should include:
    • route_id
    • stop_id - from stop_times
    • stop_name - from stops
    • stop coordinate - from stops
    • direction_id - from trips
    • day type (Weekday, Saturday, Sunday) - parsed from service_id from trips
    • departure_time - from stop_times, separated by each day type
    • stop_sequence - not completely sure how we will determine this

Example:

[
  {
    route_id: `801`,
    stop_id: `80214`,
    stop_name: 'Union Station - Metro Red & Purple Lines',
    coordinate: .....,
    stop_sequence: {
      direction_0: {
        weekday: 1,
        saturday: 1,
        sunday: 1
      },
      direction_1: {
        weekday: 14
        saturday: 14
        sunday: 14
      }
    }
    stop_times: {
      direction_0: {
        weekday: [
          '05:01:00',
          ...
        ],
        saturday: [
          '05:01:00',
          ...
        ],
        sunday: [
          '05:01:00',
          ...
        ]
      },
      direction_1: {
        weekday: [
          '05:01:00',
          ...
        ],
        saturday: [
          '05:01:00',
          ...
        ],
        sunday: [
          '05:01:00',
          ...
        ]
      }
    }
  },
  {
   ...
  }
]

• check in with Swiftly on our API key quotas, especially now that we're pinging their API every minute.
• let Rollin know

Need to figure out what we want this to look like, who we could reach out to, etc.

Not seeing this field:

• schedule_relationship

GTFS reference

If no value is provided from Swiftly, store as -1. If value is -1, return in feed as an empty string.

• [Nina] Email Swiftly & Rollin about increasing our limit with our API key
• [Albert] Increase the frequency that we pull Swiftly's data. See what's the limit on how low we can get the latency.
• [Albert] Optimize data processing

Prerequisites:

• #15
• LACMTA/metro_api_v2_archived#34
• LACMTA/metro_api_v2_archived#9

Level of Effort

https://github.com/LACMTA/metro-timetables

• ⭐ Only a few endpoints.
• ⭐⭐⭐ Return the shapes data properly will be hard because there is a one to many relationship. One route_id can have multiple trip_ids and each trip_id is associated with a shape_id. Not sure whether each trip_id can have multiple shape_ids.

THIS ISSUE IS STILL A DRAFT

This endpoint will power the redesigned Schedule page as defined in this Figma: https://www.figma.com/file/hGHBpaYBoZB6ZSJdEat0tj/Schedules?node-id=89%3A144&t=iALhZCxiEG38Thrg-0

It needs to pull in and store data that is currently in Metro.net's WordPress site, which can be accessed via the WP REST API.

It needs to provide:

• agency type (Metro Rail vs Metro Bus) - agency_id
• line label (letter for rail and number for bus) - route_code or route_id - for route_code we may want to maintain a separate table that gets updated during GTFS processing to associate route_ids with route_codes for bus because route_code is in the stop_times table.
• line description separated by terminus 1, terminus 2, 'via' description - this can be pulled from the Line Overrides content from the metro.net WP API but will need to be manually broken out as 3 separate fields
• route_id
• color - route_color from GTFS routes table for rail. Manually set for bus.
• icon image url - link to iconography repo for rail?

Structure:

[
    {
        agency_id: 'LACMTA_rail',
        routes: [
            {
                line_label: 'A Line',
                route_id: '801',
                color: '#001122',
                icon_url: 'https://....',
                description: {
                    terminus_1: 'Downtown Los Angeles',
                    terminus_2: 'Long Beach',
                    via: ''
                }
            },{
                ...
            }
        ]
    },
    {
        agency_id: 'LACMTA',
        routes: [
            ...
        ]
    }
]

Throw exceptions when we don't have data and utilize FastAPI's default behavior:
https://fastapi.tiangolo.com/tutorial/handling-errors/?h=default#override-the-default-exception-handlers

Note: for things like Cancellations, we can distinguish between the data being missing versus "0" cancellations existing.

Tasks

You know what to do!

Tied to:

• #23

Decided to go with email authentication instead of user login pages.

• Need to set up and get credentials for Metro email (#25 )
• Get token authentication working
• Design UX (#68 )
• Add secrets to deployment
• Password reset
• Track endpoint requests with counter in table
• Add authentication requirement to realtime endpoints
• Logz.io tracking:
  • registrations (through DB logging?)
  • requests per token

Right now, only the calendar_dates.txt file is automatically updated.

Processing the full static GTFS takes at least an hour so we will need to manually do that in coordination with this Sunday's shakeup.

The trip_updates/all endpoint is breaking the server!

We can save the stop_time_updates data as in a json-typed column in the main trip_updates table, in addition to having that data in a separate table. This saves us from having to run a ridiculously intensive join on this large dataset.

• Email to Glen bringing up that we need support
• Reach out to ITS to start discussion of support:
  • Anika
  • Sharada

High Priority

Normal Priority

• Line pages integration with API

Low Priority

Does it make sense to have a single datetime that indicates when the API was last updated, and what would even count as "updating" it?

Maybe we can have a last_updated field as part of a status endpoint? We'd still have to think about what that means for each endpoint since we source the data differently.

Related to:

• #10

• update landing page title Metro API 2.0.6 FRONTEND IN APP - Metro API - v2.0.6
• update API endpoint descriptions
  • Get Canceled Trip -> Get Canceled Trips for Line
  • Get Canceled Trip -> Get All Canceled Trips
  • Get Canceled Trip Summary -> `Get Summary of Canceled Trips

Bring ing Alex & Tiffany

schedule_relationship is returning the numerical values of the enum instead of the string values (defined here: https://gtfs.org/realtime/reference/#enum-schedulerelationship).

Example:

General Endpoint Naming Conventions

• All GTFS endpoints require an agency_id.
• If no specific value for a parameter is specified at the end, the endpoint will return all values for that parameter.

GTFS-rt Trip Updates

• /{agency_id}/trip_updates/ - (future: return metadata?)
• /{agency_id}/trip_updates/all - Return all trip_updates
• /{agency_id}/trip_updates/{field_name}/{field_value}
  • field_name possible values:
    • trip_id
    • vehicle_id
  • field_value possible values:
    • list - Return all unique values for this field as a subset of the trip_updates data
    • {trip_id} - Return a singular trip_update for the trip_id provided. (future: multiple comma-separated trip_ids return trip_updates for each trip_id)

GTFS-rt Vehicle Positions

• /{agency_id}/vehicle_positions/ - (future: return metadata?)
• /{agency_id}/vehicle_positions/all - Return all vehicle_positions
• /{agency_id}/vehicle_positions/{field_name}/{field_value}
  • field_name possible values:
    • vehicle_id
  • field_value possible values:
    • list - Return all unique values for this field as a subset of the vehicle_positions data
    • {vehicle_id} - Return a singular vehicle_position for the vehicle_id provided. (future: multiple comma-separated vehicle_ids return vehicle_positions for each vehicle_id)

Not sure exactly how to do this, but it looks like FastAPI has a ResponseRequest class that might be what we need. It might create cross-domain issues?

This would help address latency concerns, but we may still need to download the data and store it in a database for querying in order to implements a Predictions endpoint that allows API users to query by line_id and stop_id.

Issue

The data returns slowly because of the JSON encode that is being done on each of the geometry fields.

Potential Solutions

A) Use pandas to encode the field as per this SO article:
https://stackoverflow.com/questions/73564771/fastapi-is-very-slow-in-returning-a-large-amount-of-json-data
B) Utilize the geography field for spatial queries BUT DO NOT return it as part of the API response (this avoids the encoding process)

(in progress)

Given a vehicle_id, return:

• trip_assigned - new boolean field with a value of:
  • TRUE - if trip is not null
  • FALSE if trip is null

If trip_assigned == TRUE, also return:

• current_status - from vehicle_positions (gtfs-rt)
• stop_id - from vehicle_positions (gtfs-rt)
• stop_name from stops (gtfs-static) based on this stop_id
• position - latitude and longitude values from vehicle_positions (gtfs-rt)
• arrival or departure time - whichever exists for the matching stop_id within the list of stop_time_updates in trip_updates (gtfs-rt)
• route_code - from stop_times (gtfs-static) based on trip_id and stop_sequence from vehicle_positions (gtfs-rt). Note: the problem with using the route_id value from trip_updates is that we have combined lines where the id may actually match multiple routes.

Return vehicle_positions data properly nested.

Example of current output:

{
  agency_id: "LACMTA",
  position_latitude: 34.05466079711914,
  vehicle_id: "5818",
  trip_start_date: "20221014",
  position_bearing: 243.2840576171875,
  stop_id: "17258",
  current_stop_sequence: 39,
  timestamp: 1665795819,
  position_longitude: -118.2251968383789,
  vehicle_label: "5818",
  trip_route_id: "78-13157",
  position_speed: 5.900928020477295,
  trip_id: "10078005221731-JUNE22",
  current_status: 2
}

What it should look like:

{
  trip: {
    trip_id: "10078005221731-JUNE22",
    trip_route_id: "78-13157",
    trip_start_date: "20221014"
  },
  vehicle: {
    vehicle_id: "5818",
    vehicle_label: "5818"
  },
  position: {
    position_latitude: 34.05466079711914,
    position_longitude: -118.2251968383789,
    position_bearing: 243.2840576171875,
    position_speed: 5.900928020477295
  },
  current_stop_sequence: 39,
  stop_id: "17258",
  current_status: 2
  timestamp: 1665795819
}

Originally posted by @matikin9 in #69 (comment)

Store Rail GTFS in the database using the existing tables but specifying the agency_id.

LACMTA_Rail's trip_updats data doesn't include vehicle_id

Originally posted by @matikin9 in #70 (comment)

• upcoming_stop_time_update is sometimes null even if trip_assigned is true. Not sure exactly what this situation signifies but will investigate.
• In the case that no trip is assigned, return the result inside an array to match the normal return format:
  
• Return destinationCode value from stop_times.

More fields we can add filters for:

• Trip Updates

  • route_id
  • stop_id

• Vehicle Positions

  • trip_id
  • trip_route_id
  • stop_id

Originally posted by @matikin9 in #70 (comment)

Update api.metro.net to cache real time data and return it at the vehicle level (aim for 1 minute)

Save the GTFS-rt data from Swiftly into the Postgres database. Maybe save certain fields into table field but not everything, and just save the json as a string into its own field?

Specs for the endpoints are here:

• https://gtfs.org/realtime/reference/

Resource for JSON fields in Postgres:

• tiangolo/fastapi#2284

Resources for RESTful API best practices:

• https://amdigital-co-uk.github.io/6.-Engineering/Quality-Standards/Endpoint-Naming-Conventions/#filtering-and-pagination
• https://www.freecodecamp.org/news/rest-api-best-practices-rest-endpoint-design-examples/

Endpoint examples

• {service}/trip_updates/
• {service}/trip_updates/{trip_id}
• {service}/vehicle_positions/
• {service}/vehicle_positions/{vehicle_id}
• {service}/stop_times/

service = bus or rail

Vehicle Positions

Example object with no trip associated:

{
    "id": "5819",
    "vehicle": {
        "position": {
            "latitude": 34.055405,
            "longitude": -118.24849,
            "bearing": 308,
            "speed": 5.36448
        },
        "timestamp": "1661987614",
        "vehicle": {
            "id": "5819",
            "label": "5819"
        }
    }
}

Example object with a trip associated:

{
    "id": "5818",
    "vehicle": {
        "trip": {
            "tripId": "10070002941645-JUNE22",
            "startDate": "20220831",
            "routeId": "70-13157"
        },
        "position": {
            "latitude": 34.05364,
            "longitude": -118.23103,
            "bearing": 303,
            "speed": 0.44704
        },
        "currentStopSequence": 1,
        "currentStatus": "STOPPED_AT",
        "timestamp": "1661987603",
        "stopId": "6535",
        "vehicle": {
            "id": "5818",
            "label": "5818"
        }
    }
}

Trip Updates

Example object with a canceled trip:

{
    "id": "10180002431820-JUNE22",
    "tripUpdate": {
        "trip": {
            "tripId": "10180002431820-JUNE22",
            "startTime": "18:20:00",
            "startDate": "20220831",
            "scheduleRelationship": "CANCELED",
            "routeId": "180-13157",
            "directionId": 0
        },
        "timestamp": "1661987602"
    }
}

Example object with trip predictions:

{
    "id": "10210002981622-JUNE22_8302_58920",
    "tripUpdate": {
        "trip": {
            "tripId": "10210002981622-JUNE22",
            "startTime": "16:22:00",
            "startDate": "20220831",
            "scheduleRelationship": "SCHEDULED",
            "routeId": "210-13157",
            "directionId": 0
        },
        "stopTimeUpdate": [
            {
                "stopSequence": 1,
                "departure": {
                    "time": "1661988355"
                },
                "stopId": "30013",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 2,
                "arrival": {
                    "time": "1661988763"
                },
                "stopId": "4986",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 3,
                "arrival": {
                    "time": "1661988819"
                },
                "stopId": "4950",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 4,
                "arrival": {
                    "time": "1661989054"
                },
                "stopId": "4798",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 5,
                "arrival": {
                    "time": "1661989074"
                },
                "stopId": "4824",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 6,
                "arrival": {
                    "time": "1661989106"
                },
                "stopId": "4827",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 7,
                "arrival": {
                    "time": "1661989232"
                },
                "departure": {
                    "time": "1661989302"
                },
                "stopId": "921",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 8,
                "arrival": {
                    "time": "1661989368"
                },
                "stopId": "25400147",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 9,
                "arrival": {
                    "time": "1661989388"
                },
                "stopId": "882",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 10,
                "arrival": {
                    "time": "1661989417"
                },
                "stopId": "873",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 11,
                "arrival": {
                    "time": "1661989484"
                },
                "stopId": "850",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 12,
                "arrival": {
                    "time": "1661989554"
                },
                "stopId": "25400148",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 13,
                "arrival": {
                    "time": "1661989604"
                },
                "stopId": "865",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 14,
                "arrival": {
                    "time": "1661989667"
                },
                "stopId": "881",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 15,
                "arrival": {
                    "time": "1661989693"
                },
                "stopId": "847",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 16,
                "arrival": {
                    "time": "1661989797"
                },
                "stopId": "915",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 17,
                "arrival": {
                    "time": "1661989812"
                },
                "stopId": "859",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 18,
                "arrival": {
                    "time": "1661989871"
                },
                "departure": {
                    "time": "1661989941"
                },
                "stopId": "862",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 19,
                "arrival": {
                    "time": "1661990008"
                },
                "stopId": "4912",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 20,
                "arrival": {
                    "time": "1661990083"
                },
                "stopId": "141008",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 21,
                "arrival": {
                    "time": "1661990129"
                },
                "stopId": "4910",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 22,
                "arrival": {
                    "time": "1661990181"
                },
                "stopId": "4911",
                "scheduleRelationship": "SCHEDULED"
            },
            {
                "stopSequence": 23,
                "arrival": {
                    "time": "1661990233"
                },
                "stopId": "880",
                "scheduleRelationship": "SCHEDULED"
            }
        ],
        "vehicle": {
            "id": "8302"
        },
        "timestamp": "1661987567"
    }
}

Only matters for the predictions endpoint

Related:

• #23

Schedule pages currently use the Timetables API. We should replicate the endpoints needed so that we can switch over to using api.metro.net instead.

Some of these will need to be tested after the push to Prod.

• return full stop_name - returning just the first character
• return full route_code - returning just the first character
• return upcoming_stop_time_update with unsigned integer type values for departure and arrival
• return string value for schedule_relationship instead of the enum integer
• inside upcoming_stop_time_update, return just stop_sequence, stop_id, departure or arrival, schedule_relationship

Using this library we should encode the polyline so that it can be used in the front end Metro API.

• Reference GTFS-static values for service.
• Standardize endpoint structuring
  • calendar_dates -> gtfs-static\calendar_dates
• #10