can we add a test on the spatial extent to make sure it's not wider or longer than, e.g. an MSA? Let us know if you want distance inputs for this.

from George:
I was running the code on Jerry's data and it was taking an unexpectedly long time, then I ran out of hard drive space. There are a couple coordinates in the data which are far outside of Chicago (LA and DC actually), and a number that are in the Chicago suburbs. I think this just underscores the importance of adding some of the data checks. Ultimately, I think we do want a input data verification in the web app, but until we have that, it may be good to add a quick check in the back-end code?

Comparing Dan's matrix, a difference of 10 minutes on average.

Give the user the option to use TransitMatrix as opposed to TransitMatrix

The driving results show non-zero values in the diagonal.
Do you know why is that?

Bring spatial join (ScoreModel) back to get the community areas for each destination without clipping the destinations so we don't lose points outside the city boundaries.

Before:
Deprecated spatial join (ScoreModel) because points outside of the Chicago community boundaries are dropped (losing dest observations). In the health example, 17 points lay outside the city of Chicago boundaries going from 199 to 182 observations.

Check for conversions in C++ and Python (cost_model, nn_primary, nn_secondary)

Applies to commit: https://github.com/GeoDaCenter/spatial_access/tree/5fe8717bd6bed34b023a53b47e8f3ec49f48f201

Package builds but is unable to be imported because the .so is not found:

---> 1 from transitMatrixAdapter import *

ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/transitMatrixAdapter.cpython-35m-darwin.so, 2): Symbol not found: __ZN6google8protobuf7Message20DiscardUnknownFieldsEv
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/transitMatrixAdapter.cpython-35m-darwin.so
  Expected in: flat namespace
 in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/transitMatrixAdapter.cpython-35m-darwin.so

When running these files I got this error:

I created a new ID column instead of geoid10, but not sure what the problem is.

Hey Logan, looks like another test clause needs to be added to _test_category_weight_dict() in Models.py, AccessModel.

See build: https://travis-ci.org/GeoDaCenter/spatial_access/builds/457749357

Travis build fails because protobuf headers are not found. StackOverflow reports that the following step solves the issue:
sudo apt-get install libprotobuf-dev protobuf-compiler
However that does not appear to solve the issue in this case (see .travis.yml)

Low priority, but probably worth addressing eventually.

Scenario: an input destination file has a field named "category", but the user points to a different a field (e.g., "category_1") as containing category values.
The user also provides a list of values to the "categories" parameter of the Coverage class constructor.

It looks like the code never maps the internal data model's "category" field to the one provided by the user. It looks for the category values in the list provided by the user in the destination file's "category" field instead of the field specified and can't find them.

The main scenario I could see this coming up in is where a user has a legacy field "category", and then a current field called "category_facility" or "category_1" etc containing the 'real' category values. I've seen this kind of thing pretty frequently given the poor data management practices out there, the difficulty of updating schemas...

#49

Add the ability for the user to input a category weight dict that overrides the default weight list for the categories specified when calculating Access scores (AccessModel).

This way, users won't have to derive a list of categories each time to set a few exceptions. I think there would be enough demand for such a feature that it makes sense to build it into the module.

@ifarah Any thoughts about this, Irene?

Print out the categories used in access model.
Whenever the user does not specify a category in the weight's dictionary, the default is DIMINISH_WEIGHTS. It would be great if the categories used in the model would be printed out so the user has more control of the categories computed.

was pending pull request, see:
UDST/osmnet#11

Add 6th “model type” for dest attribute aggregation for two new results:

DestSum: Sum of attribute of point destination within boundaries of larger area (like community area)
Example: Total funding going to providers in a cmty area

PCDestSum: Same as DestSum but per capita: Sum of attribute of point destination within boundaries/ Sum of pop within boundaries
Example: Total funding going to providers in a cmty area/ per population in that area

Dan's transit matrices from OpenTrip Planner are stored in a different format (closer to our spatial weights matrix format). One example for Seattle (he's generating this for 20 cities):

https://lab.dfsnow.me:8080/matrix/seattle_matrix.csv.bz2

Please convert this format to the one that p2p can read as input.

Talking to @ifarah, I'll take a look at how to package/include this in pysal.

For me to get this through, we'd have to make sure this follows the "submodule contract"we wrote to ensure consistency across parts of pysal.

Not quite sure where this is after our website reorganization, but I'll look into it.

Add the ability for users to specify a default category weight list to AccessModel. Julia agreed that this is useful functionality.

@ifarah Would you prefer that the user has to specify the default list whenever calculating access scores, or on top of the ability to set a default list when running the code, would it also be useful to have it hard-coded into the module (i.e., in Configs.py) so the users doesn't have to set a default list each time? I believe category weights aren't relevant to AccessCount, AccessSum, AccessTime, right?

I compared Logan's, Dan's, and GoogleMaps' results for a tract in Lincoln Park, Hyde Park, and Midway:
The results increased but are still below. Could you increase the impedance a bit more so I can rerun and revalidate?

1. LP:
   Origin: 17031071100 (41.921800, -87.651100)
   Destination: 17031460400 (41.744500, -87.563700)
   Logan:1351 (22.52) NOW: 1357 (22.61)
   Dan:1440 (24)
   Googlemaps: 26-35 min

2. HP
   Origin: 17031836200 (41.790469, -87.601284)
   Destination: 17031170100 (41.955900, -87.794800)
   Logan:1468 (24.46) NOW: 1762 (29.36)
   Dan:1980 (33)
   Googlemaps: 28-60 min

3. Midway
   Origin: 17031561100 (41.788869, -87.771909)
   Destination: 17031030400 (41.986952, -87.672079)
   Logan:1674 (27.9) NOW: 1886 (31.43)
   Dan:2040 (34)
   Googlemaps: 30-50 min

An extra column of blank string values (rightmost column) is getting written into the transit matrix. TransitMatrix.write_to_file() fails when it tries to cast the blank values as floats.

Below is the error that gets thrown:

Traceback (most recent call last):

File "command_line_test.py", line 87, in

upper=int(30))

File "/Users/georgeyoliver/GitHub/CSDS/GeoDaCenter/spatial_access/spatial_access/CommunityAnalytics.py", line 301, in init

self.load_sp_matrix(sp_matrix_filename)

File "/Users/georgeyoliver/miniconda3/envs/web_app/lib/python3.6/site-packages/spatial_access/ScoreModel.py", line 307, in load_sp_matrix

self.dicto[row[0]][int(float(self.dest_2[i]))] = int(float(row[i]))     

ValueError: could not convert string to float:

Could you add:

AccessSum

Same to our AccessCount but this would sum an attribute of the destination (e.g. funding amount) within the catchment area of an origin point instead of counting the number of destination points.

Add a feature to natively support prefetching networks from OSM, without continuing with matrix computation.

if you could update:

FloatingCatchmentAreaDest --> 2SFCA (if that's the 2nd column = sum of ratios at origin)
FloatingCatchmentArea --> coverage (if that's the ratio only at destination)

Creating a more efficient way of importing the matrix into ScoreModel

The output of AccessModel gives results for "good_access".
I still don't understand the reasoning for including those results in the output instead of just having the results of the access score.

These two tables with aggregate results would be very useful to have:

Generate a solution to run parcels to amenities for all the US.

Hello Logan,
I believe the code is throwing this error for latitudes, reading from the longitude column.
(see error below)
I specified latitudes as coord_y and longitude as coord_x. coord_x is indeed ranging [-91,90], but coord_y is ranging [36,40]

I'm also copying and pasting the error just in case it's easier for you.

INFO:spatial_access.p2p:Total number of rows in the dataset: 119
/usr/local/lib/python3.6/site-packages/geopy/point.py:81: UserWarning: Latitude normalization has been prohibited in the newer versions of geopy, because the normalized value happened to be on a different pole, which is probably not what was meant. If you pass coordinates as positional args, please make sure that the order is (latitude, longitude) or (y, x) in Cartesian terms.
UserWarning)

ValueError Traceback (most recent call last)
in ()
----> 1 matrix.process()

/usr/local/lib/python3.6/site-packages/spatial_access-0.1.6.13-py3.6-macosx-10.13-x86_64.egg/spatial_access/p2p.py in process(self)
461 self.network_type, self.epsilon)
462
--> 463 self.prefetch_network()
464
465 is_symmetric = self.secondary_input is None and self.network_type is 'drive'

/usr/local/lib/python3.6/site-packages/spatial_access-0.1.6.13-py3.6-macosx-10.13-x86_64.egg/spatial_access/p2p.py in prefetch_network(self)
440 self.secondary_data,
441 self.secondary_input is not None,
--> 442 self.epsilon)
443
444 def clear_cache(self):

/usr/local/lib/python3.6/site-packages/spatial_access-0.1.6.13-py3.6-macosx-10.13-x86_64.egg/spatial_access/NetworkInterface.py in load_network(self, primary_data, secondary_data, secondary_input, epsilon)
157 self._try_create_cache()
158 self._get_bbox(primary_data, secondary_data,
--> 159 secondary_input, epsilon)
160 if self._network_exists():
161 filename = self.get_filename()

/usr/local/lib/python3.6/site-packages/spatial_access-0.1.6.13-py3.6-macosx-10.13-x86_64.egg/spatial_access/NetworkInterface.py in _get_bbox(self, primary_data, secondary_data, secondary_input, epsilon)
100 self.bbox = [lon_min, lat_min, lon_max, lat_max]
101 if self.area_threshold:
--> 102 approx_area = self._approximate_bbox_area()
103 if approx_area > self.area_threshold:
104 if self.logger:

/usr/local/lib/python3.6/site-packages/spatial_access-0.1.6.13-py3.6-macosx-10.13-x86_64.egg/spatial_access/NetworkInterface.py in _approximate_bbox_area(self)
56 lower_right_point = (self.bbox[3], self.bbox[0])
57 upper_left_point = (self.bbox[1], self.bbox[2])
---> 58 lower_edge = distance.distance(lower_left_point, lower_right_point).km
59 left_edge = distance.distance(lower_left_point, upper_left_point).km
60 area = lower_edge * left_edge

/usr/local/lib/python3.6/site-packages/geopy/distance.py in init(self, *args, **kwargs)
382 kwargs.pop('iterations', 0)
383 major, minor, f = self.ELLIPSOID # pylint: disable=W0612
--> 384 super(geodesic, self).init(*args, **kwargs)
385
386 def set_ellipsoid(self, ellipsoid):

/usr/local/lib/python3.6/site-packages/geopy/distance.py in init(self, *args, **kwargs)
160 elif len(args) > 1:
161 for a, b in util.pairwise(args):
--> 162 kilometers += self.measure(a, b)
163
164 kilometers += units.kilometers(**kwargs)

/usr/local/lib/python3.6/site-packages/geopy/distance.py in measure(self, a, b)
403 # Call geographiclib routines for measure and destination
404 def measure(self, a, b):
--> 405 a, b = Point(a), Point(b)
406 lat1, lon1 = a.latitude, a.longitude
407 lat2, lon2 = b.latitude, b.longitude

/usr/local/lib/python3.6/site-packages/geopy/point.py in new(cls, latitude, longitude, altitude)
169 )
170 else:
--> 171 return cls.from_sequence(seq)
172
173 if single_arg:

/usr/local/lib/python3.6/site-packages/geopy/point.py in from_sequence(cls, seq)
408 raise ValueError('When creating a Point from sequence, it '
409 'must not have more than 3 items.')
--> 410 return cls(*args)
411
412 @classmethod

/usr/local/lib/python3.6/site-packages/geopy/point.py in new(cls, latitude, longitude, altitude)
181
182 latitude, longitude, altitude =
--> 183 _normalize_coordinates(latitude, longitude, altitude)
184
185 self = super(Point, cls).new(cls)

/usr/local/lib/python3.6/site-packages/geopy/point.py in _normalize_coordinates(latitude, longitude, altitude)
80 '(latitude, longitude) or (y, x) in Cartesian terms.',
81 UserWarning)
---> 82 raise ValueError('Latitude must be in the [-90; 90] range.')
83
84 if abs(longitude) > 180:

ValueError: Latitude must be in the [-90; 90] range.

Throw error if the extremes of the bounding box are too big. Maybe define a feasible area for estimating the bounding box and if it exceeds this area, throw error?

Now it's only reading H5.
Now user cannot compute metrics if she has a csv matrix.

For backend users:
Lines 249-276 in p2p.py denominate xcol as latitude, when the x coordinate are the equivalent of longitude. This might be confusing for users, so change xcol to longitude and ycol to latitude.

For estimating meters, this ran super fast but gave me almost all "infinity" since the distances are really large. Is there a way to increase "infinity"?

Make sure that GPL license and your authorship are listed on both your code and the API.

Previously, Shiv implemented Kosaraju's algorithm to find the strongly connected components of the network. Some of the observations of the access score were zero because the nodes were not connected to the network.

Now the values are set to 65,535 and should be set as missing meanwhile.

geopandas spatial join requires installation of libspatialindex by brew/apt-get. Try to find an alternative implementation of spatial joins to simplify installation

Please update the chart that explains how the modules fit together, so we can update below in the notebooks (we have a Lucidchart subscription -- let me know if you need log-in):

https://github.com/GeoDaCenter/spatial_access/blob/master/docs/notebooks/notes.ipynb