Giter Club home page Giter Club logo

invest's Introduction

InVEST: Integrated Valuation of Ecosystem Services and Tradeoffs

InVEST (Integrated Valuation of Ecosystem Services and Tradeoffs) is a family of tools for quantifying the values of natural capital in clear, credible, and practical ways. In promising a return (of societal benefits) on investments in nature, the scientific community needs to deliver knowledge and tools to quantify and forecast this return. InVEST enables decision-makers to quantify the importance of natural capital, to assess the tradeoffs associated with alternative choices, and to integrate conservation and human development.

Older versions of InVEST ran as script tools in the ArcGIS ArcToolBox environment, but have almost all been ported over to a purely open-source python environment.

Note

This repository is for InVEST 3.2.1 and later

This replaces our Google Code project formerly located at http://code.google.com/p/invest-natcap. If you are looking to build InVEST <= 3.2.0, use the archived release-framework repository at https://bitbucket.org/natcap/invest-natcap.release-framework, and the InVEST repository at https://bitbucket.org/natcap/invest-natcap.invest-3.

General Information

Dependencies

Run make check to test if all required dependencies are installed on your system. OS-specific installation instructions are found either online at http://invest.readthedocs.io/en/latest/installing.html or locally at doc/api-docs/installing.rst.

Note

The make commands for InVEST require a BASH shell environment. Windows users can use Git Bash within the Git for Windows suite. More information can be found at https://gitforwindows.org

Managing python dependencies

We recommend using a virtual environment to manage your python dependencies, and there is a Makefile target to assist with this:

$ make env
$ source env/bin/activate

Or on Windows, use the following instead from a CMD prompt:

> make env
> .\env\bin\activate

This makefile target is included for convenience ... you may of course choose to manage your own virtual environment. requirements.txt, requirements-dev.txt and requirements-docs.txt list the python dependencies needed.

Using a different environment name

If you prefer a different name for your environment, you may pass the environment name as a parameter to make:

$ make ENV=myEnv env

You could then activate the environment created at myEnv.

Using a different environment management tool

The InVEST Makefile uses virtualenv to set up an environment, but this is not the only environment management tool out there. You may elect to manage your virtual environment a different way, independent of make env. The only requirement for the build process is that the required tools are available on your PATH and the required python packages can be imported.

Building InVEST Distributions

Once the required tools and packages are available, we can build InVEST.

Building natcap.invest python package

A Makefile target has been created for your convenience:

$ make python_packages

This will create a wheel for your platform and a zip source archive in dist/. Both of these files (dist/natcap.invest*.whl and dist/natcap.invest*.zip) can be installed by pip.

Building python packages without GNU make

Python distributions may be built with the standard distutils/setuptools commands:

$ python -m pip install build
$ python -m build --wheel
$ python -m build --sdist

InVEST Standalone Binaries

Once the appropriate dependencies are available, InVEST can also be built as a standalone application:

$ make binaries

An important detail about building binaries is that natcap.invest must be installed as a wheel to ensure that the distribution information is in the correct location.

This will create a directory at dist/invest holding the application binaries and relevant shared libraries.

Binaries cannot be cross-compiled for other operating systems.

InVEST Workbench

See developer instructions at workbench/readme.md.

Building InVEST Documentation

User's Guide

To build the user's guide:

$ make userguide

This will build HTML and PDF documentation, writing them to dist/userguide and dist/InVEST_*_Documentation.pdf, respectively.

API Documentation

To build the natcap.invest python API documentation and developer's guide:

$ make apidocs

This will build an HTML version of the API documentation, writing it to dist/apidocs.

InVEST Sample Data

InVEST is typically distributed with sample data, though, in the interest of disk space, these data are not included in any of the standard installers. To build zip archives of the sample data:

$ make sampledata

This will write the data zipfiles to dist/data. git command is needed.

Single archive of sample data

For trainings, it is especially convenient to distribute all sample data as a single zip archive. As an added bonus, this single zip archive can be provided to the InVEST installer for Windows as either the 'Advanced' input on the front page of the installer, or by a CLI flag, thus preventing the installer from downloading datasets from the internet. See installer/windows/invest_installer.nsi for more details. To build a single archive of all InVEST sample data:

$ make sampledata_single

This will write the single sampledata archive to dist/InVEST_*_sample_data.zip.

Tests

InVEST includes a battery of tests to ensure software quality.

Model tests

To run tests on the suite of Ecosystem Service models in InVEST:

$ make test

Changing how GNU make runs tests

The InVEST Makefile setup depends on pytest and coverage to display line coverage and produce HTML and XML reports. You can force make to use coverage with a different test runner by setting a parameter at the command line. For example, to run the tests with nose:

$ make TESTRUNNER=nose test

Running tests on binaries

This repository includes a python script to automatically execute and check the exit status of all InVEST models, running on the installed InVEST sample data. Once all sample data have been fetched and binaries built on the target computer:

$ make invest_autotest

Copyright and license information

A file called LICENSE.txt should have accompanied this distribution. If it is missing, the license may be found on our project page, https://github.com/natcap/invest

invest's People

Contributors

davemfish avatar dcdenu4 avatar emilyanndavis avatar emlys avatar joannalcy avatar omahs avatar phargogh avatar richpsharp avatar tomsaleeba avatar wbierbower avatar willb2 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

invest's Issues

Fisheries-HST FloatingPointError: invalid value encountered in true_divide

This sometimes causes our tests to fail (https://ci.appveyor.com/project/davemfish/invest/builds/28437818/job/y1psai9ejk8h2htg)

It's clear what the problem is, but unclear what the correct scientific solution is.

fisheries_hst.py line 144 divides by array n_a, which is populated by a user-input matrix of 1s and 0s (see lines 109, 110 for n_a, and trace it back to the user input in fisheries_hst_io.py)

fisheries_hst.pyline 110 is especially suspicious: n_a[n_a == 0] = 0

Originally reported by @davemfish at https://bitbucket.org/natcap/invest/issues/3925/fisheries-hst-floatingpointerror-invalid

Enable diagnostic logging in InVEST applications

As a developer, I occasionally would like to see diagnostic logging from the User Interface while it is running, without needing to rebuild the binaries and re-run the application. This would have been really useful in the InVEST 3.8.0 release process, where my mac laptop's homebrew configuration was interfering with the InVEST binaries I was attempting to run. The specific issues observed were:

  1. My homebrew-installed GEOS (brew install geos) and GDAL (brew install osgeo-gdal-python) were being used instead of the bundled GEOS and GDAL. It would have been nice to see, if possible, where those libraries were being loaded from.
  2. When I launched the InVEST binaries like normal (by double-clicking the .app), I was able to interact just fine with the inputs pane. But when I wanted to run the application itself, I could press the 'run' button, but nothing would happen. If I had access to the logging of the UI, I'd be able to take a look at what was going on. This issue was not reproducible when running the binaries through a shell.

InVEST uses python's logging package to log messages throughout the various layers of the application. It would be temendously useful if logging could be directed to an appropriate place during application execution. Possibilities to consider include:

  1. the OS-specific logging monitor (console.app on mac, Event Viewer on Windows)
  2. a file somewhere on disk (and is this possible when using the mac app sandboxing?)
  3. a widget in the InVEST GUI

When we figure this out and implement a fix, it would be nice to document in the InVEST API docs, User's Guide, or somewhere else useful to us for later reference.

  • Implement diagnostic logging to an appropriate place
  • Document how to access diagnostic logging in the API docs or somewhere else

Design an InVEST automated release process

The InVEST release process has been quite manual (documented here: https://bitbucket.org/natcap/invest/wiki/Release Checklist) and pretty involved, requiring a nontrivial amount of engineering time to become confident in our releases before we send it out.

In short, it would be a fantastic improvement to our team productivity if we could design a process that:

  • Instills confidence in our releases by thoroughly testing the final executable. Related tasks for this are:
    • Testing of models via the InVEST user interface would be the most useful thing to do here
    • Building and testing the python packages directly across all supported architectures and versions
  • Requires minimal (if any) human interactions in order to complete the release. Related tasks would be:
    • Tagging the UG repository with the correct revision found in Makefile (or possibly failing the build if the rev in Makefile is really out of date?)
    • Updating HISTORY.rst with the correct date
    • Automatically incrementing the InVEST release version number as part of the build process
    • Building the release, including:
      • Uploading artifacts to our storage bucket
      • Uploading python binaries to PyPI
    • Announcing the latest InVEST release to interested audiences via needed channels (email? slack? forums?)

Other related questions we’ll want to consider are:

  • Whether the release process above should happen for all commits, or if a release should happen at most once per unit of time (once per day? once per week?)
  • Should there be a separate branch (separate from develop) to indicate features that should be released via this automated release process?
  • How will errors that come up during an automated release be handled?

improve error handling in Habitat Quality when nodata is undefined

If the nodata value of a threat raster is undefined, but values other than 0 or 1 are present, we crash with: TypeError: ufunc 'isfinite' not supported for the input types

Here's an example from a user: https://community.naturalcapitalproject.org/t/typeerror-ufunc-isfinite-in-hq-model/874/2

It would be pretty straightforward to wrap this in a try, except and give clear direction to the user to check their nodata value:

zero_threat = numpy.isclose(block, threat_nodata)

minor difference in logfile formats from UI vs CLI

This actually has implications for datastack.extract_parameters_from_logfile, which expects to find a line starting with "Arguments".

It's simply a matter of adding a newline in the log message from cli.py.

Rec model memory errors

The recreation raises a MemoryError on very large AOIs.

To recreate this issue, create a vector polygon that extends over the continental United States. Then run the model with that single polygon and observe a server-side memory error after a few minutes.

Previous debugging has shown the memory error occurs during the point collection phase where a long list of all covered points is created in memory.

There may be a variety of solutions including an out of core ponit list, an enhanced algorithm that sidesteps the need to maintain a point list in memory, or advanced techniques of subdividing large polygons such that sub polygon's points fit in memory, then combining the result after the fact.

We'll get to this post PyGeoprocessing 1.0 refactor.

Originally reported by @richpsharp at https://bitbucket.org/natcap/invest/issues/3614/rec-model-memory-errors

Move binary builds to Github Actions

Once #24 and issue #22 are resolved, it would be really, really nice to move our binary build process from AppVeyor to Github Actions. The problems with AppVeyor binary builds are:

  1. The build takes almost a half-hour, with a substantial chunk of build time in the activation of the Google Cloud environment. I can only guess at why that might be.
  2. Appveyor builds happen sequentially, and, due to the runtime issue noted in item 1, tend to take a really long time since builds for natcap/invest and phargogh/invest are both attached to my appveyor account.
  • Add a build job
  • Remove AppVeyor configuration
  • Note change in HISTORY.rst

PyInstaller can't find DLLs on Windows

When building the InVEST binary build on github actions for #26 , I noticed that about 7.5 minutes of the build time was being occupied by the pyinstaller part of the build, and, specifically, the part of the build where it is Looking for dynamic libraries and generating a pile of warnings that look like this:

278982 WARNING: lib not found: libbispeu.KX7AQLB2Z5NFTCADW76YV5UMLBQNDALD.gfortran-win32.dll dependency of C:\hostedtoolcache\windows\Python\3.7.6\x86\lib\site-packages\scipy\interpolate\_fitpack.cp37-win32.pyd
281314 WARNING: lib not found: libdfitpack.EM6D2ZEFEP35UVNTFVXVVRYRWM22ZXEY.gfortran-win32.dll dependency of C:\hostedtoolcache\windows\Python\3.7.6\x86\lib\site-packages\scipy\interpolate\dfitpack.cp37-win32.pyd
299317 WARNING: lib not found: libansari.Q4BAGRNANLWD2YZJOKYPOAUIOLXW2LXK.gfortran-win32.dll dependency of C:\hostedtoolcache\windows\Python\3.7.6\x86\lib\site-packages\scipy\stats\statlib.cp37-win32.pyd

Example build job with these warnings: https://github.com/phargogh/invest/runs/471001452?check_suite_focus=true

The build runs so this isn't exactly an error, but it sure would be nice to trim ~7.5 minutes off of our ~20 minute runtime for the build job.

Marking this with a complexity estimate of 13 to indicate the possible worst-case situation for this fix. It's quite possible that just upgrading pyinstaller would fix the issue, or maybe it's an environment thing (the conda-based mac build doesn't have this issue), but it could also be something deeper.

Homebrew-installed libraries interfere with InVEST mac binaries

When testing out the InVEST 3.8.0 release binaries, I could launch and run the models by reaching into the application and running the compiled invest binary. But then if I tried running a model that loaded geometries from a vector, the model would segfault. The segfault exception when n_workers=-1 looked like this:

Current thread 0x0000700010e0e000 (most recent call first):
  File "env/lib/python3.7/site-packages/shapely/geometry/polygon.py", line 494 in geos_polygon_from_py
  File "env/lib/python3.7/site-packages/shapely/geometry/polygon.py", line 240 in __init__
  File "env/lib/python3.7/site-packages/shapely/geometry/geo.py", line 19 in box
  File "env/lib/python3.7/site-packages/taskgraph/Task.py", line 1071 in _call
  File "env/lib/python3.7/site-packages/taskgraph/Task.py", line 389 in _task_executor
  File "env/lib/python3.7/threading.py", line 870 in run
  File "env/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "env/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x0000700010408000 (most recent call first):
  File "env/lib/python3.7/threading.py", line 296 in wait
  File "env/lib/python3.7/threading.py", line 552 in wait
  File "env/lib/python3.7/site-packages/taskgraph/Task.py", line 1231 in join
  File "env/lib/python3.7/site-packages/taskgraph/Task.py", line 713 in join
  File "env/lib/python3.7/site-packages/natcap/invest/scenic_quality/scenic_quality.py", line 458 in execute
  File "env/lib/python3.7/site-packages/natcap/invest/ui/model.py", line 1645 in _logged_target
  File "env/lib/python3.7/site-packages/natcap/invest/ui/execution.py", line 68 in run
  File "env/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "env/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x000000010b383dc0 (most recent call first):
  File "src/natcap/invest/cli.py", line 561 in main
  File "src/natcap/invest/cli.py", line 575 in <module>
Segmentation fault: 11

After digging into this some more, it turned out that my homebrew install had GEOS 3.7.3 and GDAL 3.0.1 installed due to a homebrew-managed QGIS install, and that the InVEST binaries were clearly loading the GEOS binaries from the homebrew Cellar (verified by sampling the files that the invest process had open, visible in Activity Monitor). See the attached sample log for more information.

My questions around this are:

  1. why was GEOS being loaded from the Cellar and not from the bundled binaries?
  2. how can we prevent this from happening in the future?

It would appear that pyinstaller modifies LD_LIBRARY_PATH ... maybe that has something to do with this?

Refactor the HRA viz component to not depend on redux.

Right now this is the only component of this project depending on redux to manage state. The use of redux here is just a legacy from the prototyping of the HRA app, not a real design decision.

So unless we decide that redux is useful elsewhere, we can probably do without it here.

Consistently and helpfully handle reclassification errors across InVEST

Despite how common reclassifications are across InVEST, we don't handle errors very consistently. If a model's biophysical table is missing a row and that landcover code is encountered, either a KeyError is raised, or else there's a cryptic error coming from numpy about bins.

If there's an issue around reclassifications, InVEST should consistently and helpfully handle errors like these by raising a human-readable error message that contains enough information for a GIS person to decipher the source of the issue. Stacie Wolny specifically referenced the error message at https://bitbucket.org/natcap/invest/src/13fa555b96402d9b463aa524b170de01c046a7bf/src/natcap/invest/ndr/ndr.py#lines-931 as being clear and helpful.

SDR's LS equation does not account for MFD

When we refactored SDR to use Multiple Flow Direction, we failed to update SDR's RKLS calculations to account for the fact that aspect (a component of the LS equation) should now account for the possibility that a pixel can flow into 8 possible neighbors. LS does not currently account for this.

After talking about this with Rich, we decided on the following, corrected approach:

Given

FV = "flow value", the packed 64-bit, 8-direction (4 bits per direction) value coming from MFD Flow Direction
d = The integer direction in the form:

3 2 1
4 x 0
5 6 7

P_i(d) = The proportion of flow in pixel i in direction d
LS_i' = A modified LS equation that removes the x_d term entirely
x_d = The length of the pixel from x to neighbor d. If d is even, this will be 1, sqrt(2) otherwise.

Calculate the following

  1. Determine the proportional flow P_i(d) = FV(d) / sum(FV(j) for j in {0..7})
  2. The weighted average of the aspect based on flow value: FV_avg = sum(1/(x_d) * P_i(d) for d in {0..7})
  3. Calculate LS_i'
  4. Calculate the modified RKLS: RKLS_i = R_i * K_i * LS_i' * FV_avg

TODOs

  • Implement
  • Test
  • Update the User's Guide
  • Ask Rich to look over the updated UG
  • Note change in HISTORY.rst
  • PR this into InVEST

Datastacks should be able to be loaded on any operating system

As a cross-platform developer, I sometimes need to create a datastack archive or parameter set on Windows or Mac OS X and then load the datastack on the other operating system. When I create a datastack on mac and load it on Windows, it works well. When I create a datastack on Windows and load it on mac, it doesn't. Looks like the paths are being saved with \\ separators and perhaps that's a part of it.

Reestablish automated tests

On bitbucket, we had some appveyor integration that worked pretty well, but would take forever. We still need to test things, but I think we should revisit where these individual steps are done:

Appveyor

I think we should keep the binary build on appveyor for the moment, at least until we get a binary build working as expected on actions. Since the build already works there, I don't think this part is as critical to move, and execution could still take place in parallel with the other test steps.

Github Actions

Tests

We should move all tests to github actions, as this is a build matrix that can be done in parallel (which is something that AppVeyor doesn't provide without paying for it). Let's start out by replicating the builds we currently have on appveyor:

  • Python 3.7 32-bit model tests
  • Python 3.7 64-bit model tests
  • Python 3.7 32-bit UI tests

If possible, it'd also be nice to run model tests on both python 3.6 and python 3.7 (32- and 64-bit architectures)

Sample Data Validation

We should do this on an available python install ... maybe whatever's on the PATH.

Components

  • Update Makefile to work with git rather than mercurial
  • Set Up GitHub Action for sample data validation
  • Set Up GitHub Actions for model tests
  • Set Up GitHub Actions for UI Tests
  • Trim AppVeyor file to only include what it needs

Fix logic in `make deploy` so it works in Windows and Mac build pipelines

Right now the build pipelines share the make deploy target, which is good. But since we don't build the sample data in the Mac pipeline, make deploy crashes on the Mac when it tries to rsync the sampledata to our releases bucket.

We can either modify make deploy or the ci/mac_mini_build_process.sh script to better handle this.

Offer a more helpful validation message when a user provides an OVR file

This has happened on the forums a couple of times now, but here's the latest example: https://community.naturalcapitalproject.org/t/pollination-model-error-after-updating-to-invest-3-8-0-unable-to-initiate-run/856

When a user provides an overviews file instead of the GeoTiff it's associated with, the overviews file triggers the validation error stating that the file must be projected. It would be nice if we could check if this file is an overviews file and provide a more specific and helpful error message for this common use case.

Marked as a 3 difficulty because the slightly more interesting thing will be figuring out how to build an overviews file within GDAL as part of the tests. I think maybe @richpsharp has done that before?

Support python 3.8

Python 3.8 is the most recently-released version of python and we should make sure that the package supports it.

  • Add tests for Python 3.8
  • Note support for Python 3.8 in setup.py

Update actions checkout to v2

We just had an issue while working on natcap/taskgraph#14 where we needed to update to actions/checkout@v2 for the actions themselves to work as expected. Let's update to v2 here as well across our various workflows.

Expose raster/driver creation options in InVEST Settings

Occasionally (this has come up twice now to my knowledge), successive reads/writes to an LZW-compressed GeoTiff, such as within a batch script, will result in errors like the one below being produced.

ERROR 1: LZWDecode:Corrupted LZW table at scanline 1280
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: C:\Temp\SWY_sensitivity_test2\L_sum_avail_CMCC_TFA_500.tif, band 1: IReadBlock failed at X offset 5, Y offset 5: TIFFReadEncodedTile() failed.
Traceback (most recent call last):
  File "src\natcap\invest\seasonal_water_yield\seasonal_water_yield_core.pyx", line 309, in natcap.invest.seasonal_water_yield.seasonal_water_yield_core._ManagedRaster._load_block
AttributeError: 'NoneType' object has no attribute 'astype'

The only known workaround for this at the moment is to write uncompressed GeoTiffs, a change that requires passing the appropriate parameter to each pygeoprocessing function call.

A solution for this might involve touching many components of InVEST and so should probably be designed.

Offer better reporting of GDAL errors when a model crashes

Every once in a while we get errors like the the link below from the forums where the model crashes on one exception but the underlying error (which is logged to the logfile) actually is that they ran out of disk space.

Maybe we can raise an informative exception if there's an error of sufficient severity?

Or maybe we can report the last few GDAL error codes in the log that might have affected the final exception?

https://community.naturalcapitalproject.org/t/example-data-for-annual-water-yield-model/308/10

Remove python2 stuff from throughout natcap.invest

When natcap.invest supported both python2 and python3, we used the future package and a few other things (like conditionally importing Queue vs queue) for compatibility. It would be great to trim this to whatever degree is reasonable.

Known parts to this:

  • mock --> unittest.mock
  • Remove future from requirements.txt
  • Remove future and future-defined imports (builtins, others?) throughout the package
  • Replace instances of basestring with str (hra.py, ... )

Updates to Habitat Quality

Habitat Quality has been updated a couple of times in the past few years, mostly for pygeoprocessing updates, but it's still largely unchanged. Even though we might be doing some major work on it in the near future, it'd be good to bring it up to the latest InVEST development standards and also update it to use taskgraph.

  • Update HQ to use Taskgraph
  • Update HQ to use the latest coding standards

Release InVEST as a 64-bit binary

When we started developing and shipping binaries for InVEST, 32-bit binaries was an obvious choice as so many computers were still running Windows XP or were running on very old hardware. In this modern age, I can't remember the last time I saw someone working with a 32-bit windows computers, and macs are all 64-bit anyways. The only computer system I can think of that runs a 32-bit OS is the Raspberry Pi, but that'll be on linux and would therefore not be using our prebuilt binaries.

  • Check the InVEST usage stats ... do we have any records of people running InvEST on a 32-bit system in recent history?
  • If it's reasonable to only ship 64-bit, we should still run our tests against 32-bit and 64-bit builds of python on Windows if possible.

Append the suffix to any geopackage layers produced by InVEST

InVEST currently produces both ESRI Shapefile and GeoPackage output vectors. When the file suffix is applied to the filename (and thus the layer name) in a Shapefile, layers in a GeoPackage should also have their suffix appended so that the layers will appear in QGIS and Arc with the suffix attached.

GeoPackages are used in the following models/tools (based on a grep):

  • CV
  • Delineateit
  • HRA
  • UFRM
  • Wind Energy

Also, it would be useful to add this to the InVEST Spec for future us.

ALSO: Arc might have a maximum layer name length for geopackages, so let's look into this to make sure we understand the constraints. What's this limit? Does it only apply to certain versions of Arc? Does QGIS have such a limit?

Support GDAL 3+

InVEST tests currently fail when using GDAL 3+. This is mostly related to how GDAL 3 has changed how it handles spatial references, and particularly coordinate transformations. This issue should be blocked on the Pygeoprocessing version of this issue. The main consideration is whether we should support both GDAL 2 and GDAL 3 or move on to GDAL 3 and drop support for GDAL 2. The main thing here is that there are slight variations in spatial coordinates pertaining to bounding boxes and extents. This would mean that in certain cases we would have to carry around expected test data for both GDAL 2 runs and GDAL 3 runs. That is doable, but perhaps not ideal?

  • Pygeoprocessing supports GDAL 3+
  • Update history with changes

pygeoprocessing.routing not showing up in InVEST model logs

When running an InVEST model, specifically an under development version of SDR that uses PyGeoprocessing 1.6.x, logging does not appear in the log for pygeoprocessing.routing, though it does seem to show up as a log to Task._call. James and I did some debugging and we find that outside of InVEST the the routing module does log and so we suspect something in the way InVEST is handling logging. Here's a short snippet that uses routing and taskgraph that shows log messages from the routing module:

import logging

import taskgraph
import pygeoprocessing.routing

logging.basicConfig(level=logging.DEBUG)


def main():
    task_graph = taskgraph.TaskGraph('taskgraph_cache', 8)

    target_filled_dem_raster_path = 'filled.tif'
    dem_raster_path_band = (r"C:\Users\rpsharp\Documents\bitbucket_repos\invest\data\invest-data\Base_Data\Freshwater\dem", 1)
    task_graph.add_task(
        func=pygeoprocessing.routing.fill_pits,
        args=(
            dem_raster_path_band, target_filled_dem_raster_path),
        target_path_list=[target_filled_dem_raster_path])

    task_graph.close()
    task_graph.join()


if __name__ == '__main__':
    main()

Originally reported by @richpsharp at https://bitbucket.org/natcap/invest/issues/3859/pygeoprocessingrouting-not-showing-up-in

Check for and handle pre-defined GDAL_DATA environment variable in binaries

We had a case in the forums (https://community.naturalcapitalproject.org/t/notimplementederror-and-missing-gdal-data-environment-variable/730/3) where the user was trying to run the InVEST binaries but already had a GDAL_DATA environment variable defined, which then prevented the application from finding the local binaries.

Someone is likely to have a GDAL_DATA environment variable defined if they have a valid GDAL install on their system, but if it isn't of the correct version, it'll undoubtedly cause issues. We should be able to handle this case elegantly within the InVEST binaries.

Create wheels as a part of our regular build process

It would be really nice to be able to create wheels as a part of our regular build processes. manylinux would be great, but that's a smaller concern than windows wheels. Windows would be the highest priority, mac second.

Update how we handle the JSON file that points to the latest builds

We had previously set up a script at scripts/update_installer_urls.py to keep track of the latest application binaries, with the intent of having that be our main entry point for downloading the latest release of InVEST. Since then, we decided that the new drupal site's scheduling functionality will be a better approach for updating the latest binaries on the natcap website.

To use this new functionality, we'll need to address a couple things:

  • We'll remove the old JSON file generation from the deployment process
  • @phargogh will talk with Irina to make sure that we understand what the target format needs to be and how the drupal site will use it
  • We'll update the script to use the new format
  • We'll use the new script in response to a new release (perhaps as part of a github action?)

Rec server automated cache cleanup

The rec model server accumulates files in a cache directory each time a user runs the model. Old workspaces need to be removed periodically because disk space is limited.

A solution is to have a shell script run on cron to find and remove workspaces. We'll track that script here in invest/scripts/recreation

Allow the sharing of a taskgraph cache across model runs in different workspaces

If I'm running a parameter study of a single model, let's say SDR, it would be really nice to be able to re-run the model in multiple workspaces (to keep the files separate), but only recompute the things that are needed. It seems reasonable to pass in the taskgraph cache dir as an optional argument, and, if it's provided, the cache location should be reused.

InVEST won't run on Windows 7

This was reported by a user in https://community.naturalcapitalproject.org/t/trouble-with-versions-upgrading-to-3-8-on-windows-7/864. I'm currently waiting to hear back from the user about the specific exception being raised. I'll post it here when I hear back.

Apparently InVEST won't run in Windows 7. Usually this sort of thing is a DLL issue. I'm not necessarily saying that we absolutely need to support running InVEST under an old, unsupported operating system, but if there is a workaround, it would be nice to document it somewhere.

According to the python download page, this shouldn't be an issue with python itself since only Windows XP and earlier are unsupported.

Use the args key in datastack archive rasters and vectors

When creating a datastack archive, InVEST creates a randomly-generated directory that will definitely not conflict in name with other directories in the same datastack. To do this, a randomish directory name is generated. A downside to this is that you have no idea what that raster is supposed to represent unless you're also looking at the parameters.json file.

Can't we at least include the args key names in the directory name? It would be very nice to be able to just look at the uncompressed datastack on disk and know which parameter I'm looking at.

Coastal Blue Carbon fails when LULC lookup table column 'lulc-class' is numeric

Reported at https://forums.naturalcapitalproject.org/index.php?p=/discussion/1409/attributeerror-float-object-has-no-attribute-lower-in-running-invest-blue-carbon-model. Data is attached to the forums post.

Full exception:

Traceback (most recent call last):
  File "c:\users\natcap-servers\jenkins-home\workspace\natcap.invest\label\gce-windows-1\release_env\lib\site-packages\natcap\invest\ui\execution.py", line 68, in run
  File "c:\users\natcap-servers\jenkins-home\workspace\natcap.invest\label\gce-windows-1\release_env\lib\site-packages\natcap\invest\ui\model.py", line 1505, in _logged_target
  File "c:\users\natcap-servers\jenkins-home\workspace\natcap.invest\label\gce-windows-1\release_env\lib\site-packages\natcap\invest\coastal_blue_carbon\coastal_blue_carbon.py", line 126, in execute
  File "c:\users\natcap-servers\jenkins-home\workspace\natcap.invest\label\gce-windows-1\release_env\lib\site-packages\natcap\invest\coastal_blue_carbon\coastal_blue_carbon.py", line 684, in get_inputs
  File "c:\users\natcap-servers\jenkins-home\workspace\natcap.invest\label\gce-windows-1\release_env\lib\site-packages\natcap\invest\coastal_blue_carbon\coastal_blue_carbon.py", line 684, in <genexpr>
AttributeError: 'float' object has no attribute 'lower'
06/19/2018 10:22:10  natcap.invest.ui.execution INFO     Execution finished

This might have been addressed by the pandas refactor, but it'll be worth checking to make sure this is indeed resolved.

Originally reported at https://bitbucket.org/natcap/invest/issues/3749/coastal-blue-carbon-fails-when-lulc-lookup.

UI tests on Multi inputs crash

This has happened fairly consistently for a while, but is not reproducible outside of the appveyor VM. Since only one InVEST model (coastal blue carbon) uses a Multi input, we’ll skip this class of tests for now, and consider refactoring the CBC UI to avoid this troublesome input.

For now we're skipping the test. Eventually we might factor out the instances of the Multi objects altogether.

Originally reported by @davemfish at https://bitbucket.org/natcap/invest/issues/3936/ui-tests-on-multi-inputs-crash-on-appveyor

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.