Comments (6)
Hi Sean,
I've tinkered a lot with Github Actions on Ubuntu VMs, e.g. for PyShp, and earlier this week I put together a finished a compilation and testing pipeline on Windows runners for sDNA. Adding a Windows testing workflow is straightforward based on what you already have. The Windows runners even have Bash. I just have a couple of suggestions, which I hope are of use.
- I notice rasterio is tested in PROJ containerised jobs. Docker job containers currently require Ubuntu runners (or Linux self hosted ones) https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idcontainer So unfortunately a Windows workflow won't neatly fit into a matrix with the existing containerised tests. But as an open source project, you can run as many GHA runners as you want (up to 255?).
It's simpler to run, and easier to fix, tests in the GHA Windows VM directly. But if you really want to, you can still run windows containers on a windows runner, via docker directly. Just without the GHA workflow syntax automagic. If so:
i) You may already know a Windows container can only run on a Windows host. But moreover: a servercore-2022 container has to run on a windows-2022 VM GHA runner (not the 2019 one).
ii) Avoid nanoserver base images. Maybe someone else has the deep magic to do what I was attempting (to run an SSH server and install powershell). But without that knowledge, after persevering for a day or two, I found nanoserver containers are way way more trouble than they're worth. More like starting FROM scratch
than FROM alpine
.
- Installing PROJ on Windows. PROJ only provides source downloads. I can't find PROJ on chocolatey, nor can I find an official osgeo windows container on docker hub https://hub.docker.com/r/osgeo/proj . So I would just try a silent Windows installation of PROJ, downloading and using osgeo4w:
C:\temp\osgeo4w-setup.exe -q -k -r -A -s https://download.osgeo.org/osgeo4w/v2/ -P proj
https://proj.org/en/9.3/install.html
from rasterio.
I should've scrolled a little further down in tests.yml, and discovered tests are also run on conda. I'm pretty sure the conda tests can be done on Windows, and will fit as neatly into the amtrix as the MacOS ones do.
I've got the basics running, but could not create a test env successfully yet.
Why is the --no-use-pep517
flag needed?
Obtaining file:///D:/a/rasterio/rasterio
ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml
Obtaining file:///D:/a/rasterio/rasterio
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Checking if build backend supports build_editable: started
Checking if build backend supports build_editable: finished with status 'done'
Getting requirements to build editable: started
Getting requirements to build editable: finished with status 'error'
error: subprocess-exited-with-error
Getting requirements to build editable did not run successfully.
exit code: 1
[3 lines of output]
<string>:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
INFO:root:Building on Windows requires extra options to setup.py to locate needed GDAL files. More information is available in the README.
ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
Getting requirements to build editable did not run successfully.
exit code: 1
See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
https://github.com/JamesParrott/rasterio/actions/runs/8097345406/job/22128212640
from rasterio.
It doesn't make any difference without --no-use-pep517
:
INFO:root:Building on Windows requires extra options to setup.py to locate needed GDAL files. More information is available in the README.
ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
Getting requirements to build editable did not run successfully.
exit code: 1
See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.
https://github.com/JamesParrott/rasterio/actions/runs/8097532546/job/22128803209
I'm a bit tied up with other things until next week, but if someone wants to take this ball and run with it, be my guest.
from rasterio.
Sean, have I misunderstood something else too? It looks to me like the windows wheels are, or at least were, being tested in the sister repo rasterio-wheels, but are resulting in the following errors (that could be due to a faulty test fixture not creating a required pre-existing file):
FAILED tests/test_rio_create.py::test_create_no_overwrite_nonfile - assert "Object exists and won't be overwritten" in ''
+ where '' = <Result RasterioIOError("Attempt to create new tiff file 'D:/vsimem/test_create_no_overwrite_nonfile-ecfa656d-4fba-4167-bc79-9871b930fffc/test_create_no_overwrite_nonfile-ecfa656d-4fba-4167-bc79-9871b930fffc.tif' failed: No such file or directory")>.output
FAILED tests/test_rio_create.py::test_create_overwrite_nonfile - assert 1 == 0
+ where 1 = <Result RasterioIOError("Attempt to create new tiff file 'D:/vsimem/test_create_overwrite_nonfile-3dd21a02-8158-4c2c-a5bc-2a45ec7fe31c/test_create_overwrite_nonfile-3dd21a02-8158-4c2c-a5bc-2a45ec7fe31c.tif' failed: No such file or directory")>.exit_code
FAILED tests/test_rio_create.py::test_create_no_overwrite_nonfile_2 - assert "Dataset exists and won't be overwritten" in ''
+ where '' = <Result RasterioIOError("Attempt to create new tiff file 'D:/vsimem/test_create_no_overwrite_nonfile_2-01ea07ea-c28b-4ed9-a89f-7ff3f1cb8ac5/test_create_no_overwrite_nonfile_2-01ea07ea-c28b-4ed9-a89f-7ff3f1cb8ac5.tif' failed: No such file or directory")>.output
FAILED tests/test_rio_create.py::test_create_overwrite_nonfile_2 - assert 1 == 0
+ where 1 = <Result RasterioIOError("Attempt to create new tiff file 'D:/vsimem/test_create_overwrite_nonfile_2-7c819911-228e-4181-8f8b-62c6c27c042e/test_create_overwrite_nonfile_2-7c819911-228e-4181-8f8b-62c6c27c042e.tif' failed: No such file or directory")>.exit_code
= 4 failed, 2077 passed, 23 skipped, 35 deselected, 11 xfailed, 6 xpassed, 295 warnings in 68.69s (0:01:08) =
https://github.com/rasterio/rasterio-wheels/actions/runs/7909110963/job/21633622377#step:7:2894
from rasterio.
Rasterio 1.3.9 was tested on windows, and passed.
https://github.com/rasterio/rasterio-wheels/actions/runs/6568583403
This is outside the scope of an issue concerned solely with Testing on Windows, but from a little digging:
The 4 failing tests above (test_create_no_overwrite_nonfile, test_create_overwrite_nonfile test_create_no_overwrite_nonfile_2 test_create_overwrite_nonfile_2) all use both MemoryFile and runner (a CliRunnerfrom click.testing). MemoryFile is quite thoroughly tested in rasterio\tests\test_memoryfile.py, but neither runner nor anything from click is not used there, and none of those tests cover interation with an external process, that I can see.
click's CLiRunner creates an isolated environment for the test. This isolation might not play nicely with with both GDAL's virtual file system, and Windows. Or it could be as simple as a required directory not being created. Just a thought.
https://click.palletsprojects.com/en/8.1.x/api/#click.testing.CliRunner
from rasterio.
Related: #2351
from rasterio.
Related Issues (20)
- rasterio WarpedVRT vs QGIS Warp tool
- rasterize can no longer handle dtype instances HOT 2
- Change python opener VSI plugin prefix to /vsiriopener HOT 1
- Small typo: Release 1.4a2 requires Python 3.8 instead of 3.9 HOT 1
- Test performance: Dependencies installed twice HOT 1
- `reproject` with `src_geoloc_array` introduces unexpected shift for group of pixels HOT 2
- 1.3.10 release HOT 3
- "pytest.PytestRemovedIn8Warning: Passing None has been deprecated" test failures HOT 1
- Run pyupgrade on code
- Writing a COG to a file-like object in "wb" mode fails in 1.4a2 HOT 4
- Disallow rotated rasters in merge() HOT 3
- Add correct python syntax highlighting to README HOT 2
- update_tags does not work for special tags like PIXEL_OR_AREA HOT 2
- Boundless mask reads are incorrect for complex data and GDAL 3.8
- rasterio.merge.merge() uses ~10 times more memory than specified by `mem_limit` HOT 1
- Unexpected sieve results when mask is entirely false
- Opener registration error for files in same parent directory HOT 9
- access Sentinel-1 IW SLC tiff measurement via S3 on creodias entrypoint
- memory doesn't free using features.geometry_mask function with fastapi HOT 2
- Typo in WarpedVRT's docstring
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rasterio.