Giter Club home page Giter Club logo

Comments (6)

JamesParrott avatar JamesParrott commented on May 23, 2024 2

Hi Sean,
I've tinkered a lot with Github Actions on Ubuntu VMs, e.g. for PyShp, and earlier this week I put together a finished a compilation and testing pipeline on Windows runners for sDNA. Adding a Windows testing workflow is straightforward based on what you already have. The Windows runners even have Bash. I just have a couple of suggestions, which I hope are of use.

  1. I notice rasterio is tested in PROJ containerised jobs. Docker job containers currently require Ubuntu runners (or Linux self hosted ones) https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idcontainer So unfortunately a Windows workflow won't neatly fit into a matrix with the existing containerised tests. But as an open source project, you can run as many GHA runners as you want (up to 255?).

It's simpler to run, and easier to fix, tests in the GHA Windows VM directly. But if you really want to, you can still run windows containers on a windows runner, via docker directly. Just without the GHA workflow syntax automagic. If so:

i) You may already know a Windows container can only run on a Windows host. But moreover: a servercore-2022 container has to run on a windows-2022 VM GHA runner (not the 2019 one).

ii) Avoid nanoserver base images. Maybe someone else has the deep magic to do what I was attempting (to run an SSH server and install powershell). But without that knowledge, after persevering for a day or two, I found nanoserver containers are way way more trouble than they're worth. More like starting FROM scratch than FROM alpine.

  1. Installing PROJ on Windows. PROJ only provides source downloads. I can't find PROJ on chocolatey, nor can I find an official osgeo windows container on docker hub https://hub.docker.com/r/osgeo/proj . So I would just try a silent Windows installation of PROJ, downloading and using osgeo4w:
C:\temp\osgeo4w-setup.exe -q -k -r -A -s https://download.osgeo.org/osgeo4w/v2/ -P proj

https://proj.org/en/9.3/install.html

from rasterio.

JamesParrott avatar JamesParrott commented on May 23, 2024

I should've scrolled a little further down in tests.yml, and discovered tests are also run on conda. I'm pretty sure the conda tests can be done on Windows, and will fit as neatly into the amtrix as the MacOS ones do.

I've got the basics running, but could not create a test env successfully yet.

Why is the --no-use-pep517 flag needed?


Obtaining file:///D:/a/rasterio/rasterio
ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml
Obtaining file:///D:/a/rasterio/rasterio
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'error'
  error: subprocess-exited-with-error
  
  Getting requirements to build editable did not run successfully.
  exit code: 1
  
  [3 lines of output]
  <string>:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  INFO:root:Building on Windows requires extra options to setup.py to locate needed GDAL files. More information is available in the README.
  ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
  [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Getting requirements to build editable did not run successfully.
exit code: 1

See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

https://github.com/JamesParrott/rasterio/actions/runs/8097345406/job/22128212640

from rasterio.

JamesParrott avatar JamesParrott commented on May 23, 2024

It doesn't make any difference without --no-use-pep517:

   INFO:root:Building on Windows requires extra options to setup.py to locate needed GDAL files. More information is available in the README.
  ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
  [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
Getting requirements to build editable did not run successfully.
exit code: 1
See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.

https://github.com/JamesParrott/rasterio/actions/runs/8097532546/job/22128803209

I'm a bit tied up with other things until next week, but if someone wants to take this ball and run with it, be my guest.

from rasterio.

JamesParrott avatar JamesParrott commented on May 23, 2024

Sean, have I misunderstood something else too? It looks to me like the windows wheels are, or at least were, being tested in the sister repo rasterio-wheels, but are resulting in the following errors (that could be due to a faulty test fixture not creating a required pre-existing file):

FAILED tests/test_rio_create.py::test_create_no_overwrite_nonfile - assert "Object exists and won't be overwritten" in ''
 +  where '' = <Result RasterioIOError("Attempt to create new tiff file 'D:/vsimem/test_create_no_overwrite_nonfile-ecfa656d-4fba-4167-bc79-9871b930fffc/test_create_no_overwrite_nonfile-ecfa656d-4fba-4167-bc79-9871b930fffc.tif' failed: No such file or directory")>.output
FAILED tests/test_rio_create.py::test_create_overwrite_nonfile - assert 1 == 0
 +  where 1 = <Result RasterioIOError("Attempt to create new tiff file 'D:/vsimem/test_create_overwrite_nonfile-3dd21a02-8158-4c2c-a5bc-2a45ec7fe31c/test_create_overwrite_nonfile-3dd21a02-8158-4c2c-a5bc-2a45ec7fe31c.tif' failed: No such file or directory")>.exit_code
FAILED tests/test_rio_create.py::test_create_no_overwrite_nonfile_2 - assert "Dataset exists and won't be overwritten" in ''
 +  where '' = <Result RasterioIOError("Attempt to create new tiff file 'D:/vsimem/test_create_no_overwrite_nonfile_2-01ea07ea-c28b-4ed9-a89f-7ff3f1cb8ac5/test_create_no_overwrite_nonfile_2-01ea07ea-c28b-4ed9-a89f-7ff3f1cb8ac5.tif' failed: No such file or directory")>.output
FAILED tests/test_rio_create.py::test_create_overwrite_nonfile_2 - assert 1 == 0
 +  where 1 = <Result RasterioIOError("Attempt to create new tiff file 'D:/vsimem/test_create_overwrite_nonfile_2-7c819911-228e-4181-8f8b-62c6c27c042e/test_create_overwrite_nonfile_2-7c819911-228e-4181-8f8b-62c6c27c042e.tif' failed: No such file or directory")>.exit_code
= 4 failed, 2077 passed, 23 skipped, 35 deselected, 11 xfailed, 6 xpassed, 295 warnings in 68.69s (0:01:08) =

https://github.com/rasterio/rasterio-wheels/actions/runs/7909110963/job/21633622377#step:7:2894

from rasterio.

JamesParrott avatar JamesParrott commented on May 23, 2024

Rasterio 1.3.9 was tested on windows, and passed.
https://github.com/rasterio/rasterio-wheels/actions/runs/6568583403

This is outside the scope of an issue concerned solely with Testing on Windows, but from a little digging:

The 4 failing tests above (test_create_no_overwrite_nonfile, test_create_overwrite_nonfile test_create_no_overwrite_nonfile_2 test_create_overwrite_nonfile_2) all use both MemoryFile and runner (a CliRunnerfrom click.testing). MemoryFile is quite thoroughly tested in rasterio\tests\test_memoryfile.py, but neither runner nor anything from click is not used there, and none of those tests cover interation with an external process, that I can see.

click's CLiRunner creates an isolated environment for the test. This isolation might not play nicely with with both GDAL's virtual file system, and Windows. Or it could be as simple as a required directory not being created. Just a thought.
https://click.palletsprojects.com/en/8.1.x/api/#click.testing.CliRunner

from rasterio.

snowman2 avatar snowman2 commented on May 23, 2024

Related: #2351

from rasterio.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.