rasterio / rasterio-wheels Goto Github PK

License: MIT License

Shell 26.77% Python 39.42% CMake 33.81%

rasterio-wheels's Introduction

rasterio-wheels

This project builds the rasterio binary distributions that are uploaded to PyPI. Those distributions, or "wheels", include a GDAL shared library and other shared libraries supporting many, but not all, of GDAL's format drivers. If you need the rarely used formats and compressors not found in these wheels, you may find them in the conda-forge conda channel, or in Docker images published by the GDAL project.

Wheels for manylinux2014_x86_64, macos_10_15_x86_64, and win_amd64 are built by GitHub Actions. Wheels for macosx_11_0_arm64 are built by Cirrus CI.

Other platforms are out of scope at this time.

rasterio-wheels's People

Contributors

Stargazers

Watchers

Forkers

drnextgis vincentsarago jswhit scrarlet manishgit138 ashutoshstu surehust robert-werner kapadia premprabhat plantain 20treeai idanmiara andrewannex firefishy ewouth

rasterio-wheels's Issues

Support for pre-built aarch64/arm64 wheels

With mac folks transitioning to aarch64 as a viable platform, it would be great to provide pre-built aarch64 wheels for rasterio, too.

I'm opening this ticket so we can capture thoughts, blockers, and a plan moving forward.

I think long term having arm64 wheels would be great, but also know that right now it's very early for arm64.

Thoughts? 🤗

Wheels lack the proj-datumgrid-1.8 package

Was brought to my attention today by a user which used the rasterio-wheels that he got incorrect results when reprojecting a dataset from NAD27 to WGS84, when comparing to the result they got with a non-wheel installation or plain gdalwarp. As far as I can see in https://github.com/rasterio/rasterio-wheels/blob/master/config.sh, only PROJ is built, but the https://download.osgeo.org/proj/proj-datumgrid-1.8.zip support data, that contains the "conus" grid for NAD27->NAD83/WGS84, is not unzipped in /usr/share/proj.

Add nghttp2 to enable http/2 in CURL

👋 Hi @sgillies,
Adding HTTP/2 to rasterio wheels could be really useful. I'll open a PR later and we can discuss if this is a good idea :-)

Add HDF4 support in wheels?

Using the 1.1.2 wheel:

>>> import rasterio
>>> rds = rasterio.open("MOD14.A2019354.1155.006.2019354142314.hdf")
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__
  File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: 'MOD14.A2019354.1155.006.2019354142314.hdf' not recognized as a supported file format.

Using conda-forge:

>>> import rasterio
>>> rds = rasterio.open("MOD14.A2019354.1155.006.2019354142314.hdf")
/home/snowal/miniconda/envs/rioxarray/lib/python3.7/site-packages/rasterio/__init__.py:219: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
>>> rds.subdatasets
['HDF4_SDS:UNKNOWN:MOD14.A2019354.1155.006.2019354142314.hdf:0', 'HDF4_SDS:UNKNOWN:MOD14.A2019354.1155.006.2019354142314.hdf:1', 'HDF4_SDS:UNKNOWN:MOD14.A2019354.1155.006.2019354142314.hdf:29']

This is useful for loading in MODIS data.

Out of Travis credits

The rasterio org's free plan is out of credits. I'll ask Travis support for OSS only credits. Meanwhile, I've created a private copy of this repo in my employer's org and will build on that plan (paid by Mapbox).

Downgrade curl to 7.59

I used rio-glui for the first time today on Linux, with wheel built by this project, and experienced eventual segfaults when browsing a dataset.

Thread 11 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd51f9700 (LWP 13508)]
0x00007fffeb98b0cd in lh_retrieve () from /home/sean/.local/share/virtualenvs/vexcel-T0Omk86X/lib/python3.6/site-packages/rasterio/.libs/./libcurl-f05553a1.so.4.5.0

I see that downgrading to 7.59 has resolved a potentially-related issue and will give it a try.

mojodna/docker-gdal@aa1dd14
OSGeo/gdal#738 (comment)

Update HDF version

rasterio-wheels/env_vars.sh

Line 15 in 8b18758

HDF5_VERSION=1.10.8

ref: #81

Prevent shared library name conflict with fiona deps

Background: rasterio/rasterio#1876 (comment).

Can we patch auditwheel so that the names of rasterio's shared library dependencies are different from those of fiona's. The hash tag seems to be insufficient.

Patch GDAL 2.3.2

https://github.com/OSGeo/gdal/commit/340ad0d703534a256ec3de94176c95b0cf20cbd4.diff allows AWS errors to fully surface in GDAL's error stack.

Include GDAL 2.4.1 with patch linked in rasterio #1663

Otherwise the HDF5 bug won't be out until 2.4.2.

gdal 2.3.3 wheels?

I see the recent commits might indicate 1.0.14 will be using 2.3.3, if so you can close this issue.

HDF5 cross-compilation issues block macosx arm64 wheels

HDF5's configure needs to run some system tests, but when cross compiling the test programs can't be run on the build host. It looks like the conda-forge project has a solution in https://github.com/conda-forge/hdf5-feedstock/pull/145/files. We need much of this, but not all (no fortran or MPI needed), if we want to make macosx arm64 wheels with HDF5/netcdf drivers on an ordinary GitHub Actions x86_64 runner.

Kicking HDF5/netcdf out of these wheel profiles is another option.

Update netcdf to 4.6

Otherwise netcdf via vsicurl is broken. See OSGeo/gdal#1328.

GDAL 3.1 and PROJ 7.1

It's not that I haven't wanted to, it's that changes in PROJ 6 and 7 have impacted packaging and environment configuration. But it's time to do this.

PROJ doesn't need data files any more; it can fetch data from the PROJ CDN. I think this will be the default for new wheels.

Importing rasterio causes a HDF error with netCDF4

rasterio/rasterio#1574 by @ererkka copied to this repo.

Expected behavior and actual behavior.

Expected that importing rasterio has no effect on other libraries.
Creating a netCDF file using netCDF4 library fails when rasterio is imported first.

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    print(ds)
  File "netCDF4/_netCDF4.pyx", line 2168, in netCDF4._netCDF4.Dataset.__exit__
  File "netCDF4/_netCDF4.pyx", line 2276, in netCDF4._netCDF4.Dataset.close
  File "netCDF4/_netCDF4.pyx", line 2260, in netCDF4._netCDF4.Dataset._close
  File "netCDF4/_netCDF4.pyx", line 1754, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

Steps to reproduce the problem.

$ mktmpenv
$ pip install rasterio netCDF4
$ python test.py

test.py

import rasterio
import netCDF4

with netCDF4.Dataset('test.nc', mode='w') as ds:
    ds.createDimension('x')
    ds.createVariable('foo', float, dimensions=('x'))
    print(ds)

Commenting out import rasterio or importing it after netCDF4 removes the error.

Operating system

CentOS release 6.10 (Final)

Rasterio version

Python 3.5.3, pip 18.1

From pip freeze

affine==2.2.1
attrs==18.2.0
cftime==1.0.3.4
Click==7.0
click-plugins==1.0.4
cligj==0.5.0
netCDF4==1.4.2
numpy==1.15.4
pyparsing==2.3.0
rasterio==1.0.11
snuggs==1.4.2

Links to other issues

This might be linked to pydata/xarray#2535.

GitHub Actions for some wheel builds

I'm going to try following the example of python-pillow/pillow-wheels, which runs multibuild on GHA https://github.com/python-pillow/pillow-wheels/actions/runs/1655656052. @snowman2 I think I recall you suggesting this?

Pillow wheels for aarch64 only are built on Travis: https://app.travis-ci.com/github/python-pillow/pillow-wheels.

Build GDAL binaries against libcrypto for MacOS

The GDAL binary that ships with rasterio doesn't appear to be built against libcrypto or a reasonable alternative. When installing via wheels rasterio is unable to access rasters using GDAL's network filesystems (e.g. vsigs). One workaround is to build GDAL manually (or install from Homebrew) so that libcrypto is linked, then run pip install rasterio --no-binary rasterio to avoid using the prebuilt binaries.

If possible, the GDAL binary should be built against libcrypto or other appropriate library, and the wheel should be updated.

Here's an example of the limitation:

% rio info gs://....
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 261, in rasterio._base.DatasetBase.__init__
  File "rasterio/_shim.pyx", line 78, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 216, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_NotSupportedError: CPLRSASHA256Sign() not implemented: GDAL must be built against libcrypto++ or libcrypto (openssl)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/amit/.pyenv/versions/example/bin/rio", line 8, in <module>
    sys.exit(main_group())
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/rasterio/rio/info.py", line 66, in info
    with ctx.obj['env'], rasterio.open(input) as src:
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/rasterio/env.py", line 437, in wrapper
    return f(*args, **kwds)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/rasterio/__init__.py", line 220, in open
    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
  File "rasterio/_base.pyx", line 263, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: CPLRSASHA256Sign() not implemented: GDAL must be built against libcrypto++ or libcrypto (openssl)

I just spoke with @sgillies, and volunteered to help with this issue.

Overhaul GDAL dependency builds on OS X

Full GDAL builds, from scratch, on OS X time out. Up to now we've worked around this by downloading a pre-compiled bundle of GEOS, HDF5, and netCDF4. These aren't compatible with the libs from SciPy. Another approach to try is to build these using multibuild and cache the install directories (aka BUILD_PREFIX), like we're doing in rasterio's CI. Target: end of summer 2019.

Build wheels with namespace package and optional dependencies that include fiona

Duplicate link to sgillies/fiona-wheels#22

Ensure openssl's libcrypto is installed and configure GDAL --with-crypto

Necessary to support Google Cloud Storage.

Add ZSTD compression

Requires libzstd and probably GDAL configuration flags.

use external libtiff

This is not an issue so please feel free to close if you don't have the time to answer ;-)

So with the new rasterio wheels you updated proj to 7.2 which require libtiff but when looking at the build config I can't find where libtiff gets installed 🤷

rasterio-wheels/config.sh

Lines 17 to 33 in df979bc

 function build_proj { 

 CFLAGS="$CFLAGS -g -O2" 

 CXXFLAGS="$CXXFLAGS -g -O2" 

 if [ -e proj-stamp ]; then return; fi 

 fetch_unpack http://download.osgeo.org/proj/proj-${PROJ_VERSION}.tar.gz 

 (cd proj-${PROJ_VERSION} \ 

 && ./configure --prefix=$BUILD_PREFIX \ 

 && make -j4 \ 

 && make install) 

 if [ -n "$IS_OSX" ]; then 

 : 

 else 

 strip -v --strip-unneeded ${BUILD_PREFIX}/lib/libproj.so.* 

 fi 

 touch proj-stamp 

 }

Multiple copies of hdf and netcdf libraries in linux wheels

Rasterio 1.0.24, but also a problem in 1.0.22. What the what?

$ ls -l ~/.virtualenvs/sat/lib/python3.6/site-packages/rasterio/.libs/
total 150420
-rwxr-xr-x 1 sean sean     37712 Jun 19 17:55 libaec-2147abcd.so.0.0.4
-rwxr-xr-x 1 sean sean   3501408 Jun 19 17:55 libcurl-835381e7.so.4.4.0
-rwxr-xr-x 1 sean sean   3144312 Jan 28 10:30 libcurl-e8503455.so.4.5.0
-rwxr-xr-x 1 sean sean    224232 Jun 19 17:55 libexpat-09c47d4c.so.1.6.8
-rwxr-xr-x 1 sean sean 101652240 Jun 19 17:55 libgdal-a3e05b74.so.20.5.1
-rwxr-xr-x 1 sean sean   4883424 Jun 19 17:55 libgeos-3-cd838e67.6.2.so
-rwxr-xr-x 1 sean sean    431816 Jun 19 17:55 libgeos_c-595de9d4.so.1.10.2
-rwxr-xr-x 1 sean sean   4116192 Jan 28 10:30 libhdf5-2d94a66d.so.101.1.0
-rwxr-xr-x 1 sean sean   4116192 Feb  4 16:22 libhdf5-51ecf825.so.101.1.0
-rwxr-xr-x 1 sean sean   4116192 Jan 28 14:00 libhdf5-595bcc90.so.101.1.0
-rwxr-xr-x 1 sean sean   4116192 Jun 19 17:55 libhdf5-6068dce9.so.101.1.0
-rwxr-xr-x 1 sean sean   4116192 Feb  1 13:24 libhdf5-d381781f.so.101.1.0
-rwxr-xr-x 1 sean sean   4116192 Jan 24 20:03 libhdf5-dd13d074.so.101.1.0
-rwxr-xr-x 1 sean sean    182032 Jun 19 17:55 libhdf5_hl-d68fbc5b.so.100.1.0
-rwxr-xr-x 1 sean sean    344704 Jun 19 17:55 libjpeg-3fe7dfc0.so.9.3.0
-rwxr-xr-x 1 sean sean     60792 Jun 19 17:55 libjson-c-5f02f62c.so.2.0.2
-rwxr-xr-x 1 sean sean   1876336 Jan 28 10:30 libnetcdf-24686c8b.so.11.0.4
-rwxr-xr-x 1 sean sean   1876336 Feb  4 16:22 libnetcdf-3d72bc80.so.11.0.4
-rwxr-xr-x 1 sean sean   1876336 Jan 28 14:00 libnetcdf-533c8e68.so.11.0.4
-rwxr-xr-x 1 sean sean   1843520 Jan 24 20:03 libnetcdf-7337909d.so.11.0.4
-rwxr-xr-x 1 sean sean   1876336 Jun 19 17:55 libnetcdf-b4b8510b.so.11.0.4
-rwxr-xr-x 1 sean sean   1876336 Feb  1 13:24 libnetcdf-c7b3ee53.so.11.0.4
-rwxr-xr-x 1 sean sean    203512 Jun 19 17:55 libnghttp2-11cb20b8.so.14.17.1
-rwxr-xr-x 1 sean sean    376744 Jun 19 17:55 libopenjp2-8f6da918.so.2.3.0
-rwxr-xr-x 1 sean sean    279824 Jun 19 17:55 libpng16-898afbbd.so.16.35.0
-rwxr-xr-x 1 sean sean    580440 Jun 19 17:55 libproj-bd876d1a.so.12.0.0
-rwxr-xr-x 1 sean sean   1267504 Jun 19 17:55 libsqlite3-fdd57a2d.so.0.8.6
-rwxr-xr-x 1 sean sean     13240 Jun 19 17:55 libsz-1c7dd0cf.so.2.0.1
-rwxr-xr-x 1 sean sean    776912 Jun 19 17:55 libwebp-8ccd29fd.so.7.0.2
-rwxr-xr-x 1 sean sean     87848 Jun 19 17:55 libz-a147dcb0.so.1.2.3

Build GDAL without GEOS support on OS X

See sgillies/fiona-wheels#18

Delocate error

delocate.delocating.DelocationError: Already planning to copy library with same basename as: libwebp.7.dylib

https://github.com/rasterio/rasterio-wheels/runs/6531691333?check_suite_focus=true#step:4:4108

I'm puzzled, because I've changed almost nothing since I built the 1.3b1 wheels.

switch to libwebp 1.0.1

Ran a quick benchmark over https://github.com/vincentsarago/benchmark-img-gdal and it seems that libwebp 1.0.1 is a bit faster than the one use un rasterio wheels (1.0.0)

Happy to submit a PR 😄

Build libjpeg with libjpeg turbo

as for #20, without libjpegturbo (shipped within wheels) libjpeg seems to be slower (in comparison to the one built with libjpegturbo). Might be a nice thing to consider adding.

That's said I need to mention that user are raising question about rasterio wheels being to heavy ref: mojodna/marblecutter-virtual#8, adding another lib might make some people unhappy!

Thoughts on using a newer version of libtiff in the wheels?

rasterio-wheels/config.sh

Line 186 in e034219

--with-geotiff=internal \

I am definitely ignorant on the background for the version choice of libtiff, so feel free to fill me in there. I had issues assisting someone install rasterio on an OSX machine and everything worked except that they were not being able to read in tiff files written with LZMA compression. I attempted a no-binary install on OSX using the brew version of GDAL without success. I eventually got it working with conda.. But, I am wondering if upgrading libtiff would resolve the issue (or there could be some other dependency I missed that is causing the issue)? I will try to do some more testing later, but I don't have an OSX machine, so hard to tell 🤷‍♂️.

Python 3.4 failure on linux

See https://travis-ci.org/sgillies/rasterio-wheels/builds/430619270. Failures are of several kinds:

_____________________________ test_dst_crs_error_2 _____________________________

runner = <click.testing.CliRunner object at 0x7f301efc6a58>
tmpdir = local('/tmp/pytest-of-root/pytest-0/test_dst_crs_error_20')

    def test_dst_crs_error_2(runner, tmpdir):
        """Invalid PROJ.4 is a bad parameter."""
        srcname = 'tests/data/RGB.byte.tif'
        outputname = str(tmpdir.join('test.tif'))
        result = runner.invoke(main_group, [
            'warp', srcname, outputname, '--dst-crs', '{"proj": "foobar"}'])
>       assert result.exit_code == 2
E       AssertionError: assert -1 == 2
E        +  where -1 = <Result ValueError('Either four bounding values or ground control pointsmust be specified',)>.exit_code

tests/test_rio_warp.py:38: AssertionError

and

____________________ test_warp_reproject_check_invert_false ____________________

>   ???

rasterio/_base.pyx:199: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???

rasterio/_shim.pyx:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   rasterio._err.CPLE_OpenFailedError: /tmp/pytest-of-root/pytest-0/test_warp_reproject_check_inve1/test.tif: No such file or directory

rasterio/_err.pyx:188: CPLE_OpenFailedError

During handling of the above exception, another exception occurred:

runner = <click.testing.CliRunner object at 0x7f301fa046d8>
tmpdir = local('/tmp/pytest-of-root/pytest-0/test_warp_reproject_check_inve1')

    def test_warp_reproject_check_invert_false(runner, tmpdir):
        outputname = str(tmpdir.join('test.tif'))
        output2name = str(tmpdir.join('test2.tif'))
        srcname = 'tests/data/world.rgb.tif'
    
        # default True
        runner.invoke(main_group, [
            'warp', srcname, outputname, '--dst-crs', 'EPSG:3759'])
    
        # explicit False
        runner.invoke(main_group, [
            'warp', srcname, output2name, '--no-check-invert-proj',
            '--dst-crs', 'EPSG:3759'])
    
>       with rasterio.open(outputname) as output, rasterio.open(output2name) as output2:

tests/test_rio_warp.py:559: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/venv/lib/python3.4/site-packages/rasterio/env.py:402: in wrapper
    return f(*args, **kwds)
/venv/lib/python3.4/site-packages/rasterio/__init__.py:217: in open
    s = DatasetReader(path, driver=driver, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   rasterio.errors.RasterioIOError: /tmp/pytest-of-root/pytest-0/test_warp_reproject_check_inve1/test.tif: No such file or directory

rasterio/_base.pyx:201: RasterioIOError

Python 3.4.3 is being used to test the wheel. I wouldn't be surprised if the problem is one of the bugs fixed in 3.4.4, a huge list: https://docs.python.org/3.4/whatsnew/changelog.html#python-3-4-4-release-candidate-1.

The solution is go upstream and upgrade Python 3.4 in matthewbrett/trusty.

curl 7.54.0 not building as before

When we built wheels for 1.0.13, they included our custom built curl 7.54.0, but in the 1.0.14 wheel builds this curl was missed and the container's curl 7.61 was picked up instead. Problematic in 2 ways: there are bugs between GDAL and recent curl versions, and our nghttp2 library is not included.

It's difficult to figure out what's wrong with the curl build due to lack of logging, but I'm digging in a little this evening. I may not be able to sort this out until later in the week @vincentsarago .

Patch GDAL to fix sub-pixel bug

See OSGeo/gdal#2510.

Vulnerable shared libraries might make rasterio vulnerable. Can you help upgrade to patch versions?

Hi, @sgillies , @perrygeo , I'd like to report a vulnerability issue in rasterio_1.2.10.

Dependency Graph between Python and Shared Libraries

Issue Description

As shown in the above dependency graph (Here shows part of the dependency graph, which depends on vulnerable shared libraries), rasterio_1.2.10 directly or transitively depends on 32 C libraries (.so). However, I noticed that some C libraries are vulnerable, containing the following CVEs:
libhdf5-rasterio-5ce9c7f2.so.103.1.0 and libhdf5_hl-rasterio-92c1cdd8.so.100.1.2 from C project hdf5(version:<=1.10.6) exposed 4 vulnerabilities:
CVE-2020-10811, CVE-2020-10812, CVE-2020-10810, CVE-2020-10809
libjson-c-rasterio-5f02f62c.so.2.0.2 from C project json-c(version:<=0.12.1) exposed 1 vulnerabilities:
CVE-2020-12762

Furthermore, the vulnerable methods in the vulnerable shared libraries can be actually invoked by Python code. For instance, the following call chain can reach the vulnerable method(C code) H5AC_unpin_entry() in file H5AC.c reported by CVE-2020-10810.

call chain-----
H5Literate_by_name()->H5G_iterate()->H5G__obj_iterate()->H5G__dense_iterate()->H5HF_open()->H5HF_close()->H5HF_space_close()->H5FS_close()->H5FS_decr()->H5AC_unpin_entry()

Suggested Vulnerability Patch Versions

hdf5 has fixed the vulnerabilities in versions >=1.12.1
json-c has fixed the vulnerabilities in versions >=0.15

Python build tools cannot report vulnerable C libraries, which may induce potential security issues to many downstream Python projects.
As a popular python package (rasterio has 814,019 downloads per month), could you please upgrade the above shared libraries to their patch versions?

Thanks for your help~
Best regards,
Joe Gardner

Rasterio tests failing out of the blue

A build that passed a few days ago fails on a re-run for string quoting issues: https://travis-ci.com/github/rasterio/rasterio-wheels/jobs/294150501#L7865.

Use pre-built GEOS, HDF5, NetCDF4 bundles

Travis builds of public projects are capped at 50 minutes. This is not enough time to build GDAL and a rich set of dependencies from source. GEOS, HDF5, and NetCDF4 are the deps that we need to update less often. I'm going to try building these on my MBP, zipping them up, and extracting the zipped files onto Travis build instances to save ~10 mins.

GDAL 2.4.0 fails to build

make -j4 obscures the error, so I haven't found it yet. Our image is based on CentOS 5 and we've seen issues before.

Python 2.7 builds are failing

Super puzzling: https://travis-ci.com/github/rasterio/rasterio-wheels/builds/153259734.

add libdeflate for faster TIFF decompression

because we are using libtiff internal with gdal3.2 I think it will be good to add libdeflate: OSGeo/gdal#3068

Add support for LERC compression

My knowledge about the internals of GDAL and rasterio is probably barely scratching the surface, so excuse my ignorance, but is it possible to include support for LERC compression in the near future?
Coming from here with a bit more background info.

Regards, Marco

Use some form of shared libs

See OSGeo/gdal#3060

Exactly what a shared-libs solution could be is not entirely clear, but it would need to provide the opportunity for rasterio/fiona to use them in CI/CD systems. Ideally, it might provide something like a pip optional extra pattern for installation of rasterio version X along with binary manylinux wheels for gdal version A, B or C (i.e. a complete matrix that would be consistent with CI/CD matrix builds).

e.g. duplicate binary libs when rasterio is installed along with fiona, pyproj, shapely and other sci-libs:

libgeos

   354224  2020-10-30 12:42   python/lib/python3.6/site-packages/Fiona.libs/libgeos_c-fiona-a68605fd.so.1.13.1
  2240712  2020-10-30 12:42   python/lib/python3.6/site-packages/Fiona.libs/libgeos--no-undefined-fiona-b94097bf.so
  2240704  2020-10-30 12:42   python/lib/python3.6/site-packages/rasterio.libs/libgeos--no-undefined-b94097bf.so
   354216  2020-10-30 12:42   python/lib/python3.6/site-packages/rasterio.libs/libgeos_c-a68605fd.so.1.13.1
  2240704  2020-10-30 12:42   python/lib/python3.6/site-packages/shapely/.libs/libgeos--no-undefined-b94097bf.so
   354216  2020-10-30 12:42   python/lib/python3.6/site-packages/shapely/.libs/libgeos_c-a68605fd.so.1.13.1

libgdal

 21884960  2020-10-30 12:42   python/lib/python3.6/site-packages/Fiona.libs/libgdal-fiona-9fe15c06.so.20.5.4
 23787528  2020-10-30 12:42   python/lib/python3.6/site-packages/rasterio.libs/libgdal-044c25e5.so.20.5.4

libcurl

  3620920  2020-10-30 12:42   python/lib/python3.6/site-packages/Fiona.libs/libcurl-fiona-ea538880.so.4.4.0
  3620912  2020-10-30 12:42   python/lib/python3.6/site-packages/rasterio.libs/libcurl-ea538880.so.4.4.0

libsqlite3

  1261392  2020-10-30 12:42   python/lib/python3.6/site-packages/Fiona.libs/libsqlite3-fiona-25a4bc97.so.0.8.6
  1271584  2020-10-30 12:42   python/lib/python3.6/site-packages/pyproj/.libs/libsqlite3-b65a32f0.so.0.8.6
  1419416  2020-10-30 12:42   python/lib/python3.6/site-packages/rasterio.libs/libsqlite3-bc0a2dd7.so.0.8.6

libz

    87856  2020-10-30 12:42   python/lib/python3.6/site-packages/Fiona.libs/libz-fiona-a147dcb0.so.1.2.3
    92080  2020-10-30 12:42   python/lib/python3.6/site-packages/pyproj/.libs/libz-eb09ad1d.so.1.2.3
    87848  2020-10-30 12:42   python/lib/python3.6/site-packages/rasterio.libs/libz-a147dcb0.so.1.2.3
    92080  2020-10-30 12:42   python/lib/python3.6/site-packages/numpy.libs/libz-eb09ad1d.so.1.2.3

HDF5 prints errors/warnings to terminal

This ticket replaces rasterio/rasterio#1847. We may be overlooking some HDF5 build or config option.

$ rio info http://download.osgeo.org/gdal/data/hdf5/vlstr_metadata.h5
HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
  #000: H5Adeprec.c line 193 in H5Aopen_name(): can't open attribute: '_FillValue'
    major: Attribute
    minor: Can't open object
  #001: H5Aint.c line 455 in H5A_open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #002: H5Oattribute.c line 528 in H5O_attr_open_by_name(): can't locate attribute: '_FillValue'
    major: Attribute
    minor: Object not found
/home/sean/.virtualenvs/sat/lib/python3.6/site-packages/rasterio/__init__.py:219: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
{"bounds": [0.0, 1673.0, 3024.0, 0.0], "colorinterp": ["undefined"], "count": 1, "crs": null, "descriptions": [null], "driver": "HDF5Image", "dtype": "uint8", "height": 1673, "indexes": [1], "mask_flags": [["all_valid"]], "nodata": null, "res": [1.0, -1.0], "shape": [1673, 3024], "tiled": false, "transform": [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0], "units": [null], "width": 3024}

Wheels for 1.0.25

Changes:

multibuild upgrade
GDAL 2.4.2 --with-hide-internal-symbols
Drop support for Python 3.4

Patch to get GDAL PR #3181

OSGeo/gdal#3181. Otherwise we have to wait until 3.3.0.

Upgrade curl to 7.80.0

The version which is currently being used by rasterio (7.79.1) has problems with Letsencrypt certificates:

$ rio info https://demo.nextgis.com/api/resource/5511/cog
WARNING:rasterio._env:CPLE_AppDefined in HTTP response code on https://demo.nextgis.com/api/resource/5511/cog.xml: 0
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 261, in rasterio._base.DatasetBase.__init__
  File "rasterio/_shim.pyx", line 78, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 216, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_HttpResponseError: CURL error: SSL certificate problem: certificate has expired

It seems the issue has been fixed in the recent version (7.80.0) of curl.

WebP-in-TIFF support

Trying to use WebP-in-TIFF TIFF's using these wheels results in: Cannot open TIFF file due to missing codec. Some sleuthing suggests libtiff needs to be built with WebP support to make this happen, and it looks like it should autodetect it.

I suspect this is not working because build_webp doesn't have build_tiff as a dependency, so it hasn't been built at the time libtiff is compiled
(

rasterio-wheels/config.sh

Line 130 in 8f14010

function build_tiff {

)

I am still trying to build a wheel myself to confirm this is the right fix and will update this again if successful but thought I would make this now before I forget...

Patch for GDAL PR 2310

See OSGeo/gdal#2310 and rasterio/rasterio#1877.

Python 3.8 wheels

I did it for Fiona last night, will copy the needed bits here and issue a post-release.

libgdal is too big

Actual status

$ ls -lh libgdal-a3e05b74.so.20.5.1
-rwxr-xr-x 1 root root 97M Jul  2 13:24 libgdal-a3e05b74.so.20.5.1
$ file libgdal-a3e05b74.so.20.5.1
libgdal-a3e05b74.so.20.5.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=8a8032a095e4f6d37c963305af0c7ab2963a079c, not stripped

Expected status

The lib should be stripped
The lib should be smaller e.-g. in the libgdal20 package =>

$ ls -lh /usr/lib/libgdal.so.20.3.2
-rw-r--r-- 1 root root 16M Feb  6  2018 /usr/lib/libgdal.so.20.3.2

Operating system

For example: Docker image: ubuntu:18.04

Rasterio version and provenance

1.0.24 manylinux1 wheel installed from PyPI using pip 9.0.1.
Filename: rasterio-1.0.24-cp37-cp37m-manylinux1_x86_64.whl

Update GDAL to latest 3.4.x

Or at least get the patch for OSGeo/gdal#4965 so that users don't hit rasterio/rasterio#2386.

Update multibuild, GEOS, GDAL

See https://github.com/matthew-brett/multibuild/issues/298.

GEOS -> 3.8.0

GDAL -> 2.4.4

cibuildwheel

This was brought up in #73 by @vincentsarago. I didn't want to go through the headache of transitioning from multi-build for pyproj. But, after troubles with multi-build, I decided to give it a try and am happy with it so far. I added reference to the PRs in case this becomes something you want to try.

Reference implementation:

	function build_proj {
	CFLAGS="$CFLAGS -g -O2"
	CXXFLAGS="$CXXFLAGS -g -O2"
	if [ -e proj-stamp ]; then return; fi
	fetch_unpack http://download.osgeo.org/proj/proj-${PROJ_VERSION}.tar.gz
	(cd proj-${PROJ_VERSION} \
	&& ./configure --prefix=$BUILD_PREFIX \
	&& make -j4 \
	&& make install)
	if [ -n "$IS_OSX" ]; then
	:
	else
	strip -v --strip-unneeded ${BUILD_PREFIX}/lib/libproj.so.*
	fi
	touch proj-stamp
	}

rasterio / rasterio-wheels Goto Github PK

rasterio-wheels's Introduction

rasterio-wheels

rasterio-wheels's People

Contributors

Stargazers

Watchers

Forkers

rasterio-wheels's Issues

Expected behavior and actual behavior.

Steps to reproduce the problem.

test.py

Operating system

Rasterio version

Links to other issues

Dependency Graph between Python and Shared Libraries

Issue Description

Suggested Vulnerability Patch Versions

Actual status

Expected status

Operating system

Rasterio version and provenance

Recommend Projects

Recommend Topics

Recommend Org