Comments (6)
looks like a duplicate to #72, should be fixed by #73.
from delta-rs.
@martindut fix as been merged into master, please reopen if you are still experiencing this issue with 0.2.1 release in PyPI.
from delta-rs.
@houqp , now I'm getting this error if I run df = dt.to_pyarrow_table():
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/site-packages/deltalake/__init__.py", line 38, in to_pyarrow_table return self.to_pyarrow_dataset().to_table() File "/usr/local/lib/python3.8/site-packages/deltalake/__init__.py", line 33, in to_pyarrow_dataset return dataset(keys, filesystem=f"{paths[0].scheme}://{paths[0].netloc}") File "/usr/local/lib/python3.8/site-packages/pyarrow/dataset.py", line 674, in dataset return _filesystem_dataset(source, **kwargs) File "/usr/local/lib/python3.8/site-packages/pyarrow/dataset.py", line 426, in _filesystem_dataset fs, paths_or_selector = _ensure_multiple_sources(source, filesystem) File "/usr/local/lib/python3.8/site-packages/pyarrow/dataset.py", line 298, in _ensure_multiple_sources filesystem, is_local = _ensure_fs(filesystem) File "/usr/local/lib/python3.8/site-packages/pyarrow/dataset.py", line 232, in _ensure_fs filesystem, prefix = FileSystem.from_uri(fs_or_uri) File "pyarrow/_fs.pyx", line 347, in pyarrow._fs.FileSystem.from_uri File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status pyarrow.lib.ArrowInvalid: Unrecognized filesystem type in URI: abfss://[email protected]
from delta-rs.
@martindut I think our integration tests aren't covering the use of pyarrow and the Azure storage engine. Would you mind opening a new issue with some details?
from delta-rs.
@houqp I think this is still an issue. I am on version
deltalake-0.4.8 numpy-1.20.3 pyarrow-4.0.0
When I run:
from deltalake import DeltaTable
import os
os.environ['AZURE_STORAGE_ACCOUNT']='xxx'
os.environ['AZURE_STORAGE_KEY']='xx'
dt = DeltaTable('abfss://[email protected]/delta_example/')
print(f'table version: {dt.version()}')
print(f'list of files: {dt.file_paths()}')
# convert to data.frame
df = dt.to_pyarrow_table().to_pandas()
The dt.to_pyarrow_table()
causes the following error:
ArrowInvalid: Unrecognized filesystem type in URI: abfss://[email protected]
from delta-rs.
@samuel100 are you able to read one of the parquet file that's causing the error with pyarrow
directly? I don't have an azure environment to test and debug this at the moment.
from delta-rs.
Related Issues (20)
- Deletion `_change_type` does not appear in change data feed HOT 3
- Don't commit if commit has transaction identifier that has been committed already HOT 3
- Newest docs (potentially) not released
- DeltaTable over S3: creation hangs up when being run from current thread scheduler
- Compaction can rewrite files without reducing file count
- AsyncChunkReader::get_bytes error: Generic MicrosoftAzure error: error decoding response body HOT 6
- Converting DeltaTable to Pandas causes timestamp error HOT 14
- `to_pyarrow_table()` on a table in S3 kept getting "Generic S3 error: error decoding response body" HOT 1
- Allow `pyarrow.dataset.Expression` in `filters` kwarg of `deltalake.DeltaTable.to_pyarrow_table`
- panic in `push_down_filter` HOT 2
- 0.18.1 reintroduces S3 multipart upload bug HOT 5
- Concurrent writers return error despite successfully written to the file HOT 4
- InvalidTableLocation("Unknown scheme: gs") on 0.18.0
- Support HDFS via hdfs-native package HOT 1
- Support string and binary view types. HOT 3
- Document Databricks/delta-rs protocol version support by release
- Arrow: Parquet does not support writing empty structs when creating checkpoint
- Unable to specify columns with a dot in the name in predicate HOT 1
- Support for SQL syntax for filtering similar to PySpark
- Getting error when converting a partitioned parquet table to delta table HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from delta-rs.