Comments (7)
Ah, right, Binary
type support was added in https://jira.mongodb.org/browse/ARROW-52, which has not been released yet.
from mongo-arrow.
Hi @mertbakir, can you please post the ValueError
traceback that you're seeing?
from mongo-arrow.
@blink1073 hello, here it is.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[14], line 1
----> 1 schema = Schema({'_id': ObjectId,
2 'client_id': Binary,
3 'location': {
4 'accuracy': int,
5 'battery': int,
6 'bearing': int,
7 'speed': int,
8 'total_distance': int,
9 'location': {'coordinates': list}
10 }
11 })
File (file:///C:/Users/.../venv/lib/site-packages/pymongoarrow/schema.py:45), in Schema.__init__(self, schema)
38 """Create a :class:`~pymongoarrow.schema.Schema` instance from a
39 mapping or an iterable.
40
41 :Parameters:
42 - `schema`: A mapping.
43 """
44 if isinstance(schema, abc.Mapping):
---> 45 normed = type(self)._normalize_mapping(schema)
46 else:
47 raise ValueError("schema must be a mapping or sequence")
File (file:///C:/Users/.../venv/lib/site-packages/pymongoarrow/schema.py:58), in Schema._normalize_mapping(mapping)
56 normed = {}
57 for fname, ftype in mapping.items():
---> 58 normed[fname] = _normalize_typeid(ftype, fname)
59 return normed
File (file:///C:/Users/.../venv/lib/site-packages/pymongoarrow/types.py:168), in _normalize_typeid(typeid, field_name)
166 return normalizer(typeid)
167 else:
--> 168 raise ValueError("Unsupported type identifier {} for field {}".format(typeid, field_name))
ValueError: Unsupported type identifier for field client_id
from mongo-arrow.
We can't infer an arrow data type from a list
, since we need to know what type of data it contains.
The schema should look something like:
from pyarrow import list_, float64
schema = Schema({'_id': ObjectId,
'client_id': Binary,
'location': {
'accuracy': int,
'battery': int,
'bearing': int,
'speed': int,
'total_distance': int,
'location': {'coordinates': list_(float64())}
}
})
This would make a good example to add to our docs for 1.0.
from mongo-arrow.
@blink1073 but client_id Binary throws error too. I am on 0.7, should I try it with 1.0?
from mongo-arrow.
Ah, right,
Binary
type support was added in https://jira.mongodb.org/browse/ARROW-52, which has not been released yet.
So my only option is to wait for the release? I thought I could install the in-dev. version.
from mongo-arrow.
For now, you can install from source.
from mongo-arrow.
Related Issues (20)
- Dataframe is all Nat and None after loading HOT 6
- Ability to query _id as string if it is of type ObjectId (e.g. "63fcb5aa5e1d7530a517dc44") HOT 1
- bson_iter_type(): precondition failed: iter->raw on find_pandas_all() HOT 6
- Casting timestamp in find_panads_all() HOT 1
- AttributeError: 'pyarrow.lib.DataType' object has no attribute '_type_marker' HOT 1
- Bug: find_arrow_all in version 1.0.1 returns wrong schema for nested bson.ObjectId while bson.ObjectId on root level works as documented HOT 2
- ARROW-175 Bug: nested data seems to be decoded even if not in schema HOT 3
- aggregate_arrow_all(...) >four times slower in version 1.0.2 compared to 1.0.1 with fields objects HOT 5
- Can `find_all_pandas` treat list of struct as nested dataframe? HOT 1
- Any chance you could fix the docs? HOT 4
- Does mongo-arrow provide real zero copy in the chain mongodb->arrow->pandas? HOT 3
- java version HOT 1
- Nested Data With Schema ERRor
- MongoDB's Decimal128 seems to be returned as fixed_size_binary[16] HOT 3
- Support for Tool HOT 1
- undefined symbol: _ZN5arrow6StatusC1ENS_10StatusCodeERKSs with airflow 2.8.1 HOT 1
- Trouble reading documents with empty embedded arrays HOT 1
- [documentation update request / feature request] `write` supported types
- Add an optional bool flag to the `write` function to skip writing `null` fields HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mongo-arrow.