Comments (2)
Have the same problem using date. It is supported according to the AVRO specification, see here.
@grkhr I guess writing it into a string field will not be supported
So, the type casting here should be extended:
import datetime
NUMPY_TO_AVRO_TYPES = {
....
datetime.date: {'type': 'int', 'logicalType': 'date'}
}
I just don't know where the date would be converted to int (starting at unix epoch 1 January 1970)
from pandavro.
- Regarding
datetime.date
:
pandas
will represent any dataframe containing datetime.date
values as having a dtype of object
. As a result, the solution suggested by @hz-lschick won't behave as expected, because even if you put an entry for datetime.date
into NUMPY_TO_AVRO_TYPES
, it won't get looked at, because pandas
will give that column's dtype as object
instead of datetime.date
.
pandavro
largely works off of the dtypes given by pandas
, so a simpler solution would be to convert your datetime.date
columns into np.datetime64
columns, which you could accomplish like this:
df['column_name'] = df['column_name'].astype(np.datetime64)
- Regarding
pd.NaT
:
This issue actually comes from fastavro
, rather than pandavro
. pandavro
correctly generates the Avro schema for a column containing pd.NaT
values. But when the fastavro
library tries to actually write this to file, it tries to access the timestamp
attribute, which pd.NaT
does not have.
The easiest solution to this (other than trying to fix this in fastavro
) would be to either replace all the NaT
values with a placeholder np.datetime64
, or to cast the column to a string, which can then be written by fastavro
.
from pandavro.
Related Issues (14)
- Latest pandas does not have DatetimeTZDtypeType HOT 2
- Release 1.5.x HOT 1
- allow for process_record() while reading in avro
- Support for bytes HOT 4
- Tests failing in master HOT 6
- pypi homepage points to the wrong link HOT 1
- Auto-deploy on merge to master?
- Does the fastavro dependency version need to be pinned? HOT 2
- from_records() got an unexpected keyword argument 'na_dtypes' HOT 2
- decimal logical type : TypeError: can only concatenate str (not "int") to str HOT 1
- Add compatibility with Python 3.11 HOT 1
- add support of snappy codec (ValueError: snappy codec is supported but you need to install python-snappy)
- Problem with datatype datetime64[ns] HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pandavro.