n0nsmoker / sqlalchemy-serializer Goto Github PK
View Code? Open in Web Editor NEWSerrializer for SQLAlchemy models.
License: MIT License
Serrializer for SQLAlchemy models.
License: MIT License
I think the default the default datetime format should change. I have a situation where a lot of my tables are storing datetime to the second, but the serializer rounds them to the nearest minute.
this fixes it, but I think that this should be the default as it's probably more common to have data stored to the second than not.
datetime_format = '%Y-%m-%d %H:%M:%S'
Using the serializer successfully with my data model, but I was wondering if there's a simple way to get it to filter out "null" items when serializing models or dictionaries?
My data is pulled from Postgresql, and as I'm feeding much of it to SOLR for indexing purposes, I'd like to reduce the document size and eliminate unneeded elements from the JSON output. If there's already an option in the Serializer I'm missing or an easy way to accomplish this via the model, I'd appreciate a pointer.
Thanks in advance.
Reproduced on PostgreSQL JSONb fields.
class MyModel(Model, SerializerMixin):
data = Column(JSONB)
data = dict(some_key='hi there!')
entity = MyModel(data=data)
print(entity.data)
{'some_key': 'hi there!'}
print(entity.to_dict()['data'])
['some_key']
If I use SerializerMixin
inside my model
class Category(Model, SerializerMixin):
__tablename__ = 'category'
category_id = Column(Integer(), primary_key=True)
category_name = Column(String(100))
It doesn't have support for list type, It works on the single item Category.query.first()
but not on queries with list type, I need to use list comprehension for list type like in the example below:
categories = Category.query.all()
response = [category.to_dict() for category in categories]
DESIRED SOLUTION:
categories = Category.query.all()
response = categories.to_dict()
I'm trying to use the sqlalchemy-serializer to transform my query outputs to json. My models link to each other with relationships. To make sure I can export the tables on their own (without the link to children tables) I added some negative serialize rules in the models. However, when I try to overwrite them at the query level, I encountered the following error:
test.py 72 <module>
pp.pprint(i.to_dict(rules=("areas")))
serializer.py 92 to_dict
return s(self, only=only, extend=rules)
serializer.py 117 __call__
return self.serialize(value)
serializer.py 168 serialize
return callback(value)
serializer.py 265 serialize_model
v = getattr(value, k)
AttributeError:
'Object' object has no attribute 's'
You can find the test.py code attached to reproduce this issue
I've run into an issue where I have a lot of columns on a model and I want to only exclude one specific column from serialization. Is there a way to do it besides typing in all the columns except the one I want to exclude in the serialize_only
field?
我来自**江苏无锡,最近在模仿php的laravel写一份flask版的api,作者这个库帮助了我,但是orm的all()方法查询的结果是list,to_dict是无法处理的,希望作者能完善下,我自己在基类里也做了相应的修改,在此表示感谢!thank author~
I added the lib to my requirements.txt and installed it using pip -r
. When I try to run my flask app I got this error:
Traceback (most recent call last):
File "run.py", line 2, in <module>
from app.routes import *
File "/home/lucivaldo/Desenvolvimento/python_apis/parentesco-api/app/routes/__init__.py", line 3, in <module>
from app.dao.ParentescoDao import ParentescoDao
File "/home/lucivaldo/Desenvolvimento/python_apis/parentesco-api/app/dao/ParentescoDao.py", line 1, in <module>
from app.models.Parentesco import Parentesco
File "/home/lucivaldo/Desenvolvimento/python_apis/parentesco-api/app/models/Parentesco.py", line 3, in <module>
from sqlalchemy_serializer import SerializerMixin
File "/home/lucivaldo/.local/lib/python3.6/site-packages/sqlalchemy_serializer/__init__.py", line 1, in <module>
from .serializer import SerializerMixin, Serializer
File "/home/lucivaldo/.local/lib/python3.6/site-packages/sqlalchemy_serializer/serializer.py", line 9, in <module>
from lib.timezones import to_local_time, format_date, format_datetime
ModuleNotFoundError: No module named 'lib'
I'm using SQLAlchemy-serializer 0.6, SQLAlchemy 1.1.15, Flask 0.12.2 and python 3
Making it easier to parse the class to json.
Something like:
import json
# [....]
class SerializerMixin(object):
# [...]
def to_json(self):
json.dumps(self.to_dict())
Hi
I have been playing with this neat addin all day.. its great
I did pull my hair out for an hour trying to debug with rules were not working
turns out using your code samples in the readme do not work if you only pass in 1 rule.. if you pass in more than 1 then it works
from the readme.. this works
result = item.to_dict(rules=('-somefield', '-anotherone.nested1.nested2'))
this does not
result = item.to_dict(rules=('-somefield'))
seems RULES is setting a Set()
so the correct way to pass in rules etc would be using curly brackets
result = item.to_dict(rules={'-somefield', '-anotherone.nested1.nested2'})
using curly brackets
passing in just one {'some field'} does not cause errors
just FYI for anyone looking
I read the README
, it have two lines:
serialize_only and serialize_rules work the same way as to_dict's arguments
If you pass rules in serialize_only the serializer becomes NOT greedy and returns ONLY fields listed there.
While I test to_dict(only=())
and serialize_only
in my codes:
class Flow(SerializerMixin, Model):
__tablename__ = 'flows'
id = Column(db.Integer, primary_key=True)
serialize_only = ('name', 'nodes.name', 'nodes.category', 'nodes.mode', 'nodes.sub_flow')
name = Column(db.String(100), nullable=False)
nodes = relationship('Node',
backref=db.backref('flow', uselist=False),
single_parent=True,
cascade='all, delete, delete-orphan',
order_by='asc(Node.seq)')
class Node(SerializerMixin, Model):
__tablename__ = 'nodes'
id = Column(db.Integer, primary_key=True)
flow_id = reference_col('flows', nullable=False)
name = Column(db.String(100), nullable=False)
category = Column(db.String(100), default=0)
mode = Column(db.Integer(), default=0)
@property
def sub_flow(self):
if self.category == 'sub_flow':
flow = Flow.query.filter_by(id=self.mode).first()
return flow
else:
return None
I found, if I use flow.to_dict(only=('name', 'nodes.name', 'nodes.category', 'nodes.mode', 'nodes.sub_flow'))
, it will Max recursion
. And set serialize_only = ('name', 'nodes.name', 'nodes.category', 'nodes.mode', 'nodes.sub_flow')
in Flow
, everything works fine.
Is serialize_only
works for all instances and only
just work for the instance that called to_dict
?
And I found serialize_only
will override only
in to_dict
, so I can't serialize another data structure.
Is there a way to set a param for making only
for all instances which is same model class?
Collecting SQLAlchemy-serializer
Using cached https://files.pythonhosted.org/packages/07/dc/bd77ce9bddf561c4eeb82103795c96f6c83e52469ca73c48790dfb6841dc/SQLAlchemy-serializer-1.1.3.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-l8bsyi_q/SQLAlchemy-serializer/setup.py", line 6, in <module>
with open(path.join(this_directory, 'README.md'), encoding='utf-8') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-l8bsyi_q/SQLAlchemy-serializer/README.md'
So I had to force a logging change for two reasons, the print statements slow things down and secondly fills up my logs with a crap load of actions. While maybe helpful in debug situations, not ideal in production. Maybe I missed in documentation, but can we change the default log level to WARN, unless defined otherwise? I did the following in my models file for my Flask app and solved for me.
serializer_logger = logging.getLogger('serializer')
serializer_logger.setLevel(level="WARN")
Hey, I'm trying to serialize many to many relationship. I have tried many varieties of combinations of rules but none of them worked. When user has exam then max recursion error occurs, Since it tries to traverse again and again from user->exams->users->exams ......
Following is my code...
UserExamMappingTable = db.Table("UserExamMapping", db.Model.metadata,
db.Column("id", db.Integer, index=True, primary_key=True),
db.Column("userId", db.Integer, db.ForeignKey("User.id")),
db.Column("examId", db.Integer, db.ForeignKey("Exam.id")),
db.Column("isActive", db.Boolean(), default=True))
class User(BaseModel, db.Model):
__tablename__ = "User"
serialize_rules = ('-exams.users',)
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(200))
email = db.Column(db.String(200))
password = db.Column(db.String(200))
exams = db.relationship("Exam", secondary=UserExamMappingTable, back_populates="users")
class Exam(BaseModel, db.Model):
__tablename__ = "Exam"
serialize_rules = ('-users.exams',)
id = db.Column(db.Integer, primary_key=True)
examName = db.Column(db.String(200))
answers = db.Column(db.String(5000))
users = db.relationship("User", secondary=UserExamMappingTable, back_populates="exams")
If I use a name, in this case, email_template by itself, it doesn't work. However, if I do something like
'-steps.email_template.name' , it works.
class EmailTemplates(Resource):
# @jwt_required
def get(self):
email_templates = EmailTemplateModel.find_by_id(1)
# try:
return {
'email_templates': email_templates.to_dict(rules=('-steps.email_template', ))
}, 200
Result with ('-steps.email_template',):
{
"email_templates": {
"id": 1,
"type": "reply",
"name": "myTemplate",
"steps": [
{
"id": 8,
"email_template_id": 1,
"email_template": {
"id": 1,
"type": "reply",
"name": "myTemplate",
"body": "some html",
"subject": "test"
},
"campaign_id": 99
}
],
"body": "some html",
"subject": "test"
}
}
Result with (-steps.email_template', '-steps.email_template.name'):
{
"email_templates": {
"steps": [
{
"email_template_id": 1,
"campaign_id": 99,
"id": 8,
"email_template": {
"type": "reply",
"subject": "test",
"id": 1,
"body": "some html"
}
}
],
"type": "reply",
"subject": "test",
"id": 1,
"name": "myTemplate",
"body": "some html"
}
}
As you can see, steps.email_template.name is gone.
If I change "email_template" to "template", it works.
class EmailTemplateModel(db.Model, SerializerMixin):
__tablename__ = 'email_templates'
id = db.Column(db.Integer, primary_key = True)
name = db.Column(db.String(120))
type = db.Column(db.String(120))
subject = db.Column(db.String(120))
body = db.Column(db.VARCHAR(2000))
steps = db.relationship(
'StepModel', backref='template', lazy = 'select' <----
)
class EmailTemplates(Resource):
# @jwt_required
def get(self):
email_templates = EmailTemplateModel.find_by_id(1)
# try:
return {
'email_templates': email_templates.to_dict(rules=('-steps.template', ))
}, 200
# except:
# return {'message': 'Something went wrong'}, 500
Now result with ('-steps.template',):
{
"email_templates": {
"name": "myTemplate",
"type": "reply",
"steps": [
{
"campaign_id": 99,
"email_template_id": 1,
"id": 8
}
],
"body": "some html",
"subject": "test",
"id": 1
}
}
If the rule is something like '-step.underscore_name' , it doesn't work.
*** UPDATED ***
Hi,
I'm facing an issue with a simple recursive model for an app using Flask / SQL Alchemy and SQLAlchemy-serializer. I don't really know if the issue come from my code or if it is a real issue. Here is my code:
models.py
class Node(db.Model, SerializerMixin):
__tablename__ = 'node'
__table_args__ = {'mysql_engine': 'InnoDB'}
id = db.Column(db.Integer, primary_key=True)
parent_id = db.Column(db.Integer, db. ForeignKey('node.id'))
name = db.Column(db.String(50))
children = db.relationship("Node")
node.py
def get_node(node_id):
node = Node.query.get(node_id)
return node.to_dict()
My data
id | parent_id | name
1 | NULL | ROOT
2 | 1 | node 1
3 | 1 | node 2
4 | 3 | subnode 1
5 | 4 | sub subnode 1
When the get_node function is executed with node_id = 1, I have the following results:
{
"children": [
{
"children": [],
"id": 2,
"name": "node 1",
"parent_id": 1
},
{
"children": [
{
"children": [
{
"children": [],
"id": 5,
"name": "node 2 subnode 1 subnode 1",
"parent_id": 4
}
],
"id": 4,
"name": "node 2 subnode 1",
"parent_id": 3
}
],
"id": 3,
"name": "node 2",
"parent_id": 1
}
],
"id": 1,
"name": "ROOT",
"parent_id": null
}
And this is normal. Now, I would like to have only the node id 1 and its direct subnodes, not the full tree. From the README and troubleshooting section, I've seen to limit recursive depth we can exclude from serialization nested field. I've changed my model class to this but still having the same results:
class Node(db.Model, SerializerMixin):
__tablename__ = 'node'
__table_args__ = {'mysql_engine': 'InnoDB'}
serialize_rules = ('-children.children',)
id = db.Column(db.Integer, primary_key=True)
parent_id = db.Column(db.Integer, db. ForeignKey('node.id'))
name = db.Column(db.String(50))
children = db.relationship("Node")
The expected results is :
{
"children": [
{
"id": 2,
"name": "node 1",
"parent_id": 1
},
{
"id": 3,
"name": "node 2",
"parent_id": 1
}
],
"id": 1,
"name": "ROOT",
"parent_id": null
}
Thanks in advance for your help.
Hi,
I found myself systematically disabling every relation of my models, so that by default it does not recurse over all my tables.
Is there a way to disable relationship recursion by default ?
Thanks in advance.
my model entity contains an boolean property like
deleted = db.Column(db.Boolean, default=False)
after a model entity write to mysql, mysql schema the according delete column type is tinyint, so the property 'deleted' in mysql is 0 or 1.
while I read the value of my entity from mysql, the property 'deleted' can not serialized.
I want to learn more from you. thanks.
Hello, first of all thank you for this serializer .
I have issue when i try to serialize Model.query.all() when i have multiple results.
Example code:
contacts = Contact.query.all() return contacts.to_dict()
I got error: AttributeError: 'list' object has no attribute 'to_dict'
I get a recursion iWhen I use .to_dict()
on an instance with bi-directional relationships.
File "D:\Brendan\sbc\venv37-32\lib\site-packages\sqlalchemy_serializer\serializer.py", line 159, in serialize
if isinstance(value, types):
File "C:\Program Files (x86)\Python37-32\lib\abc.py", line 139, in instancecheck
return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
If I add a rule to exclude the relationship in the output then it works ok.
my_inst.to_dict( rules=( '-some_relationship', ) )
Is there a work-around or fix for this?
/app/init.py
from flask_sqlalchemy import SQLAlchemy
# [...]
sqlalchemy_db = SQLAlchemy()
# [...]
sqlalchemy_db.init_app(app)
/app/models.py
import uuid
from app import sqlalchemy_db
from sqlalchemy_serializer import SerializerMixin
def generate_uuid():
return str(uuid.uuid4())
class SomeTable(sqlalchemy_db.Model, SerializerMixin):
id = sqlalchemy_db.Column(sqlalchemy_db.Integer, primary_key=True)
uuid = sqlalchemy_db.Column(UUID(as_uuid=True), default=generate_uuid)
/app/routes.py
from http import HTTPStatus
from flask import jsonify
from flask import request, Response
# [...]
@bp.route('/find/by/id/<int:sometable_id>')
def testing_route(sometable_id: int) -> (Response, HTTPStatus):
some_content= SomeTable.query.filter_by(id=sometable_id).first()
if some_content:
return jsonify(some_content.to_dict()), HTTPStatus.OK
return jsonify(dict()), HTTPStatus.NO_CONTENT
Error when serializing:
sqlalchemy_serializer.serializer.IsNotSerializable: Unserializable type:<class 'uuid.UUID'> value:49a03933-977c-48b6-b992-d1baf4f2e096
Not sure if I've done something wrong, but it seems like SQLAlchemy-serializer is not currently getting str(uuid)
when attempting to serialize.
if result is none its gives "'NoneType' object has no attribute 'to_dict'" how to fix this?
Is there a way to deserialise from dict back to model object? a from_dict()
perhaps?
I serialze to dict, and the to JSON via dumps()
. Works a treat.
Now I want to do the reverse. JSON => dict => Model.
I tried using the model constructor with a unpacked dict. exampe:
d = json.loads( json_input )
log = Log( **d )
However one of the fields is DateTime
and I get the following error.
sqlalchemy.exc.StatementError: (builtins.TypeError) SQLite DateTime type only accepts Python datetime and date objects as input.
[SQL: INSERT INTO "Log" (timestamp, temperature) VALUES (?, ?)]
[parameters: [{'timestamp': '2020-08-06 13:11', 'temperature': 23.4}]]
``
Hello,
I have an application which used the Flask/Sqlite/SqlAlchemy + REST. I want to serialize and deserialize a list of object from the query result
@app.route('/all', methods=["GET"])
def all():
bookList = Book.query.all()
response = app.response_class(
response=json.dumps(bookList.__dict__, lambda o: o.__dict__, indent=4),
mimetype='application/json'
)
return response
class Book(db.Model, SerializerMixin):
title = db.Column(db.String(80), unique=True, nullable=False, primary_key=True)
But this doesn't work.
I didn't find any information for this problem, can you tell me how to do it, please?
Thanks.
class Users(db.Model, BaseModel, SerializerMixin):
__tablename__ = 'users'
__schema_extend__ = ('-password',)
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(255), nullable=True)
email = db.Column(db.String(255))
tel = db.Column(db.String(20), nullable=True)
password = db.Column(db.String(255))
status = db.Column(db.Integer)
remember_token = db.Column(db.String(100), nullable=True)
created_at = db.Column(db.Integer, nullable=True)
updated_at = db.Column(db.Integer, nullable=True)
class Suggest(BaseModel, db.Model, SerializerMixin):
__tablename__ = 'suggest'
id = db.Column(db.Integer, primary_key=True)
add_time = db.Column(db.Integer)
users_id = db.Column(db.Integer, db.ForeignKey('users.id'))
users = db.relationship('Users', backref='backSuggest', lazy='dynamic')
message = db.Column(db.String(255))
@app.route('/api/v2/userInfo', methods=['POST'])
def getInfo():
id = request.json.get('id')
data = Users.query.filter_by(id=id).all()
datas = Utils.db_l_to_d(data)
return BaseController().successData(datas)
class Utils:
@staticmethod
def db_l_to_d(data):
data_list = []
for val in data:
val_dict = val.to_dict()
data_list.append(val_dict)
data = {}
data = data_list
return data
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 93, in _fork
return serializer(value, **kwargs)
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 45, in call
return self.serialize_iter(value)
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 110, in serialize_iter
res.append(self._fork(value=v))
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 93, in _fork
return serializer(value, **kwargs)
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 52, in call
return self.serialize_model(value)
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 135, in serialize_model
res[k] = self._fork(key=k, value=v)
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 93, in _fork
return serializer(value, **kwargs)
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 52, in call
return self.serialize_model(value)
File "C:\python3\lib\site-packages\sqlalchemy_serializer\serializer.py", line 135, in serialize_model
res[k] = s
I can't backref='backSuggest' give Users models
Is there any way to specify a custom serialization function for one column by name? Like the serialize_types
for custom type, something like
class CustomSerializerMixin(SerializerMixin):
additional_serialize_rule = (('column1', lambda x: x),)
edit: my current get around is this, but it would be nice if this step could be done during serialization and reduce the redundant logic
class CustomSerializerMixin(SerializerMixin):
def to_dict(self, *args, **kwargs):
obj_dict = super().to_dict(*args, **kwargs)
# ...custom logic to transform obj_dict
return obj_dict
Instead of throwing RecursionError
please add a default depth for example: 2nd level or 3rd level
Using just RECURSION_DEPTH : 2
as a global variable and should be modifiable inside the model or during .to_dict(depth=2)
I was trying to convert my result into a dictionary.
Model:
class User(Base, SerializerMixin):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False, unique=False)
ic_number = Column(Integer, nullable=False, unique=True)
phone_number = Column(Integer, nullable=True)
pin_number = Column(String(350), nullable=True)
email = Column(String(250), nullable=True)
nfc_binds = relationship("Nfc", backref='user', lazy=True)
timestamp_created = Column(DateTime, default=datetime.datetime.utcnow)
details = relationship("Details", backref='user', lazy=True) #Params 123
created_by = Column(String(250), nullable=False) #Machine ID that created this user
class Nfc(Base, SerializerMixin):
__tablename__ = 'nfc_card'
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey('user.id'), nullable=False)
nfc_id = Column(String(20), nullable=False)
>>> item.__dict__
{'_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x7f9611e742b0>, 'timestamp_created': datetime.datetime(2019, 10, 18, 23, 39, 8, 577158), 'pin_number': None, 'ic_number': 715, 'id': 4, 'created_by': '123411111111', 'email': None, 'phone_number': None, 'name': 'Sayanee', 'details': [], 'nfc_binds': [<database_setup.Nfc object at 0x7f9611e0f4e0>]}
>>> item.to_dict()
Traceback error message:
if subclass in cls._abc_cache:
RecursionError: maximum recursion depth exceeded
my models
class HtMsg(Base):
__tablename__ = 'ht_msg'
id = Column(Integer, primary_key=True)
name = Column(String(255), nullable=False, server_default=FetchedValue())
msg = Column(Text, nullable=False)
room_uuid = Column(String(255), nullable=False, server_default=FetchedValue())
user_id = Column(Integer, nullable=False)
type = Column(Integer, nullable=False)
head_img = Column(String(255), nullable=False, server_default=FetchedValue())
created_at = Column(BigInteger, nullable=False)
send_status = Column(Integer, nullable=False)
class Msg(Base, HtMsg, SerializerMixin):
serialize_rules =('msg', 'formatMsg')
def formatMsg(self):
return json.loads(super().msg)
I want to reset 'msg' but python tell me it is none.Can you help me ?
class Msg(Base, HtMsg, SerializerMixin):
serialize_rules =('msg')
def msg(self):
return json.loads(super().msg)
Hi,
First of all, it is a very useful module. However, it will be more wonderful if it can support custom property. The .to_dict()
seems cannot detect any property assigned from @property
, so I have to set the key and the value in the dict manually.
Sincerely,
Edwin Lu
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.