encode / typesystem Goto Github PK
View Code? Open in Web Editor NEWData validation, serialization, deserialization & form rendering. 🔢
Home Page: https://www.encode.io/typesystem/
License: BSD 3-Clause "New" or "Revised" License
Data validation, serialization, deserialization & form rendering. 🔢
Home Page: https://www.encode.io/typesystem/
License: BSD 3-Clause "New" or "Revised" License
Add an Array field.
hey,
i'm wondering what made you decide to use a classmethod for Schema.validate()
which then returns an instance of the schema.
i am wondering how this would work nicely when you're actually interested in creating custom (application specific) class instances, such as sqlalchemy orm class instances (‘database models’), since all attributes need to be copied afterwards.
also, to me ‘an instance of a schema’ is not the obvious answer for the question ‘what's the result of my validation’, yet typesystem
seems to think that is fine.
just curious to hear your thoughts. thanks in advance!
@tomchristie Are you open to support @dataclass
-decorated classes in typesystem
? Similar to what attrs
project is doing with @attr.s(auto_attribs=True)
. I'd like to try contributing the support, but wanted to check if it's within the scope of the project.
Here is how some of the examples from typesystem
documentation could be rewritten, if dataclasses were supported:
from dataclasses import dataclass
from datetime import date
from typesystem import field, validate
# Example from README.md
@dataclass
class Artist:
name: str = field(max_length=100)
@dataclass
class Album:
title: str = field(max_length=100)
release_date: date
artist: Artist
validate(Album, {
"title": "Double Negative",
"release_date": "2018-09-14",
"artist": {"name": "Low"}
})
# Example from documentation
@dataclass
class Organisation:
name: str = field(title="Name", max_length=100)
date_created: date = field(title="Date created", default=datetime.date.today)
owner: User = field(allow_null=True)
typesystem.field
could be a thin wrapper around around dataclasses.field
, using metadata
argument. For example, Artist
and Organisation
above are the same as this:
@dataclass
class User:
name: str = dataclasses.field(metadata={'typesystem': {'max_length': 100}})
@dataclass
class Organisation:
name: str = dataclasses.field(
metadata={'typesystem': {'title': 'Name', 'max_length': 100}},
)
date_created: date = dataclasses.field(
default_factory=datetime.date,
metadata={'typesystem': {'title': 'Date created'}},
)
owner: User = dataclasses.field(
metadata={'typesystem': {'allow_null': True}},
)
Add trim=True
argument to the String
field.
Eg.
class Something(typesystem.Schema):
created = typesystem.DateTime(default=datetime.datetime.now)
...
Autocompletion / intellisense is a nice-to-have feature. Currently neither PyCharm nor VScode offer any autocompletion for typesystem schema classes. This may lessen the likelihood of its usage and adoption.
It would be nice if a script was included with typesystem that could be configured to be called automatically on file save and that auto-generates/updates a .pyi stub file for classes subclassing typesystem.Schema
.
Auto-generated stub code could be surrounded by special markers in python comments that serve double duty.
Including typesystem schema classes along with other code would result in .pyi files that didn't include stubs for the other code. Some possible options:
What do you think? Comments?
Just needs a tweak in Form.render_field
to set value=""
, and a test case.
Database validation needs an async
/await
interface. ValidationResult
should be an awaitable, and should error if __bool__
or __iter__
are called, and it has awaitable validators, and has not yet been awaited.
would TypeSystem be able to handle CSRF token so the form could be validated w/o risk ?
I've copy-pasted "Tokenized Errors" example from documentation to example.py
:
$ cat example.py
import typesystem
class Config(typesystem.Schema):
num_worker_processes = typesystem.Integer()
enable_auto_reload = typesystem.Boolean()
text = '''{
"num_worker_processes": "x",
"enable_auto_reload": "true"
}'''
value, messages = typesystem.validate_json(text, validator=Config)
assert value is None
for message in messages:
line_no = message.start_position.line_no
column_no = message.start_position.column_no
print(f"Error {message.text!r} at line {line_no}, column {column_no}.")
# Error 'Must be a number.' at line 2, column 29.
However, it fails with ValidationError
. I expected it to print 'Must be a number ...'
instead:
$ pipenv install typesystem
$ pipenv run python3 -c 'import typesystem; print(typesystem.__version__)'
0.2.2
$ pipenv run python3 example.py
Traceback (most recent call last):
File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/tokenize/positional_validation.py", line 13, in validate_with_positions
return validator.validate(token.value)
File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/schemas.py", line 147, in validate
value = validator.validate(value, strict=strict)
File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/fields.py", line 545, in validate
raise ValidationError(messages=error_messages)
typesystem.base.ValidationError: {'num_worker_processes': 'Must be a number.'}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "example.py", line 12, in <module>
value, messages = typesystem.validate_json(text, validator=Config)
File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/tokenize/tokenize_json.py", line 197, in validate_json
return validate_with_positions(token=token, validator=validator)
File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/tokenize/positional_validation.py", line 36, in validate_with_positions
raise ValidationError(messages=messages)
typesystem.base.ValidationError: {'num_worker_processes': 'Must be a number.'}
errors = {}
argument for overriding the error strings.errors
as an attribute adding the the defaults, rather than overriding them.Would it be possible for the Choice
class to include a list of acceptable choices in the validation error output?
Here is a snippet doing this using subclassing.
import typesystem
class MyChoice(typesystem.fields.Choice):
def __init__(self, *, choices=None, **kwargs):
super().__init__(choices=choices, **kwargs)
self.single_choices = [x[0] for x in self.choices]
self.errors["choice"] = "Not a valid choice. Please choose: {single_choices}"
field = MyChoice(choices=[('foo', 'foo')])
field.validate('bar')
This will raise the following error message (this is an ipython notebook error message, but hopefully demonstrates ok):
---------------------------------------------------------------------------
ValidationError Traceback (most recent call last)
<ipython-input-65-854f15494f03> in <module>
8 field = MyChoice(choices=[('foo', 'foo')])
9
---> 10 field.validate('bar')
~/Documents/entanglement/venv/lib/python3.7/site-packages/typesystem/fields.py in validate(self, value, strict)
384 return None
385 raise self.validation_error("required")
--> 386 raise self.validation_error("choice")
387 return value
388
ValidationError: Not a valid choice. Please choose: ['foo']
Is it possible to mark some field as optional? What I mean:
property = typesystem.String(max_length=100, optional=True)
This would mean that property
field may be ommited, but if present it must be String
with maximum length equal 100 chars.
Similar to List(unique=True)
, except that uniqueness should only be a failure case if strict=True
Display as a <select multiple>
Hey,
I was fiddling with composite data types, in particular with Reference
to implement nested schemas, and there seems to be an issue with how arrays and objects of references are serialized when calling dict()
on the parent class.
Consider the following schemas:
import typesystem as ts
class Child(ts.Schema):
pass
class ParentA(ts.Schema):
child = ts.Reference(to=Child)
class ParentB(ts.Schema):
children = ts.Array(ts.Reference(Child))
class ParentC(ts.Schema):
children = ts.Object(properties=ts.Reference(Child))
Test cases with what should happen IMO:
pa = ParentA(child=Child())
assert dict(pa) == {"child": {}} # OK
pb = ParentB(children=[Child()])
assert dict(pb) == {"children": [{}]} # FAIL
# {"children": [Child()]}
pc = ParentC(children=[Child()])
assert dict(pc) == {"children": {"john": {}}} # FAIL
# {"children": {"john": Child()}}
This causes issues when passing the result of dict(pb)
or dict(pc)
to, say, json.dumps()
, because it gets Child
instances which are not JSON-serializable.
Am I simply abusing Array
and Object
? Is there another way of implementing composite fields with nested schemas?
Currently unimplemented. See https://github.com/encode/typesystem/blob/master/typesystem/formats.py#L128 and https://github.com/encode/typesystem/blob/master/typesystem/formats.py#L87
Needs:
The documentation indicates that the DateTime, Date, and Time fields return the appropriate datetime object instances. However, the validate methods actually return strings instead.
I defind a Schema which contains Array field
class Node(Schema):
name = String()
cams = Array(items=String(), min_items=0, max_items=4)
when I use dict(node) after validated, It raised error:
attribute of type 'list' is not callable
help
Dear @tomchristie,
I'm in the process of adding a feedstock for typesystem
to conda-forge: conda-forge/staged-recipes#8008 in order to be able to have the latest versions of apistar
available there as well.
Please let me know if that's okay with you :) and thank you very much for those awesome packages!
E.g. allow back-references or cyclical references.
We could register schema class names whenever Schema is subclassed, which would allow us to do this:
artist = typesystem.Nested('Artist')
A lazy import string would also work:
artist = typesystem.Nested('myproject.schemas:Artist')
date_of_travel = Date(validator=may_not_be_in_past)
@classmethod
def validate_date_of_travel(cls, value):
...
validate_all
or something similar.I'm using typesystem to implement a client for an HTTP API, and it's almost perfect for the jobs of constructing payloads to send and parsing payloads that have been received.
The API in question includes nested structures, so I'd like to be able to feed response.json()
straight into a schema to get back what's effectively a data class. Using MySchema.validate(payload)
works, but it makes the library fragile to changes in the API, like extra options being added to Choice
fields that would then cause validation errors on all responses.
MySchema(payload)
has the right (liberal) behaviour for flat schemas, but nested structures don't get parsed into the Reference
fields:
import typesystem
class MyNestedSchema(typesystem.Schema):
name = typesystem.String()
class MySchema(typesystem.Schema):
inner = typesystem.Reference(to=MyNestedSchema)
data = {'nested': {'name': 'My Name'}}
MySchema.validate(data)
# MySchema(nested=MyNestedSchema(name='My Name'))
MySchema(data)
# MySchema(nested={'name': 'My Name'}) # expecting same as previous
Could you explain the reasoning behind the current behaviour of the Schema.__init__
method when passed a single arg
?
Would it be possible to instantiate Reference
fields in this way? E.g. the current:
for key in self.fields.keys():
if key in item:
setattr(self, key, item[key])
could become
for key, schema in self.fields.items():
if key in item:
if isinstance(schema, Reference):
setattr(self, key, schema(item[key]))
else:
setattr(self, key, item[key])
What do you think about supporting UUIDs? I guess they are somewhat common when dealing with database objects.
String
field could be extended to support it and the eventual API could be:
typesystem.String(format="uuid")
I have noticed that String have the enum argument in apistar's typesystem.
So we can use String as follows
size = validators.String(enum=['small', 'medium', 'large'])
What's the advantage of deleting this parameter?
We should support the following:
precision="minute"
precision="second"
precision="millisecond"
precision="microsecond"
precision="0.01"
# Decimal.quantize styleEquivalent to Array
with items
as a list, no max_items
, min_items
, additional_items
, exact_items
.
Either a = Tuple(items=[Integer(), String()])
, or a = Tuple(items=2)
Possibly allow returning a named tuple.
Support Object(properties=Integer())
style.
Should assert that additional_properties
is None, since it makes no sense in that context.
Might need to rename the instance attributes, eg. self.named_properties
, self.pattern_properties
, self.additional_properties
.
the LICENSE.md
file is missing, but linked to from the README.
I haven't found this to be documented. Initially I assumed that deserialized JSON Schema is equivalent to the coded one, but it seems not.
I cannot use result of typesystem.from_json_schema(schema)
to call forms.Form()
as from_json_schema returns only a validator, not the Schema class.
Steps to reproduce:
As consequence of this bug, Forms example doesn't work (https://www.encode.io/typesystem/forms/):
File "example.py", line 92, in exec_example
print(form)
File "/usr/local/lib/python3.7/site-packages/typesystem/forms.py", line 106, in __str__
return self.render_fields()
File "/usr/local/lib/python3.7/site-packages/typesystem/forms.py", line 52, in render_fields
field_name=field_name, field=field, value=value, error=error
File "/usr/local/lib/python3.7/site-packages/typesystem/forms.py", line 71, in render_field
template = self.env.get_template(template_name)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 830, in get_template
return self._load_template(name, self.make_globals(globals))
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 804, in _load_template
template = self.loader.load(self, name, globals)
File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 113, in load
source, filename, uptodate = self.get_source(environment, name)
File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 235, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: forms/input.html
Workaround: get 'templates' directory from this repository, put it into site-packages/typesystem (as PackageLoader of Jinja2 is used by default).
string fields can have a pattern
for validation, and since re.search()
is used under the hood, the pattern can be either a regular expression as a string or a compiled regular expression (re.compile(...)
is cached and can have flags).
so far so good, but the error message uses this:
"pattern": "Must match the pattern /{pattern}/.",
the str()
conversion for compiled regexes looks like code:
>>> my_regex = re.compile('foo')
>>> f"oops: {my_regex}"
"oops: re.compile('foo')"
but the .pattern
attribute is a string:
>>> f"oops: {my_regex.pattern}"
'oops: foo'
so, perhaps the error message for regular expressions could be improved. 🌳
Eg:
required
should not just hardcode-overwrite the error text.invalid_key
should reference the key, not the dict.invalid_property
should reference the key, not the value.Is there a way to create schemes with dynamic Choice fields? E.g. I want to render a form where the user is supposed to select from his existing objects. The list of these objects must be retrieved from the database on each request.
Even adding some sort of a callable (similar to defaults) won't help because I need to pass a context for the database query somehow...
i admit i do not understand completely what's going on, but the Field._creation_counter
seems like a hack to me, and i wonder why it's necessary?
since python 3.6, dictionaries retain insertion order which, iiuc, means the attrs
passed to __new__
are in source code order already.
why not assign all fields to __schematype_fields__
instead and use that when ordering is needed?
while looking at it, parent schema fields are included in child schema, but does schematype
handle overrides of fields correctly at all? the attrs
project has some code that may be relevant to look at here, since it deals with similar problems: https://github.com/python-attrs/attrs/blob/4fe28966e88b9b85c9c2df77ffb34f70175c4492/src/attr/_make.py#L351-L362
For rendering the initial blank option in an HTML <select>
What do you think about this kind of field?
I was thinking about expanding String type as a first option:
typesystem.String(format='password')
or adding new kind of field typesystem.Password
Both options will ensure to create "password" type of input with hidden value.
The latter would have some additional options of allowed characters (password policy), ie.:
password = typesystem.Password(
min_length: int,
require_digit: boolean,
require_lowercase: boolean,
require_uppercase: boolean,
require_non_alphanumeric: boolean
)
Consider the following code:
import traceback
import typesystem
try:
typesystem.Integer().validate("hello")
except typesystem.ValidationError:
traceback.print_exc()
Because of an issue with Python 3.6 discussed in issue 28803, since ValidationError
implements __eq__
but not __hash__
it is not hashable and print_exc()
fails to process it correctly.
TypeError: unhashable type: 'ValidationError'
(Full traceback available here)
Looks like we'd need to implement ValidationError.__hash__()
to fix this.
Schema.__repr__
is the most important case here.
Also important: ValidationResult
, ValidationError
, ErrorMessage
typesystem/typesystem/schemas.py
Line 175 in 7e8f1a8
It seems to me that the dictionary key and attribute lookups occur twice. I believe the following code will not have any side effects and avoids the double lookups.
def __getitem__(self, key: typing.Any) -> typing.Any:
try:
field = self.fields[key]
value = getattr(self, key)
except (KeyError, AttributeError):
raise KeyError(key) from None
else:
return field.serialize(value)
Overall, the code looks great! (I love type hinting in conjunction with PEP8)
Is there any reason you don't use is
for Boolean comparison? You seem so precise and strict elsewhere in your code that this seems to be a bit of an outlier (to me).
Thanks for your efforts! I am looking forward to putting Starlette and Typesystem through some testing and perhaps using it for our project at work (replacing both Django + Channels and Flask + Websockets in our existing code base).
class User(typesystem.Schema):
...
class Org(typesystem.Schema):
owner = typesystem.Nested(User)
...
Should also support lazy import strings, for recursive definitions.
owner = typesystem.Nested("myproject.schemas.User")
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.