encode / typesystem Goto Github PK

I've copy-pasted "Tokenized Errors" example from documentation to example.py:

$ cat example.py 

import typesystem

class Config(typesystem.Schema):
    num_worker_processes = typesystem.Integer()
    enable_auto_reload = typesystem.Boolean()

text = '''{
    "num_worker_processes": "x",
    "enable_auto_reload": "true"
}'''

value, messages = typesystem.validate_json(text, validator=Config)

assert value is None
for message in messages:
    line_no = message.start_position.line_no
    column_no = message.start_position.column_no
    print(f"Error {message.text!r} at line {line_no}, column {column_no}.")
# Error 'Must be a number.' at line 2, column 29.

However, it fails with ValidationError. I expected it to print 'Must be a number ...' instead:

$ pipenv install typesystem
$ pipenv run python3 -c 'import typesystem; print(typesystem.__version__)'
0.2.2
$ pipenv run python3 example.py 
Traceback (most recent call last):
  File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/tokenize/positional_validation.py", line 13, in validate_with_positions
    return validator.validate(token.value)
  File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/schemas.py", line 147, in validate
    value = validator.validate(value, strict=strict)
  File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/fields.py", line 545, in validate
    raise ValidationError(messages=error_messages)
typesystem.base.ValidationError: {'num_worker_processes': 'Must be a number.'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "example.py", line 12, in <module>
    value, messages = typesystem.validate_json(text, validator=Config)
  File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/tokenize/tokenize_json.py", line 197, in validate_json
    return validate_with_positions(token=token, validator=validator)
  File "/Users/me/.local/share/virtualenvs/example-AxLzGwo/lib/python3.7/site-packages/typesystem/tokenize/positional_validation.py", line 36, in validate_with_positions
    raise ValidationError(messages=messages)
typesystem.base.ValidationError: {'num_worker_processes': 'Must be a number.'}

Currently unimplemented. See https://github.com/encode/typesystem/blob/master/typesystem/formats.py#L128 and https://github.com/encode/typesystem/blob/master/typesystem/formats.py#L87

Needs:

• Uncommenting out of both those two cases.
• Test coverage.

Eg.

class Something(typesystem.Schema):
    created = typesystem.DateTime(default=datetime.datetime.now)
    ...

What do you think about this kind of field?
I was thinking about expanding String type as a first option:

typesystem.String(format='password')

or adding new kind of field typesystem.Password

Both options will ensure to create "password" type of input with hidden value.
The latter would have some additional options of allowed characters (password policy), ie.:

password = typesystem.Password(
    min_length: int,
    require_digit: boolean,
    require_lowercase: boolean,
    require_uppercase: boolean,
    require_non_alphanumeric: boolean
)

Would it be possible for the Choice class to include a list of acceptable choices in the validation error output?

Here is a snippet doing this using subclassing.

import typesystem

class MyChoice(typesystem.fields.Choice):
    
    def __init__(self, *, choices=None, **kwargs):
        super().__init__(choices=choices, **kwargs)
        self.single_choices = [x[0] for x in self.choices]
        self.errors["choice"] = "Not a valid choice. Please choose: {single_choices}"
    
field = MyChoice(choices=[('foo', 'foo')])

field.validate('bar')

This will raise the following error message (this is an ipython notebook error message, but hopefully demonstrates ok):

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
<ipython-input-65-854f15494f03> in <module>
      8 field = MyChoice(choices=[('foo', 'foo')])
      9 
---> 10 field.validate('bar')

~/Documents/entanglement/venv/lib/python3.7/site-packages/typesystem/fields.py in validate(self, value, strict)
    384                     return None
    385                 raise self.validation_error("required")
--> 386             raise self.validation_error("choice")
    387         return value
    388 

ValidationError: Not a valid choice. Please choose: ['foo']

I defind a Schema which contains Array field

class Node(Schema):
    name = String()
    cams = Array(items=String(), min_items=0, max_items=4)

when I use dict(node) after validated, It raised error:

attribute of type 'list' is not callable

help

E.g. allow back-references or cyclical references.

We could register schema class names whenever Schema is subclassed, which would allow us to do this:

artist = typesystem.Nested('Artist')

A lazy import string would also work:

artist = typesystem.Nested('myproject.schemas:Artist')

string fields can have a pattern for validation, and since re.search() is used under the hood, the pattern can be either a regular expression as a string or a compiled regular expression (re.compile(...) is cached and can have flags).

so far so good, but the error message uses this:

        "pattern": "Must match the pattern /{pattern}/.",

the str() conversion for compiled regexes looks like code:

>>> my_regex = re.compile('foo')

>>> f"oops: {my_regex}"
"oops: re.compile('foo')"

but the .pattern attribute is a string:

>>> f"oops: {my_regex.pattern}"
'oops: foo'

so, perhaps the error message for regular expressions could be improved. 🌳

Similar to List(unique=True), except that uniqueness should only be a failure case if strict=True

• We'll want to mark up our error messages as translatable strings.
• Support an errors = {} argument for overriding the error strings.
• Support errors as an attribute adding the the defaults, rather than overriding them.

Consider the following code:

import traceback
import typesystem

try:
    typesystem.Integer().validate("hello")
except typesystem.ValidationError:
    traceback.print_exc()

Because of an issue with Python 3.6 discussed in issue 28803, since ValidationError implements __eq__ but not __hash__ it is not hashable and print_exc() fails to process it correctly.

TypeError: unhashable type: 'ValidationError'

(Full traceback available here)

Looks like we'd need to implement ValidationError.__hash__() to fix this.

Just needs a tweak in Form.render_field to set value="", and a test case.

I haven't found this to be documented. Initially I assumed that deserialized JSON Schema is equivalent to the coded one, but it seems not.

I cannot use result of typesystem.from_json_schema(schema) to call forms.Form() as from_json_schema returns only a validator, not the Schema class.

The documentation indicates that the DateTime, Date, and Time fields return the appropriate datetime object instances. However, the validate methods actually return strings instead.

Add an Array field.

Schema.__repr__ is the most important case here.
Also important: ValidationResult, ValidationError, ErrorMessage

Support Object(properties=Integer()) style.
Should assert that additional_properties is None, since it makes no sense in that context.
Might need to rename the instance attributes, eg. self.named_properties, self.pattern_properties, self.additional_properties.

Display as a <select multiple>

you may want to consider a ‘squash only’ setting for this project (repo setting in github). history like this:

is a pain to look at, and not useful at all for debugging (e.g. git bisect). all that noise in small commits after a review + the merges make the main changes invisible.

Is it possible to mark some field as optional? What I mean:

property = typesystem.String(max_length=100, optional=True)

This would mean that property field may be ommited, but if present it must be String with maximum length equal 100 chars.

• Partial serialization.
• Use schema class to serialize other instances.

@tomchristie Are you open to support @dataclass-decorated classes in typesystem? Similar to what attrs project is doing with @attr.s(auto_attribs=True). I'd like to try contributing the support, but wanted to check if it's within the scope of the project.

Here is how some of the examples from typesystem documentation could be rewritten, if dataclasses were supported:

from dataclasses import dataclass
from datetime import date
from typesystem import field, validate

# Example from README.md

@dataclass
class Artist:
    name: str = field(max_length=100)

@dataclass
class Album:
    title: str = field(max_length=100)
    release_date: date
    artist: Artist

validate(Album, {
    "title": "Double Negative",
    "release_date": "2018-09-14",
    "artist": {"name": "Low"}
})

# Example from documentation

@dataclass
class Organisation:
    name: str = field(title="Name", max_length=100)
    date_created: date = field(title="Date created", default=datetime.date.today)
    owner: User = field(allow_null=True)

typesystem.field could be a thin wrapper around around dataclasses.field, using metadata argument. For example, Artist and Organisation above are the same as this:

@dataclass
class User:
    name: str = dataclasses.field(metadata={'typesystem': {'max_length': 100}})

@dataclass
class Organisation:
    name: str = dataclasses.field(
        metadata={'typesystem': {'title': 'Name', 'max_length': 100}},
    )
    date_created: date = dataclasses.field(
       default_factory=datetime.date,
       metadata={'typesystem': {'title': 'Date created'}},
    )
    owner: User = dataclasses.field(
        metadata={'typesystem': {'allow_null': True}},
    )

Add trim=True argument to the String field.

Database validation needs an async/await interface. ValidationResult should be an awaitable, and should error if __bool__ or __iter__ are called, and it has awaitable validators, and has not yet been awaited.

hey,

i'm wondering what made you decide to use a classmethod for Schema.validate() which then returns an instance of the schema.

i am wondering how this would work nicely when you're actually interested in creating custom (application specific) class instances, such as sqlalchemy orm class instances (‘database models’), since all attributes need to be copied afterwards.

also, to me ‘an instance of a schema’ is not the obvious answer for the question ‘what's the result of my validation’, yet typesystem seems to think that is fine.

just curious to hear your thoughts. thanks in advance!

the LICENSE.md file is missing, but linked to from the README.

Is there a way to create schemes with dynamic Choice fields? E.g. I want to render a form where the user is supposed to select from his existing objects. The list of these objects must be retrieved from the database on each request.

Even adding some sort of a callable (similar to defaults) won't help because I need to pass a context for the database query somehow...

We should support the following:

• precision="minute"
• precision="second"
• precision="millisecond"
• precision="microsecond"
• precision="0.01" # Decimal.quantize style

It seems to me that the dictionary key and attribute lookups occur twice. I believe the following code will not have any side effects and avoids the double lookups.

    def __getitem__(self, key: typing.Any) -> typing.Any:
        try:
            field = self.fields[key]
            value = getattr(self, key)
        except (KeyError, AttributeError):
            raise KeyError(key) from None
        else:
            return field.serialize(value)

Overall, the code looks great! (I love type hinting in conjunction with PEP8)

Is there any reason you don't use is for Boolean comparison? You seem so precise and strict elsewhere in your code that this seems to be a bit of an outlier (to me).

Thanks for your efforts! I am looking forward to putting Starlette and Typesystem through some testing and perhaps using it for our project at work (replacing both Django + Channels and Flask + Websockets in our existing code base).

• required should not just hardcode-overwrite the error text.
• invalid_key should reference the key, not the dict.
• invalid_property should reference the key, not the value.

class User(typesystem.Schema):
    ...

class Org(typesystem.Schema):
    owner = typesystem.Nested(User)
    ...

Should also support lazy import strings, for recursive definitions.

owner = typesystem.Nested("myproject.schemas.User")

Hey,

I was fiddling with composite data types, in particular with Reference to implement nested schemas, and there seems to be an issue with how arrays and objects of references are serialized when calling dict() on the parent class.

Consider the following schemas:

import typesystem as ts

class Child(ts.Schema):
    pass

class ParentA(ts.Schema):
    child = ts.Reference(to=Child)

class ParentB(ts.Schema):
    children = ts.Array(ts.Reference(Child))

class ParentC(ts.Schema):
    children = ts.Object(properties=ts.Reference(Child))

Test cases with what should happen IMO:

pa = ParentA(child=Child())
assert dict(pa) == {"child": {}}  # OK

pb = ParentB(children=[Child()])
assert dict(pb) == {"children": [{}]}  # FAIL
# {"children": [Child()]}

pc = ParentC(children=[Child()])
assert dict(pc) == {"children": {"john": {}}}  # FAIL
# {"children": {"john": Child()}}

This causes issues when passing the result of dict(pb) or dict(pc) to, say, json.dumps(), because it gets Child instances which are not JSON-serializable.

Am I simply abusing Array and Object? Is there another way of implementing composite fields with nested schemas?

What do you think about supporting UUIDs? I guess they are somewhat common when dealing with database objects.

String field could be extended to support it and the eventual API could be:

typesystem.String(format="uuid")

would TypeSystem be able to handle CSRF token so the form could be validated w/o risk ?

I have noticed that String have the enum argument in apistar's typesystem.
So we can use String as follows

size = validators.String(enum=['small', 'medium', 'large'])

What's the advantage of deleting this parameter?

i admit i do not understand completely what's going on, but the Field._creation_counter seems like a hack to me, and i wonder why it's necessary?

since python 3.6, dictionaries retain insertion order which, iiuc, means the attrs passed to __new__ are in source code order already.

why not assign all fields to __schematype_fields__ instead and use that when ordering is needed?

while looking at it, parent schema fields are included in child schema, but does schematype handle overrides of fields correctly at all? the attrs project has some code that may be relevant to look at here, since it deals with similar problems: https://github.com/python-attrs/attrs/blob/4fe28966e88b9b85c9c2df77ffb34f70175c4492/src/attr/_make.py#L351-L362

For rendering the initial blank option in an HTML <select>

Dear @tomchristie,

I'm in the process of adding a feedstock for typesystem to conda-forge: conda-forge/staged-recipes#8008 in order to be able to have the latest versions of apistar available there as well.

Please let me know if that's okay with you :) and thank you very much for those awesome packages!

I'm using typesystem to implement a client for an HTTP API, and it's almost perfect for the jobs of constructing payloads to send and parsing payloads that have been received.

The API in question includes nested structures, so I'd like to be able to feed response.json() straight into a schema to get back what's effectively a data class. Using MySchema.validate(payload) works, but it makes the library fragile to changes in the API, like extra options being added to Choice fields that would then cause validation errors on all responses.
MySchema(payload) has the right (liberal) behaviour for flat schemas, but nested structures don't get parsed into the Reference fields:

import typesystem

class MyNestedSchema(typesystem.Schema):
    name = typesystem.String()

class MySchema(typesystem.Schema):
    inner = typesystem.Reference(to=MyNestedSchema)

data = {'nested': {'name': 'My Name'}}

MySchema.validate(data)
# MySchema(nested=MyNestedSchema(name='My Name'))

MySchema(data)
# MySchema(nested={'name': 'My Name'})  # expecting same as previous

Could you explain the reasoning behind the current behaviour of the Schema.__init__ method when passed a single arg?
Would it be possible to instantiate Reference fields in this way? E.g. the current:

for key in self.fields.keys():
	if key in item:
		setattr(self, key, item[key])

could become

for key, schema in self.fields.items():
	if key in item:
		if isinstance(schema, Reference):
			setattr(self, key, schema(item[key]))
		else:
			setattr(self, key, item[key])

1. Support a validator, or list of validators:

date_of_travel = Date(validator=may_not_be_in_past)

2. Support validator methods:

@classmethod
def validate_date_of_travel(cls, value):
    ...

3. Support a validate_all or something similar.

Equivalent to Array with items as a list, no max_items, min_items, additional_items, exact_items.

Either a = Tuple(items=[Integer(), String()]), or a = Tuple(items=2)

Possibly allow returning a named tuple.

Autocompletion / intellisense is a nice-to-have feature. Currently neither PyCharm nor VScode offer any autocompletion for typesystem schema classes. This may lessen the likelihood of its usage and adoption.

It would be nice if a script was included with typesystem that could be configured to be called automatically on file save and that auto-generates/updates a .pyi stub file for classes subclassing typesystem.Schema.

Auto-generated stub code could be surrounded by special markers in python comments that serve double duty.

1. warn developers that modification within could be overwritten without warning
2. simple start / end line number identification (don't overwrite user additions to .pyi files)

Including typesystem schema classes along with other code would result in .pyi files that didn't include stubs for the other code. Some possible options:

1. don't mix your typesystem schemas with other code
2. manually write stubs for the other code
3. use mypy stub generator
4. use make-stub-files

What do you think? Comments?

Eg:

1. Use a schema class, but just instantiate a plain dict.
2. Use a schema class, but instantiate a different type. (Eg. seperate schema from model class)

Steps to reproduce:

• download typesystem-0.2.2.tar.gz from https://pypi.org/project/typesystem/
• extract archive
• observe that 'templates' directory is missing

As consequence of this bug, Forms example doesn't work (https://www.encode.io/typesystem/forms/):

   File "example.py", line 92, in exec_example
     print(form)
   File "/usr/local/lib/python3.7/site-packages/typesystem/forms.py", line 106, in __str__
     return self.render_fields()
   File "/usr/local/lib/python3.7/site-packages/typesystem/forms.py", line 52, in render_fields
     field_name=field_name, field=field, value=value, error=error
   File "/usr/local/lib/python3.7/site-packages/typesystem/forms.py", line 71, in render_field
     template = self.env.get_template(template_name)
   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 830, in get_template
     return self._load_template(name, self.make_globals(globals))
   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 804, in _load_template
     template = self.loader.load(self, name, globals)
   File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 113, in load
     source, filename, uptodate = self.get_source(environment, name)
   File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 235, in get_source
     raise TemplateNotFound(template)
 jinja2.exceptions.TemplateNotFound: forms/input.html

Workaround: get 'templates' directory from this repository, put it into site-packages/typesystem (as PackageLoader of Jinja2 is used by default).