seandstewart / typical Goto Github PK
View Code? Open in Web Editor NEWTypical: Fast, simple, & correct data-validation using Python 3 typing.
Home Page: https://python-typical.org
License: MIT License
Typical: Fast, simple, & correct data-validation using Python 3 typing.
Home Page: https://python-typical.org
License: MIT License
Constraints generated for typed classes are incorrectly loose, allowing for extra fields. These should not be allowed when applying validation, even if we do allow them when transmuting inputs.
Since DirectoryPath is a subclass of Path, should it be possible to translate from Path to DirectoryPath if Path is a directory?
typic.translate(pathlib.Path.cwd(), typic.DirectoryPath)
I now get a the following error:
TranslatorTypeError: Cannot translate to type <class 'typic.types.path.DirectoryPath'>. Unable to determine target fields.
A common pattern for polymorphic data in statically typed languages is the use of a discriminator
field on the object which can be used to determine which type to deserialize said data into. This is main use-case for users registering custom type deserializers in typical
. A better route would be to expose a method at time of declaration by which the user my declare a discriminator.
from typing import ClassVar, Union
import typic
@typic.klass
class Something:
type: ClassVar[str] = "something"
@typic.klass
class Else:
type: ClassVar[str] = "else"
discriminator = typic.discriminator(field="type", mapping={"something": Something, "else": Else})
@typic.klass
class Poly:
morphic: Union[Something, Else] = typic.field(discriminator=discriminator)
This relates to #56
It seams that typical
doesn't support some types ini collections
e.g. defaultdict.
import typic
@typic.klass
class Foo:
s: typing.DefaultDict[str, int]
# ValueError: no signature found for builtin type <class 'collections.defaultdict'>
Subclass of builtin types like set
, list
, without __init__
, will raise the same error:
import typic
class MySet(set):
...
@typic.klass
class Foo:
s: MySet
# ValueError: no signature found for builtin type <class '__main__.MySet'>
Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
Hey, @seandstewart . I have another feature request.
For data model, there is default_factory
which can be used to set dynamic default value. For function argument, if set the default value as a instance of typic.Field
, it won't be validated as a default value. datetime.now
as a default_factory is quite common. Every time I have to handle it like this:
from datetime import datetime
import typic
@typic.al
def foo(dt: datetime = None):
dt = datetime.now() if dt is None else dt
...
Especilly when we have many functions with dynamic default value, it would be a problem. Currently, beacuse the default value would not be validated, I defined a Helper class to solve this.
from typing import Callable, Optional, Any
from functools import wraps
import inspect
from dataclasses import MISSING
import typic
class DefaultArgument:
def __init__(self, default: Any = MISSING, *, factory: Optional[Callable] = None):
assert default is not MISSING or callable(factory)
self.default = default
self.factory = factory
def __call__(self, *args, **kwargs):
if self.default is not MISSING:
return self.default
return self.factory(*args, **kwargs)
def __repr__(self):
return f"{self.__class__.__name__}(default={self.default}, factory={self.factory!r})"
def validate(func: Optional[Callable] = None, *, coerce: bool = True, strict: bool = False) -> Callable:
def _validate(_func: Callable):
@wraps(func)
def wrapped(*args, **kwargs):
bind = inspect.signature(_func).bind(*args, **kwargs)
bind.apply_defaults() # apply_defaults is not available in typic.bind
for k, v in bind.arguments.items():
if isinstance(v, DefaultArgument):
bind.arguments[k] = v()
return typic.bind(_func, *bind.args, partial=False, coerce=coerce, strict=strict, **bind.kwargs).eval()
return wrapped
if callable(func):
return _validate(func)
else:
return _validate
Now we can redefine foo
like this:
from datetime import datetime
@validate
def foo(dt: datetime = DefaultArgument(factory=datetime.now)):
...
By the way, what if the user want to use the key words such as dict
, schema
, coerce
, strict
and so on, as the function argument or data attribute?
2.0.12
(and I verified the bug exists in current master.)I tried using typic.al
and typic.klass
with the type annotation Mapping[Path, Path]
.
This failed with the error TypeError: sequence item 1: expected str instance, PosixPath found
-- full backtrace at the very end, not that it matters, along with a little other debug output.
The goal was to have strict runtime validation; the decoration reflects the actual type passed. If it matters, I wanted runtime validation to catch type punning errors like passing a str
where a List[str]
was expected. The code in question mostly runs external applications, and operates on very small datasets, so the performance cost was not a concern.
#!/usr/bin/env python3
from typing import Mapping
import typic
@typic.al(strict=True)
def bad(val: Mapping[int, str]) -> None:
pass
bad({1: 1})
The validation code tries to use the raw key in a call to str.join
, which fails. Comes from the code in MappingConstraints._set_item_validator_keys_values_line()
in typic/constraints/mapping.py
The code generated for describing the invalid mapping value is:
field = f"'.'.join(({self.VALTNAME}, {self.X}))"
The correct version is probably:
# too horrible for words, really...
field = f"f'{{{self.VALTNAME}}}[{{repr({self.X})}}]'"
# nice version that someone could actually read...
field = ''.join(("f'{", self.VALTNAME, "}[{repr(", self.X, ")}]'"))
That fixes the bug in two ways, first by not using an almost-Any
as an argument to str.join
where a str
is expected, but second by using repr
to get a python-ish version of the value, not a string, which is kind of confusing.
Regardless, you could also fix it in the current model with:
field = f"'.'.join(({self.VALTNAME}, repr({self.X})))"
FWIW, I'd very strongly argue that, at least for Mapping
and Dict
, using the .
dot separator is wrong. That really belongs to class attributes, and that isn't the right context here. Given:
@dataclass
class MyDataclass:
MyDictField: Dict[str, str]
MyDataclass(MyDictField={'hello': 12}
The validation output should probably be something like
MyDataclass.MyDictField['hello'] was int, expected str
Rather than:
MyDataclass.MyDictField.hello was int, expected str
] python3 tmp/demo.py
bad: {1: PosixPath('/Users/slippycheeze')}
sequence item 1: expected str instance, int found
/Users/slippycheeze
ugly: {PosixPath('/Users/slippycheeze'): PosixPath('/Users/slippycheeze')}
Given value <PosixPath('/Users/slippycheeze')> fails constraints: (type='int', nullable=False, coerce=False)
/Users/slippycheeze
worse: {(1, 2): True}
sequence item 1: expected str instance, tuple found
True
#!/usr/bin/env python3
from typing import Any, Dict, Mapping, Tuple
from pathlib import Path
import typic
@typic.al(strict=True)
def bad(val: Mapping[Path, Path]) -> None:
pass
@typic.al(strict=True)
def ugly(val: Dict[int, Path]) -> None:
pass
@typic.al(strict=True)
def worse(val: Dict[Tuple, Any]) -> None:
pass
# In each case the demo is that the key *is* valid, and is not simply an
# identity comparison. Not that this is necessary, but y'know, thorough.
try:
data = {1: Path.home()}
print("bad:", repr(data))
bad(data)
except Exception as e:
print(e)
print(repr(data[1]))
try:
data = {Path.home(): Path.home()}
print("ugly:", repr(data))
ugly(data)
except Exception as e:
print(e)
print((data[Path.home()]))
try:
data = {(1,2,): True}
print("worse:", repr(data))
worse(data)
except Exception as e:
print(e)
print(repr(data[(1,2,)]))
[...elided the uninteresting parts]
File "/Users/slippycheeze/share/fonts/conform.py", line 83, in task_fonts
CopyFiles(copies)
File "<string>", line 3, in __init__
File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/api.py", line 220, in __setattr_typed__
self, name, __trans[name](item) if name in protos else item,
File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/constraints/common.py", line 103, in validate
valid, value = self.validator(value)
File "<typical generated validator_4371568336>", line 9, in validator_4371568336
valid, val = validator_4371568336_item_validator(val, addtl)
File "<typical generated validator_4371568336_item_validator>", line 7, in validator_4371568336_item_validator
retx, rety = validator_4371568336_item_validator_keys_validator(x), validator_4371568336_item_validator_vals_validator(y, field='.'.join((valtname, x)))
] python3 tmp/demo.py
> <typical generated validator_4460259152_item_validator>(7)validator_4460259152_item_validator()
-> retx, rety = validator_4460259152_item_validator_keys_validator(x), validator_4460259152_item_validator_vals_validator(y, field='.'.join((valtname, x)))
(Pdb) p validator_4460259152_item_validator_vals_validator(y)
PosixPath('/Users/slippycheeze/b')
(Pdb) p '.'.join((valtname, x)
*** SyntaxError: unexpected EOF while parsing
(Pdb) p '.'.join((valtname, x))
*** TypeError: sequence item 1: expected str instance, PosixPath found
(Pdb) p valtname
'Mapping'
(Pdb) p x
PosixPath('/Users/slippycheeze/a')
(Pdb) p validator_4460259152_item_validator_keys_validator(x)
PosixPath('/Users/slippycheeze/a')
(Pdb) p validator_4460259152_item_validator_keys_validator
<bound method __AbstractConstraints.validate of (type='Path', nullable=False)>
(Pdb) p '.'.join((valtname, str(x)))
'Mapping./Users/slippycheeze/a'
NetAddrInfo is a Frozen dataclass with slots, which is currently broken for deepcopy: PR in cPython
So I played around a bit with the new support for TypedDict schema generation and noticed a few things:
>>> class G(TypedDict, total=False):
... a: str
...
>>> typic.schema(G)
ObjectSchemaField(title='G', properties={'a': StrSchemaField()}, additionalProperties=False, required=('a',))
Since the TypedDict is not total, the field a
is not required to be present, so shouldn't be required
in the JSON schema.
Also, somewhat related: Making a field Optional
seems to make it not required even though it doesn't have a default value. But Optional
just means that the value can be None
, not that it could be missing.
>>> class G(TypedDict, total=True):
... a: Optional[int]
...
>>> typic.schema(G)
ObjectSchemaField(title='G', properties={'a': IntSchemaField()}, additionalProperties=False)
Compiling down to Cython at build time could provide some quick wins on performance if the effort isn't too great.
Hello! Your project is awesome!
I think we should consider to add dataclass_factory by @Tishka17 to benchmarks because it's fast too.
We should add JSON Schema defined types and formats.
I suspect this may be a known issue, caused by having more than one plausible conversion applied to the union type, but:
from pathlib import Path
from typing import *
import typic
good = [str, Path, Union[str], Union[Path]]
for types in good:
typic.validate(this_type, Path.home())
# success, whatever you homedir is, like:
WindowsPath('c:/Users/me')
typic.validate(Union[str, Path], Path.home())
# fails:
ConstraintValueError: Given value <WindowsPath('C:/Users/me')> fails constraints: (constraints=((type=str, nullable=False, coerce=False), (type=Path, nullable=False)), nullable=False)
# also fails, identically other than the order of output
typic.validate(Union[Path, str], Path.home())
This confusing type union came from my discovering that I don't actually want strict mode, I want the type coercion to happen, and removing the strict=True
from various things. The original type annotation was Union[str, Path, PosixPath]
because I found this on macOS, and worked around it blindly thinking it was odd, but strict=True
required it.
Now I'm pretty sure it is a ... well, not ideal, but probably an issue picking which type to coerce into or something?
This might still be related to MRO
:
import typic
@typic.klass
class Base:
...
@typic.klass
class Foo(Base):
foo: str
def __setattr__(self, k, v):
# print('__setattr__')
super().__setattr__(k, v)
foo = Foo(foo='foo') # RecursionError
Translation between higher-order objects isn't behaving properly when signatures don't exactly align.
Additionally, calling .transmute()
on data with unknown fields fails, but should succeed.
import typic
from typing import Optional
@typic.klass
class Source:
test: Optional[str] = typic.field(init=False)
field_to_ignore: str = "Ignore me"
def __post_init__(self):
self.test = "Something"
@typic.klass
class Dest:
test: Optional[str] = None
Dest.transmute(Source())
# => throws "Source is missing fields: ('test',)"
Dest.transmute(Source().primitive())
# => throws "TypeError: __init__() got an unexpected keyword argument 'field_to_ignore'"
According to the docs:
For instance, pydantic will coerce a float to an int, but will raise an error if a str is passed to a field marked as datetime.
This is not true:
import datetime
import pydantic
class Foo(pydantic.BaseModel):
a_date: datetime.date
a_datetime: datetime.datetime
a = Foo(a_date="2019-01-01", a_datetime="2019-01-01 17:31:31")
print(type(a.a_date))
print(type(a.a_datetime))
You might want to rephrase that
pandas.Timestamp
is a sub-class of datetime
, but as an annotation will cause TypeError:
import typic
from datetime import datetime
import pandas as pd
issubclass(pd.Timestamp, datetime) # True
@typic.klass
class Foo:
dt: pd.Timestamp
# TypeError: __init__() got an unexpected keyword argument 'additionalProperties'
This is OK at version 2.0.5
.
Personally, I think pandas.Timestamp
is more convenient than builtin datetime
in data analysis.
I didn't expect that None
would be coerced to a string for a field marked as str
. Can that behavior be disabled?
>>> @typic.al
... @dataclass
... class Y:
... a: str
...
>>> Y(None)
Y(a='None')
The same behavior happens if you try to coerce other types:
>>> Y(dict)
Y(a="<class 'dict'>")
Heyo! I very like your library! But when i saw the docs, i found that typical
uses python dataclasses
from standart library which, of course, will decrease performance because they are slow. What do you think about reinventing it for more performance? :)
Currently, the largest bottle-neck for coercing callable inputs is the inspect.Signature object and binding inputs to the function args, which we need to do manually so that we can coerce to the appropriate type.
In my testing, running a simple function with one or two args is up to 100x slower as a result of using that object.
Would like to see a feature to implement custom validations that could span multiple fields. For example:
Let's say you had a vacation start date and vacation end date field. Vacation start date should be before vacation end date.
Did not expected this. Have I misunderstood something?
typic.validate(int, 'a')
#> 'a'
Hi again,
I should probably soon try to make some pr, but take some time to understand the structure of typical and how things is put together. But until then here are an observation of a something that do not behave as expected.
import typic
import pendulum
from datetime import datetime
import pytz
dt = datetime(2020,5,24,tzinfo=pytz.UTC)
I would expect that,
x = typic.transmute(pendulum.DateTime, dt)
x
#> DateTime(2020, 5, 24, 0, 0, 0, tzinfo=UTC)
give same the same as,
y = pendulum.instance(dt)
y
#> DateTime(2020, 5, 24, 0, 0, 0, tzinfo=Timezone('UTC'))
but as you can see timezone type is diffrent
type(x.tzinfo)
#> pytz.UTC
type(y.tzinfo)
#> pendulum.tz.timezone.FixedTimezone
What do you think?
Cool library. It's nice to have a competitor to pydantic which seems a bit too loose for my taste. I was hoping I might be able to use it to:
(not sure if this is one of the goals of the project to make this work, but it seems like it fits in with the philosophy of this library as being unopinionated)
I tried the both, and they both fail. The latter fails with:
>>> from typing_extensions import TypedDict
>>> class F(TypedDict):
... a: str
... b: int
...
>>> typic.schema(F)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 868, in schema
annotation = self.resolve(obj)
File "/.../venv/lib/python3.7/site-packages/typic/util.py", line 286, in _cached_method_wrapper
result = func(*args, **kwargs)
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 500, in resolve
use, default=parameter.default, is_optional=is_optional
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 430, in get_coercer
return self._build_coercer(annotation, default=default, is_optional=is_optional)
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 402, in _build_coercer
self._build_mapping_coercer(func, args, anno_name)
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 316, in _build_mapping_coercer
key_type, item_type = args
ValueError: not enough values to unpack (expected 2, got 0)
# Same happens when it's wrapped in a dataclass
>>> from dataclasses import dataclass
>>> @typic.al
... @dataclass
... class Wrapper:
... foo: str
... bar: F
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/.../venv/lib/python3.7/site-packages/typic/api.py", line 184, in typed
return _typed(_cls_or_callable) if _cls_or_callable is not None else _typed
File "/.../venv/lib/python3.7/site-packages/typic/api.py", line 176, in _typed
return wrap_cls(obj, delay=delay) # type: ignore
File "/.../venv/lib/python3.7/site-packages/typic/api.py", line 116, in wrap_cls
coerce.annotations(klass)
File "/.../venv/lib/python3.7/site-packages/typic/util.py", line 327, in _fast_cached_method_wrapper
return memoget(arg)
File "/.../venv/lib/python3.7/site-packages/typic/util.py", line 315, in __missing__
self[key] = ret = func(instance, key)
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 606, in annotations
annotation, parameter=param, name=name, constraints=constraints
File "/.../venv/lib/python3.7/site-packages/typic/util.py", line 286, in _cached_method_wrapper
result = func(*args, **kwargs)
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 500, in resolve
use, default=parameter.default, is_optional=is_optional
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 430, in get_coercer
return self._build_coercer(annotation, default=default, is_optional=is_optional)
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 402, in _build_coercer
self._build_mapping_coercer(func, args, anno_name)
File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 316, in _build_mapping_coercer
key_type, item_type = args
ValueError: not enough values to unpack (expected 2, got 0)
2.0.12
Looks like optional mapping validation in non-strict mode fails to compile. The data being checked is irrelevant, problem happens during type compilation.
from typing import *
import typic
typic.validate(Optional[Mapping[str, str]], {})
Note: I'd expect the minimal repro to validate, but this is about the type validation compiler, not the validate
call itself -- that is just the easy way to trigger it. I ran into it in an annotated function where one argument was Optional[Mapping[Strict[str], str]]
, but cut it down to the above.
Optional[Mapping]
will succeed, so it seems to be an assumption in the key/value validation code for Mapping
that didn't expect the Union
type above it or something.
] python3 -c 'from typing import *; import typic; typic.validate(Optional[Mapping[str, str]], None)'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/resolver.py", line 120, in validate
resolved: SerdeProtocol = self.resolve(annotation)
File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/resolver.py", line 458, in resolve
deserializer, validator = self.des.factory(anno, constraints)
File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/des.py", line 527, in factory
deserializer = self._build_des(annotation)
File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/des.py", line 438, in _build_des
self._build_mapping_des(func, anno_name, annotation)
File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/des.py", line 324, in _build_mapping_des
if issubclass(annotation.origin, defaultdict):
TypeError: issubclass() arg 1 must be a class
We get the return annotation for free, so we should allow users to do something with it.
Related: #44
orjson
now supports serialization of subclassed.builtins, which potentially allows us to use this library (if available) for .tojson()
.
Allowing for the real-time streaming of the results of calls to primitive
could have huge performance gains when dumping an object directly to JSON. This should be investigated.
Not sure if it's possible, but it would be useful if you could get a list of e.g. ValueError
s, rather than just the first one that failed when there are multiple issues.
It would make typical
much more useful for validation purposes.
Perhaps it could be configured somehow which behavior you want, since there could be a performance hit, I suppose.
Example:
>>> @typic.al
... @dataclass
... class N:
... a: str
... b: int
... c: int
...
>>> N(a="a", b="a", c="b")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 3, in __init__
File "../typic/api.py", line 143, in __setattr_coerced__
value = ann[name](value) if name in ann else value
File "../typic/coercer.py", line 117, in coerce
return self.coercer(value)
File "<string>", line 3, in coerce__class_int___class_inspect__empty__False
ValueError: invalid literal for int() with base 10: 'a'
Iterable can be infinitely.
from typing import Iterable
import typic
def infinite_numbers():
i = 0
while 1:
yield i
i += 1
@typic.al
def foo(iterable: Iterable[int]):
print(iterable)
foo(range(5)) # [0, 1, 2, 3, 4]
foo(infinite_numbers()) # never stop
Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
transmute truncate everything after hours:
In: typic.transmute(pendulum.DateTime, pendulum.datetime(2020, 3, 23, 16, 27, 12))
Out: DateTime(2020, 3, 23, 0, 0, 0)
transmute do not preserve timezone:
In: typic.transmute(pendulum.DateTime, pendulum.datetime(2020, 3, 23, 16, 27, 12, tz='Europe/Oslo'))
Out: DateTime(2020, 3, 23, 0, 0, 0)
Trying to run example from documentation.
import dataclasses
import datetime
import enum
import uuid
import typic
class DuckType(str, enum.Enum):
WHT: "white"
BLK: "black"
MLD: "mallard"
@typic.al
@dataclasses.dataclass
class Duck:
name: str
type: DuckType
created_on: datetime.datetime = dataclasses.field(
default_factory=datetime.datetime.utcnow
)
id: uuid.UUID = dataclasses.field(
default_factory=uuid.uuid4
)
Imported module above, and observed followint traceback:
Traceback (most recent call last):
File "/home/user/dev/test/blya.py", line 15, in <module>
@dataclasses.dataclass
File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 367, in typed
return _typed(_cls_or_callable) if _cls_or_callable is not None else _typed
File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 359, in _typed
return wrap_cls(obj, delay=delay, strict=strict) # type: ignore
File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 324, in wrap_cls
wrapped: Type[WrappedObjectT] = cls_wrapper(klass)
File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 322, in cls_wrapper
return _resolve_class(cls_, strict=strict, jsonschema=jsonschema, serde=serde)
File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 193, in _resolve_class
protos = protocols(cls, strict=strict)
File "/home/user/.local/lib/python3.7/site-packages/typic/serde/resolver.py", line 535, in protocols
annotation, parameter=param, name=name, is_strict=strict
File "/home/user/.local/lib/python3.7/site-packages/typic/serde/resolver.py", line 448, in resolve
flags=flags,
File "/home/user/.local/lib/python3.7/site-packages/typic/serde/resolver.py", line 379, in annotation
if is_static
File "/home/user/.local/lib/python3.7/site-packages/typic/serde/resolver.py", line 277, in _get_configuration
for x, y in util.cached_type_hints(origin).items()
File "/home/user/.local/lib/python3.7/site-packages/typic/util.py", line 375, in cached_type_hints
return get_type_hints(obj)
File "/usr/local/lib/python3.7/typing.py", line 978, in get_type_hints
value = _eval_type(value, base_globals, localns)
File "/usr/local/lib/python3.7/typing.py", line 263, in _eval_type
return t._evaluate(globalns, localns)
File "/usr/local/lib/python3.7/typing.py", line 467, in _evaluate
eval(self.__forward_code__, globalns, localns),
File "<string>", line 1, in <module>
NameError: name 'white' is not defined
This library is poorly supported by mypy, and as a result makes mypy complain quite loudly for some normal operations.
Tasks to complete:
py.typed
file to project.Hi, I like typical and I think it's more lightweight than pydantic. But add user defined type by typic.register
is something tedious or there's another convenient way I didn't found. If so, is it possible to simplify it like pydantic's __get_validators__
or some else?
Literal is a way of specifying that a variable is one of a set of specific values.
It'd be great if typical would be able to validate data using Literal annotations.
An example where I did a similar validation manually: https://github.com/bluelabsio/records-mover/blob/master/records_mover/records/delimited/hint.py#L43-L71
Expected that when using strict with datetime, that it only accept datetime object. And do not do any type coercer. But the following example do not behave as I expected. Expected to get an exception in both cases.
>>> import typic
>>> typic.transmute(typic.Strict[pendulum.DateTime], '2020-04-09')
DateTime(2020, 4, 9, 0, 0, 0, tzinfo=Timezone('UTC'))
or similary
>>> @typic.klass(strict=True)
... class Test:
... date: pendulum.DateTime
>>> Test('2020-04-09')
Test(date=DateTime(2020, 4, 9, 0, 0, 0, tzinfo=Timezone('UTC')))
I haven't read through this project thoroughly but it looks like it overlaps a lot with https://github.com/agronholm/typeguard, at least on the validation side. Just thought you might want to know about it and maybe mention it in your docs somewhere. Maybe you two can even help each other.
@seandstewart , is this a bug?
import typic
@typic.klass
class Foo:
foo: str
@typic.klass
class FooBar(Foo):
bar: str
isinstance(Foo.transmute({'foo': 'foo'}), Foo) # True
isinstance(FooBar.transmute({'foo': 'foo', 'bar': 'bar'}), FooBar) # False
Hey, @seandstewart , there still some problem with defaultdict.
import typic
from collections import defaultdict
from typing import List, DefaultDict
@typic.klass
class Foo:
foo: DefaultDict[str, List[int]] = typic.field(
default_factory=lambda: defaultdict(list), init=False
)
@typic.klass
class Bar:
bar: Foo
# TypeError: keywords must be strings
There's many cases that user wants to transform the data object to a common dict, tuple, or namedtuple. Method dict
maybe have some args like exclude
or include_private
and so on.
The attr named __typic_fields_tuple__
, maybe some other name, is a tuple only contains the filed names. Although __typic_fileds__
is also a tuple, but contains detailed field info. Everytime I want to get the field names, I have to do something like:
fields = [f.name for f in self.__typic_fields__]
Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
Ran into some weird bugs when using multiple classes with the same name (defined in different files). There must be some kind of caching-by-class-name going on.
Seems to be related to calling .transmute()
with one another as an argument.
# index.py
from other import my_function
import typic
@typic.klass
class MyClass:
field: int
def __post_init__(self):
print("index.py: MyClass is being constructed")
# Construct a MyClass in other.py
other = my_function()
# Construct a local MyClass
MyClass(field=1)
# This should only construct a local MyClass, but instead calls __post_init__
# in other.py
MyClass.transmute(other)
# other.py
import typic
@typic.klass
class MyClass:
field: int
def __post_init__(self):
print("other.py: MyClass is being constructed")
def my_function():
val = MyClass(field=1)
return val
Running python index.py
prints this:
other.py: MyClass is being constructed
index.py: MyClass is being constructed
other.py: MyClass is being constructed
You would expect this:
other.py: MyClass is being constructed
index.py: MyClass is being constructed
index.py: MyClass is being constructed
pandas
is quite a common used lib for data proccessing. Except pd.Timestamp
, the most basic types DateFrame and Series is not supported currently.
import typic
import pandas as pd
@typic.klass
class Foo:
df: pd.DataFrame
# TypeError: 'NoneType' object is not callable
@typic.klass
class Bar:
ss: pd.Series
# TypeError: 'NoneType' object is not callable
Is there a simple way that typical
can support arbitrary types?
Pydantic features an "orm" mode for translating directly from an ORM model.
This should be a relatively easy feature to implement, since we're really just creating a mapping between objects.
https://github.com/samuelcolvin/pydantic/blob/master/pydantic/main.py#L448
We could probably easily support to/from with very little effort.
We should be able to generate JSON Schema definitions from typed classes.
It's stated in the README:
The following annotations are special forms which cannot be resolved:
Union
Any
Because these signal an unclear resolution, Typical will ignore this flavor of annotation ...
However, when I try to coerce some values to Unions, I didn't expect Typical to silently ignore Union annotations: (e.g. it returns a str
when a str
is passed in, when none of the Union members are str
). Since it is ignored, I can't rely on passing my value through coercion to actually get the correct type returned to me. I think this is a major limitation and a source of errors.
from typing import Union
import typic
# Case A. Incorrect. Expected: ValueError
print(typic.coerce("foo", Union[float, int])) # foo
from enum import Enum
class MyEnum(Enum):
a = "foo"
b = "bar"
# Case C1. Correct: Union with None works.
print(typic.coerce("foo", Union[MyEnum, None])) # MyEnum.a
print(typic.coerce(None, Union[MyEnum, None])) # None
# Case C1. Incorrect. Expected: MyEnum.a
print(typic.coerce("foo", Union[MyEnum, int])) # foo
print(typic.coerce("foo", Union[MyEnum, bool])) # foo
class Sentinel:
pass
# Case B. Incorrect. Expected: MyEnum.a
print(typic.coerce("foo", Union[MyEnum, Sentinel])) # foo
# Case C1: Correct: The value is an instance of one of the types
print(typic.coerce(1, Union[bool, int])) # 1
print(typic.coerce(False, Union[bool, int])) # False
# Case B. Incorrect. Expected: True (since it can be coerced to bool, but not to int)
print(typic.coerce("asdf", Union[bool, int])) # asdf
# Case C2. Expected: TypeError, since it could be coerced to both
print(typic.coerce("1", Union[bool, int])) # 1
Would it really be problematic to handle coercing to Union in some cases?
I can imagine scenarios where it's unambiguous, and one scenario where it is and an error could be raised:
Union[T, None]
)What do you think?
In my mind, it might look something like this:
from typing import Any, Iterable, Type
from typic import coerce
# Sentinel value
class Empty:
pass
_empty = Empty()
def coerce_union(value: Any, union: Iterable[Type]) -> Any:
coerced_value = _empty
for typ in union:
# If the value is already an instance of one of the types in the Union, coerce using that type
# Presumably, if the type was constrained, coercing could raise errors here, but that would be expected.
if isinstance(value, typ):
return coerce(value, typ)
error = None
for typ in union:
try:
new_coerced_value = coerce(value, typ)
except (ValueError, TypeError) as e:
# Store the error and continue
error = e
continue
if coerced_value is not _empty:
raise TypeError("Ambiguous coercion: the value could be coerced to multiple types in the Union")
coerced_value = new_coerced_value
if coerced_value is not _empty:
return coerced_value
# The value couldn't be coerced to any of the types in the Union
raise error
@seandstewart , there's some prolems with Callable
in the new version 2.0.17
, and it's OK in version 2.0.15
.
import typic
import typing
@typic.al
def foo(f: typing.Callable[[typing.Any], None], a: typing.Any):
f(a)
def func(a: typing.Any):
print(a)
ls = []
foo(func, 1) # AttributeError
foo(ls.append, 1) # TranslatorTypeError
typic.validate(typing.Callable[[typing.Any], None], func) # TypeError
typic.validate(typing.Callable[[typing.Any], None], ls.append) # TypeError
typic.validate(typing.Callable, func) # OK
typic.validate(typing.Callable, ls.append) # OK
typic.transmute(typing.Callable, ls.append) # TranslatorTypeError
typic.transmute(typing.Callable, func) # AttributeError
typic.transmute(typing.Callable[[typing.Any], None], func) # AttributeError
typic.transmute(typing.Callable[[typing.Any], None], ls.append) # TranslatorTypeError
Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
We should post definitive benchmarks - perhaps make use of Pydantic's own benchmarking suite.
varargs are not properly validated - each item should be checked in turn.
See #43 for an explanation.
It seems like typical doesn't convert the StringFormat.DATE enum field into a primitive (e.g. str), which fastjsonschema doesn't like.
>>> import typic
>>> from datetime import date
>>> from dataclasses import dataclass
>>> @typic.al
... @dataclass
... class Foo:
... foo: date
...
>>> typic.schema(Foo)
ObjectSchemaField(title='Foo', description='Foo(foo: datetime.date)', properties={'foo': StrSchemaField(format=<StringFormat.DATE: 'date'>)}, additionalProperties=False, required=('foo',))
>>> typic.schema(Foo).validate({})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../venv/lib/python3.7/site-packages/typic/schema/field.py", line 194, in validate
return self.validator(obj)
File ".../venv/lib/python3.7/site-packages/typic/util.py", line 226, in __get__
cache[attrname] = self.func(instance)
File ".../venv/lib/python3.7/site-packages/typic/schema/field.py", line 189, in validator
return fastjsonschema.compile(self.asdict())
File ".../venv/lib/python3.7/site-packages/fastjsonschema/__init__.py", line 167, in compile
exec(code_generator.func_code, global_state)
File "<string>", line 5
raise JsonSchemaException("data must be object", value=data, name="data", definition={'type': 'object', 'title': 'Foo', 'description': 'Foo(foo: datetime.date)', 'properties': {'foo': {'type': 'string', 'format': <StringFormat.DATE: 'date'>}}, 'additionalProperties': False, 'required': ['foo'], 'definitions': {}}, rule='type')
^
SyntaxError: invalid syntax
Right now typical eagerly generates the field path for displaying in error messages. This should ideally be lazy, so that the cost of a potentially complex __str__
or __repr__
is avoided in the common -- non-failure -- case.
This costs an object construction and a couple refcounts, but avoids creating and destroying just as many strings as those objects -- and storing a list is probably cheap enough.
I'm sure there is a module for this, but last time I needed it I just wrote my own. The implementation is pretty trivial, honestly, but this is off the top of my head, and in the github editor, so if it don't compile, don't be surprised. :)
class LazyReprConcat(object):
__slots__ = ['things']
def __init__(self, *things):
self.things = things
def __str__(self):
return ''.join(repr(thing) for thing in self.things)
__repr__ = __str__
The reason it did the concat was to allow easy recursive use: field = LazyReprConcat(parentPath, fieldName)
, but there are probably a million viable ways to handle all that, including more automagic with +
and stuff...
As the title said. If I subclass a base class with __init_subclass__
, the method super
will raise TypeError
.
import typic
@typic.klass
class Base:
def __init_subclass__(cls, **kwargs):
print('subclassed!')
super().__init_subclass__(**kwargs)
@typic.klass
class Foo(Base):
foo: str
I guess this might be related to #58
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.