ltworf / typedload Goto Github PK

Hey, I have a behaviour which I think is a bug / missing functionality.

Here is the code (using typedload==2.7):

from typing import Optional, TypedDict

import typedload

class ExampleTypedDict(TypedDict):
    a: int
    b: Optional[int]

d = {"a": 0}

print(f"{d=}")

td = ExampleTypedDict(**d)

print(f"{td=}")

tdl = typedload.load(d, ExampleTypedDict)

And here is the result of running it:

➜  optional_typeddict pipenv run python main.py    
d={'a': 0}
td={'a': 0}
Traceback (most recent call last):
  File "/Users/kkom/Desktop/optional_typeddict/main.py", line 17, in <module>
    tdl = typedload.load(d, ExampleTypedDict)
  File "/Users/kkom/.local/share/virtualenvs/optional_typeddict-hmzsbyx_/lib/python3.9/site-packages/typedload/__init__.py", line 200, in load
    return loader.load(value, type_)
  File "/Users/kkom/.local/share/virtualenvs/optional_typeddict-hmzsbyx_/lib/python3.9/site-packages/typedload/dataloader.py", line 294, in load
    raise e
  File "/Users/kkom/.local/share/virtualenvs/optional_typeddict-hmzsbyx_/lib/python3.9/site-packages/typedload/dataloader.py", line 290, in load
    return cast(T, func(self, value, type_))
  File "/Users/kkom/.local/share/virtualenvs/optional_typeddict-hmzsbyx_/lib/python3.9/site-packages/typedload/dataloader.py", line 500, in _namedtupleload
    raise TypedloadValueError(
typedload.exceptions.TypedloadValueError: Value does not contain fields: {'b'} which are necessary for type ExampleTypedDict
Path: .

PS: Here is my exact setup:

➜  optional_typeddict cat Pipfile                  
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
typedload = "==2.7"

[dev-packages]

[requires]
python_version = "3.9"

CC: @sid-at-github

https://github.com/ltworf/typedload/blob/master/typedload/dataloader.py#L226

@ltworf - this particular code in _namedtupleload() saved my butt today:

        # Prepare the list of the needed name changes
        transforms = []  # type: List[Tuple[str, str]]
        for field in fields:
            if type_.__dataclass_fields__[field].metadata:
                name = type_.__dataclass_fields__[field].metadata.get('name')
                if name:
                    transforms.append((field, name))

So, I truly take my hat to you for even thinking about this...

Having said that, it would be nice, if this particular feature were documented in the official guide. It took me a while to find this in the source code - I was actually trying to figure out how to implement this exact Python -> JSON mapping. Imagine my surprise, when I discovered that you have already thought about this :)

Once I found this code, it did not take me long to figure out that it is as simple as this:

@dataclass
class Foo():
  foo_bar: str = field(metadata={"name": "foo-bar"})

But I am not sure if everybody will find that at all...

I still I owe you a beer :D

https://mypy.readthedocs.io/en/latest/more_types.html#typeddict

It will be in the next python. Will just wait for it to be in typing instead.

json = Union[int, float, str, bool, None, List['json'], Dict[str, 'json'] ]
data = [{},[]]
assert typedload.load(data, json, frefs={'json': json}, basiccast= False) == data

This code, it fails or succeeds depending on the order of the arguments of the union, which changes between different executions of python.

It might be better to extend the initial check about basic types to avoid doing a cast, if that is possible.

This would mean that when loading something into Union[List[t], Tuple[t]], if the data is a list, it would always pick the list, and if it's a tuple, it'd pick the tuple.

Had to go back 1 version

You wrote in your README

Note that it is released with a GPL license and it cannot be used inside non GPL software.

What is your reason that your code under GPL can't be used in non-GPL software?

They gave me a limited amount of minutes to run tests, and after that they will want to be paid, so evaluate and migrate to something else is necessary.

Hello

I'm a big fan of this library and have used it in personal projects, but can't use it at work.

I'd love to use at my company and contribute to this library, but we can't due to the copyleft. Is there a specific reason for GPL 3.0? Any chance a more permissive license could be used? MIT/BSD or LGPL?

Version 3 exists solely for the deferred type evaluation feature of python 3.10.

However that change is for now being reverted, and pushed on to an unknown time in the future.

So it makes no sense to break API at this point. Revert the changes and make a 2.8 version instead.

They can be useful when writing code that needs to check types and run on multiple python versions.

Use this mypy configuration

[mypy]
python_version=3.5
warn_unused_ignores=True
warn_redundant_casts=True
strict_optional=True
scripts_are_modules=True
check_untyped_defs=True

Just presume it's an empty dictionary

Great library, thanks for creating and sharing it!

I wanted to report that I had problems when using it with dates and datetimes:

from datetime import date
from typing import NamedTuple

import typedload

class Point(NamedTuple):
    x: float
    y: float
    created_at: date

print(
    typedload.dump(
        Point(x=5.0, y=6.0, created_at=date(2019, 1, 1))
    )
)

It crashes with:

    raise TypedloadValueError('Unable to dump %s' % value, value=value)
typedload.exceptions.TypedloadValueError: Unable to dump 2019-01-01
Value: 2019-01-01
Type: None

Not sure if this is a bug, a feature, or simply not implemented yet.

Is it possible to change the license to one that does not requires the dependent codebase be open sourced?

TypeError: Cannot deal with value {'x': 5, 'y': 6, 'next': None} of type _ForwardRef('Node')

class Node(NamedTuple):
    x: int = 1
    y: int = 2
    next: 'Node' = None

For the parameters, so there is no need to set them later on.

field_defaults from the list.

Remove elif isinstance(value, tuple) and {'_fields', '_field_defaults', '_asdict'}.issubset(set(dir(value))):

.

https://docs.python.org/3/library/dataclasses.html#post-init-processing

As suggested here: https://github.com/pappasam/serdelicacy#notes

So that NamedTuple when converted to dict would have "no key" instead of None.

Perhaps, Optional[] will do?

If a field has a list factory and the value is empty, the field is dumped with hidedefaults=True.

Investigate and possibly fix.

If a field is defined as a compound type, e.g. List[T] or Dict[str,T], but the JSON contents specifies null as the value for the entire field, typedload.load(...) should detect that explicitly and generate an error along the lines of "null can not be converted to ...". The current behavior generates confusing errors (see below, I only checked lists and dictionaries).

For List[T]

The actual error thrown:
AssertionError()

What really happens is that enumerate(...) can not be called on a None object here:

Traceback (most recent call last):
  File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 216, in load
    return func(self, value, type_)
  File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 273, in _listload
    return [l.load(v, t, annotation=Annotation(AnnotationType.INDEX, i)) for i, v in enumerate(value)]
TypeError: 'NoneType' object is not iterable

For Dict[str,T]

The actual error thrown:
TypedloadAttributeError("'NoneType' object has no attribute 'items'")

What really happens is that .items() can not be called on a None object here:

Traceback (most recent call last):
  File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 286, in _dictload
    for k, v in value.items()}
  File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 286, in <dictcomp>
    for k, v in value.items()}
  File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 220, in load
    raise e
  File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 216, in load
    return func(self, value, type_)
  File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 288, in _dictload
    raise TypedloadAttributeError(str(e), type_=type_, value=value)
typedload.exceptions.TypedloadAttributeError: 'NoneType' object has no attribute 'items'
Value: None
Type: typing.Dict[str, typing.List[str]]

Load trace:
Type: typing.Dict[str, typing.List[str]] Annotation: (AnnotationType.VALUE None) Value: None
Path: 

has unrecognized fields and cannot be loaded into

This exception would be better if there was a list of which fields are too many.

On error, it'd be nice to print all the hierarchy, to know exactly where the error happened, in the data tree.

When loading a huge amount of items into a list it would probably cause a great speed up. Maybe do some tests to see how much would that be.

It would be easy to support name mangling like for attr.

Literal types. See PEP 586 and typing.Literal. Literal types indicate that a parameter or return value is constrained to one or more specific literal values:

    def get_status(port: int) -> Literal['connected', 'disconnected']:
        ...

This seems kinda the same as an enum but without the names for the values.

        try:
            return cast(T, func(self, value, type_))
        except Exception as e:
>           assert isinstance(e, TypedloadException)
E           AssertionError

../../../.local/lib/python3.8/site-packages/typedload/dataloader.py:236: AssertionError

I get this error in my application when calling typedload.load and I'm not sure what I should do about it - seems like that assert is there to catch a bug in typedload?

Could it be possible to mention apischema as an alternative in the comparison section of the documentation?

List the value and the options (if they are few) in the exception.

A dictionary type with per-key types. See PEP 589 and typing.TypedDict. TypedDict uses only string keys. By default, every key is required to be present. Specify “total=False” to allow keys to be optional:

    class Location(TypedDict, total=False):
        lat_long: tuple
        grid_square: str
        xy_coordinate: tuple

With the non pre evaluated types :(

In [21]: load(None, Optional[str])
    ...: 
Out[21]: 'None'

This happens.

Should NoneType be a basic type?

Hi,

I am trying to load a sub-set of data into a data_class, but its not allowing me, is this not supported?
Are there any special flags I need to send in?

Error:

<class 'typedload.exceptions.TypedloadValueError'> Value does not contain fields: {'f1', 'f2', 'f3', 'f4'} which are necessary for type <class 'app.models.MyData'>

In your main example, you have sessions with a default value of empty array. If you do not include sessions on your data, then it should assume empty array (the default) or at least have the option to.

Thanks

The annotations are useful to track specifically which element is causing the issue. All the functions calling l.load() should have annotations.

I have some things that are a Dict[str, Any] where any can be any thing representable in json. So I basically want typeshed to "pass through" the values of that dict. I'm now using Dict[str, Union[str, int, float, bool, NoneType]] but to make it work recursively the type is even longer.

The typing.NamedTuple class has deprecated the _field_types attribute in favor of the annotations attribute which has the same information. (Contributed by Raymond Hettinger in bpo-36320.)

Having that will cause everything to fail. It looks like I want to support it.

document literal+union usage

Maybe link to mypy where it illustrates how that is done