pawelzny / dotty_dict Goto Github PK

View Code? Open in Web Editor NEW

91.0 3.0 27.0 182 KB

Dictionary wrapper for quick access to deeply nested keys.

Home Page: http://dotty-dict.readthedocs.io

License: MIT License

Makefile 2.71% Python 97.29%

dot notation dict wrapper helper utils lib

dotty_dict's Introduction

Dotty-Dict

Info: Dictionary wrapper for quick access to deeply nested keys.
Author: Pawel Zadrozny @pawelzny <[email protected]>

Features

Simple wrapper around python dictionary and dict like objects
Two wrappers with the same dict are considered equal
Access to deeply nested keys with dot notation: dot['deeply.nested.key']
Create, read, update and delete nested keys of any length
Expose all dictionary methods like .get, .pop, .keys and other
Access dicts in lists by index dot['parents.0.first_name']
key=value caching to speed up lookups and low down memory consumption
support for setting value in multidimensional lists
support for accessing lists with slices

Installation

pip install dotty-dict

Package: https://pypi.org/project/dotty-dict/
Source: https://github.com/pawelzny/dotty_dict

Documentation

Full documentation: http://dotty-dict.readthedocs.io
Public API: http://dotty-dict.readthedocs.io/en/latest/api.html
Examples and usage ideas: http://dotty-dict.readthedocs.io/en/latest/examples.html

TODO

Waiting for your feature requests ;)

Quick Example

Create new dotty using factory function.

>>> from dotty_dict import dotty
>>> dot = dotty({'plain': {'old': {'python': 'dictionary'}}})
>>> dot['plain.old']
{'python': 'dictionary'}

You can start with empty dotty

>>> from dotty_dict import dotty
>>> dot = dotty()
>>> dot['very.deeply.nested.thing'] = 'spam'
>>> dot
Dotty(dictionary={'very': {'deeply': {'nested': {'thing': 'spam'}}}}, separator='.', esc_char='\\')

>>> dot['very.deeply.spam'] = 'indeed'
>>> dot
Dotty(dictionary={'very': {'deeply': {'nested': {'thing': 'spam'}, 'spam': 'indeed'}}}, separator='.', esc_char='\\')

>>> del dot['very.deeply.nested']
>>> dot
Dotty(dictionary={'very': {'deeply': {'spam': 'indeed'}}}, separator='.', esc_char='\\')

>>> dot.get('very.not_existing.key')
None

NOTE: Using integer in dictionary keys will be treated as embedded list index.

Install for development

Install dev dependencies

$ make install

Testing

$ make test

Or full tests with TOX:

$ make test-all

Limitations

In some very rare cases dotty may not work properly.

When nested dictionary has two keys of different type, but with the same value. In that case dotty will return dict or list under random key with passed value.
Keys in dictionary may not contain dots. If you need to use dots, please specify dotty with custom separator.
Nested keys may not be bool type. Bool type keys are only supported when calling keys with type defined value (e.g. dot[True], dot[False]).

dotty_dict's People

Contributors

Stargazers

Watchers

dotty_dict's Issues

no_list has no effect when adding entries

When creating an empty dotty object and adding new entries using dotted strings, the no_list flag has no effect.

from my understanding this is a bug and there is a simple fix for that.

If it is of interest, I can make a pull request with a fix.

Problem description

d = dotty(no_list=True)
d["my.string.with.numeric.123.field"] = "abc"

Creates a list with 124 elements in "numeric".

Expected behavior:
"123" is a key in sub dict "numeric"

Accessing root level list items by index fails

Thanks for the library! Using latest version (v1.1.1), this test fails

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import unittest

from dotty_dict import dotty_l as dotty


class TestListInDotty(unittest.TestCase):
    def setUp(self):
        self.dot = dotty({
            'field1': 'Value of F1',
            'field2': 'Value of F2',
            'field3': [
                {
                    'subfield1': 'Value of subfield1 (item 0)',
                    'subfield2': 'Value of subfield2 (item 0)'
                },
                {
                    'subfield1': 'Value of subfield1 (item 1)',
                    'subfield2': 'Value of subfield2 (item 1)'
                },
            ],
            'field4': 'Not wanted',
            'field5': [
                {
                    'subfield1': [{'subsubfield': 'Value of sub subfield (item 0)'}]
                }
            ],
            'field6': ['a', 'b']
        })

    def test_root_level_list_element(self):
        self.assertEqual(self.dot['field6.0'], 'a')

This fails on dotty_dict.py", line 122, in get_from since no subfield is specified

    it = items.pop(0)
IndexError: pop from empty list

Object of type InMemoryUploadedFile is not JSON serializable

Hi
Thank you for your wonderful Library.
I've found a minor issue in the to_dict() method, when I wanna use it to convert Dotty to a regular dict() and the Dotty() contains File data.
The following error is raised:
{TypeError}Object of type InMemoryUploadedFile is not JSON serializable

Numeric keys + deep lists = trouble

Please consider the following:

"2019": {
    "uat": {
      "pkey": true,
      "user": "testval",
      "testkey": true,
      "mylist": [
      {
        "foo": "bar",
        "foo2": "bar2"
      },
      {
        "baz": "milk",
        "bar": "foo"
      }
      ]
    },
    "dev": {
      "pkey": true,
      "testval": "testval",
      "testval2": true
    },
    "test1": [1, 2, "testval"],
    "test2": [{"one": "foo", "two": "bar", "three": "testval"}]
  }
}

There is no way to access deep lists now. If I use no_list=True I will be in trouble because 2019 will work as key but then I can't access mylist. This has me stuck in a project. I've been trying to make some function to parse the key tree and flip no_list on and off while traversing but it's not straight forward what to do.

Reason:

If I need to copy the data schema, I can't make new dotty() and then say:

new["2019.uat.mylist.0"] = dotty(data)

because numeric will always be a list or a key. No mix.

Solution Request:

Identifier for lists in leafy strings. Maybe like:

"2019.uat.mylist.[0]"

dotty items

I'd like to be able to iterate over items of a dotty, where it returns the dotty key and value.

d = dotty({"foo": [{"food": [{"sand": "meat"}]},{"drink": [{"tea":"black", "soda": "cream"}]}]})

>> for k, v in d.dotty_items(breadth_first=True):
>>     print(k)
"foo"
"foo.0"
"foo.1"
"foo.0.food"
"foo.0.food.0"
"foo.0.food.0.sand"
"foo.1.drink"
"foo.1.drink.0"
"foo.1.drink.0.tea"
"foo.1.drink.0.soda"

maybe have method params for a key to start at and a depth limit.

No values returned when one item is missing using slices to get a subset of the data

When using slices without missing data it works perfectly:

data = dotty(
    {
        "users": [
            {"first_name": "Sam", "last_name": "Gamji"},
            {"first_name": "John", "last_name": "Doe"},
            {"first_name": ""}
         ]
    }
)

data.get("users.*.first_name") #== ["Sam", "John", ""]

But with one missing item in the subset it returns just None:

data = dotty(
    {
        "users": [
            {"first_name": "Sam", "last_name": "Gamji"},
            {"last_name": "Doe"},
            {"first_name": ""}
         ]
    }
)

data.get("users.*.first_name") #== None

I would expect to get:

data.get("users.*.first_name") #== ["Sam", None, ""]

Is it wanted ? Is it a bug ?

It would be really nice to have this behaviour !
Thanks 🙏 .

get method could return a dotty object

Hi!
So, as the title says, it would be great if the get method on dotty object returned another dotty object (if relevant of course).

This way, it would be easier to parse a whole document, as well as all its subparts using only get methods (and not having to convert every sub-dict into new dotty objects).

This is not critical though, this tool is already amazing :)

Unable to install the package using pip install command. throwing error: [Errno 2] No such file or directory: 'pipenv'

Dotty.pop(key, default) raises KeyError when key doesn't exist.

When default is given, Cut.pop(key, default) is expected to return default if key doesn't exist.

Reproducing steps:

Python 3.6.8 (default, Aug 20 2019, 17:12:48)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dotty_dict import dotty
>>> a = dotty()
>>> a.pop('does.not.exist', None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 263, in pop
    return pop_from(self._split(key), self._data)
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 258, in pop_from
    data = data[it]
KeyError: 'does'
>>>

Ubuntu @ Windows Subsystem Linux | Windows 10

Iterator for dotty_dict

I was trying to use dotty like a regular dictionary and it gives an unexpected KeyError exception when trying to iterate over it.

import dotty_dict

d = dotty_dict.dotty(
            {
                "top": {
                    "first": {"a": "alpha", "b": "beta"},
                    "second": {},
                },
                "bottom": {
                },
            }                        
        )
for item in d:
    print(item)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tpow/example/.venv/lib/python3.11/site-packages/dotty_dict/dotty_dict.py", line 162, in __getitem__
    return get_from(self._split(item), self._data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tpow/example/.venv/lib/python3.11/site-packages/dotty_dict/dotty_dict.py", line 154, in get_from
    data = data[it]
           ~~~~^^^^
KeyError: 0

This also throws the same exception when trying to use dotty_dict as a comprehension:

b = [item for item in d if item.startswith("b")]
assert b == ["bottom"]  # <-- never gets here

(This is simply an example. I know there are other convenient ways to retrieve items using dotty_dict.)

Although I could use keys() when iterating, that isn't typically needed for dictionaries and it would be nice if dotty_dict worked the same.

In order to make it behave more like a regular dictionary, I think all dotty_dict needs is a simple __iter__ like this:

    def __iter__(self):                                                        
        for key in self.keys():
            yield key

It may also be that there is something broken in get_from() that I don't quite understand.

KeyError: 0 when a digit is not a list index, but a key

Hello @Dysproz

Today I found an issue while testing another scenario, could you please check what could be wrong:

from dotty_dict import dotty

dot = dotty({'0': {'key': 'value'}})
print(dot['0'])

Traceback (most recent call last):
  File "z.py", line 8, in <module>
    print(dot['0'])
  File "/usr/local/lib/python3.7/dist-packages/dotty_dict/dotty_dict.py", line 139, in __getitem__
    return get_from(self._split(item), self._data)
  File "/usr/local/lib/python3.7/dist-packages/dotty_dict/dotty_dict.py", line 131, in get_from
    data = data[it]
KeyError: 0

Thank you

do not convert numeric keys to indexes

Having a choice how to treat ints was a great option.
It was possible to decide how to deal with 'KeyA.1.KeyB' in 1.1.2 but unfortunalelly it was removed.

Setup py error

Hello, getting this error when installing the package in CICD

Collecting dotty-dict
  Downloading dotty_dict-1.2.1.tar.gz (37 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/user/miniconda/envs/py36/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rm8rt_sd/dotty-dict/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rm8rt_sd/dotty-dict/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-bo5mkxxv
         cwd: /tmp/pip-install-rm8rt_sd/dotty-dict/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-rm8rt_sd/dotty-dict/setup.py", line 16, in <module>
        readme = readme_file.read()
      File "/home/user/miniconda/envs/py36/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 113: ordinal not in range(128)
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Request: Need a way to access the list elements using square brackets

Hello guys,

I need a way to access the list elements from a dotty dict by using square brackets, here is an example:

dict['something.0.other'] => dict['something[0].other']

Could you please tell me if is possible to have this implemented in future?

Thanks

IndexError: pop from empty list

When I try to see if a key exist in the dotty dict by using "in", I receive an IndexError

if key in dot:
pass

The input that I give is "something.0', line number is 103, in search_in method

Thanks

Handling key with None value

In version 1.2.1, if you tried: dot('ParentWithNoneValue.Child') it would return None. However, on version 1.3.0, the same syntax returns exception TypeError: argument of type 'NoneType' is not iterable.

I had to downgrade to version 1.2.1, could you please keep the behaviour the same as 1.2.1? Thank you.

Access a specific item in a subfield?

I have an example dict that looks like this:

{
        'field1': 'Value of F1',
        'field2': 'Value of F2',
        'field3': [
                {
                    'subfield1': 'Value of subfield1 (item 0)',
                    'subfield2': 'Value of subfield2 (item 0)'
                },
                {
                    'subfield1': 'Value of subfield1 (item 1)',
                    'subfield2': 'Value of subfield2 (item 1)'
                },
            ],
        'field4': 'Not wanted'
    }

This is my full script to test:

from dotty_dict import dotty
wanted_fields = ['field1', 'field2', 'field3.subfield1']
# field3.subfield1 works only if field3 isn't a list, but a single item
# field3.0.subfield1 throws a type error
# field3[0].subfield1 throws a keyerror

devices = [
    {
        'field1': 'Value of F1',
        'field2': 'Value of F2',
        'field3': [
                {
                    'subfield1': 'Value of subfield1',
                    'subfield2': 'Value of subfield2'
                }
            ],
        'field4': 'Not wanted'
    }
]

for device in devices:
    devices_map = dotty(device)
    for item in wanted_fields:
        print("{item} => {value}".format(item=item, value=devices_map[item]))

How can I get just element 0 from the field3 key?

I tried field3.0.subfield1 and field3[0].subfield1. Neither works.

With the first, I receive a TypeError: TypeError: list indices must be integers or slices, not str
With the second, I receive a KeyError: KeyError: 'field3[0]'

Additionally, how can I get every subfield1 in the field3 list?

Special characters in the author's name

The author's name contains special characters that prevent the installation on Windows with Japanese encoding.

pip raises the following error:

File "C:\Users\test\AppData\Local\Temp\pip-install-qogza7z4\dotty-dict\setup.py", line 16, in <module>
        readme = readme_file.read()
UnicodeDecodeError: 'cp932' codec can't decode byte 0x82 in position 119: illegal multibyte sequence

Support `pop` for lists

Is there a way do pop or remove elements from lists?

Say I have a dotty like dot["foo.bar.0"] = 1, I'd like to pop that element. Something like dot.pop("foo.bar.0") doesn't work. My only idea at this point is to split the path and pop it like dot["foo.bar"].pop(0).

path = "foo.bar.0"
parent, element = path.rsplit(separator, maxsplit=1)
dot[parent].pop(element)

no_list=True ... fails?

i think this is a bug:

import dotty_dict
d = dotty_dict.dotty(no_list=True)
d["one.two"] = "three"     # works
d["one.2"] = "three"       # boom!!

immediate result:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tm/.virtualenvs/backend/lib/python3.9/site-packages/dotty_dict/dotty_dict.py", line 198, in __setitem__
    set_to(self._split(key), self._data)
  File "/Users/tm/.virtualenvs/backend/lib/python3.9/site-packages/dotty_dict/dotty_dict.py", line 190, in set_to
    set_to(items, data[it])
  File "/Users/tm/.virtualenvs/backend/lib/python3.9/site-packages/dotty_dict/dotty_dict.py", line 194, in set_to
    self.set_list_index(data, it, value)
  File "/Users/tm/.virtualenvs/backend/lib/python3.9/site-packages/dotty_dict/dotty_dict.py", line 210, in set_list_index
    data.append(None)
AttributeError: 'dict' object has no attribute 'append'

Cannot serialize a dotty object with dotty objects

First of all - thanks a lot (dziekuję :)) for dotty_dict. It wonderfully simplifies my code.

I ran into an issue when trying to serialize some nested dotty() objects because .to_dict() does not process them:

>>> from dotty_dict import dotty 
>>> a = dotty()
>>> a['hello.world'] = 1
>>> b = dotty()
>>> b['wazaa'] = 3
>>> a['nested.dotty'] = b
>>> a
Dotty(dictionary={'hello': {'world': 1}, 'nested': {'dotty': Dotty(dictionary={'wazaa': 3}, separator='.', esc_char='\\')}}, separator='.', esc_char='\\')
>>> a.to_dict()
{'hello': {'world': 1}, 'nested': {'dotty': Dotty(dictionary={'wazaa': 3}, separator='.', esc_char='\\')}}
>>> import json
>>> json.dumps(a)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Program Files (x86)\Python38-32\lib\json\__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "C:\Program Files (x86)\Python38-32\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Program Files (x86)\Python38-32\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "C:\Program Files (x86)\Python38-32\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Dotty is not JSON serializable

It may be worthwhile to add this to the docs and suggest to .to_dict() the intermediate object.

BTW it took me some time to find out how to convert a dotty() object to a dict, fortunately this was in one of the examples.

no_list should be smart_list :)

that would be perfect:

d = dotty(smart_list=True)
some_list = [1, 2, {"work": "right?"}]

# if there is no list - just treat it as a key
d["one.2.three"] = "hey ho"
assert d["one.2.three"] == "hey ho"

# if there is a list, use it :)
d["this.should"] = some_list
assert d["this.should.2.work"] = "right?"

Return all in a list

I'd love to be able to return a list based on a wildcard character, example:

dot = dotty({
    'annotations': [
        {'label': 'app', 'value': 'webapi'},
        {'label': 'role', 'value': 'admin'},
    ],
    'spec': {
         'containers': [
             ['gpu', 'tensorflow', 'ML'],
             ['cpu', 'webserver', 'sql'],
        ]
    }
})

dot['annotations.*.label'] would return: ["app", "role"]

getitem / get_from raises KeyError when key is numeric string.

KeyError is thrown instead of existing value.
Everything works when the numeic key is a numeric type (i.e. int)

Reproducing steps:

Python 3.6.8 (default, Aug 20 2019, 17:12:48)                                                                          
[GCC 8.3.0] on linux                                                                                                   
Type "help", "copyright", "credits" or "license" for more information.                                                 
>>> from dotty_dict import dotty                                                                                       
>>> dict_data = {'a': {'1': 'a_one'}}                                                                                  
>>> a = dotty(dict_data)                                                                                               
>>> a['a.1']                                                                                                           
Traceback (most recent call last):                                                                                     
  File "<stdin>", line 1, in <module>                                                                                  
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 139, in __getitem__ 
    return get_from(self._split(item), self._data)                                                                     
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 135, in get_from    
    return get_from(items, data)                                                                                       
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 131, in get_from    
    data = data[it]                                                                                                    
KeyError: 1                                                                                                            
>>>

Also:

>>> dict_data = {'2': 'b'}
>>> a = dotty(dict_data)
>>> a[2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 139, in __getitem__
    return get_from(self._split(item), self._data)
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 307, in _split
    key = key.replace(*skp_stamp).replace(*esc_stamp)
AttributeError: 'int' object has no attribute 'replace'
>>> a['2']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 139, in __getitem__
    return get_from(self._split(item), self._data)
  File "/home/gil/.virtualenvs/Sandbox/lib/python3.6/site-packages/dotty_dict/dotty_dict.py", line 131, in get_from
    data = data[it]
KeyError: 2
>>>