Currently, rtoml converts fields with value None to a value of string "null" in the dumped string.
Loading the string back in results in a "null" string in python instead of None.

I can only imagine two possible correct behaviors for loading "null" strings instead:

• "null" strings should be converted to None in python, or
• fields with value "null" should not be imported at all.

Conversely, for dumping None values:

• convert to null (without the quotation marks) -- representing a json null value
• do not dump fields with value None at all

Here are two tests demonstrating the failing scenario with rtoml and the passing example using toml:

import rtoml

def test_none_to_null_string():
    """ fails with 

    ```AssertionError: assert {'optional_field': None} == {'optional_field': 'null'}```

    because None is dumped to "null" string value which is also loaded back in as the string "null".
    """
    d = {"optional_field": None}
    d_goal1 = d
    d_goal2 = {}
    dumped = rtoml.dumps(d)
    loaded = rtoml.loads(dumped)
    assert loaded == d_goal1 or loaded == d_goal2

def test_none_to_null_string_passing_toml_package():
    """ passes with toml packages, 
    because fields with value None are not dumped.
    """
    import toml
    d = {"optional_field": None}
    d_goal1 = d
    d_goal2 = {}

    dumped = toml.dumps(d)
    loaded = toml.loads(dumped)
    assert loaded == d_goal1 or loaded == d_goal2

I've encountered this error when testing locally with pytest 6.0.2 installed as well as in the build steps in #11.

The issue went away locally after I installed pytest 5.3.1 (i.e. the version pinned in tests/requirements.txt).

For example, the "test (3.8)" step has this output on the GitHub CI:

pytest --cov=rtoml
============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-5.3.1, py-1.9.0, pluggy-0.13.1

The "build py 3.8 on linux" step, however, has this:

+ pip install pytest
Collecting pytest
  Downloading pytest-6.1.0-py3-none-any.whl (272 kB)

followed by

+ sh -c 'pytest /project/tests -s'
============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.0, py-1.9.0, pluggy-0.13.1

Hi,

Thank you for rtoml, rust is sure fast !

The type hint of the return values are well defined for dumps(obj: Any, *, pretty: bool = False) -> str and dump(obj: Any, file: Union[Path, TextIO], *, pretty: bool = False) -> int.

But not for loads(toml: str) -> Any and load(toml: Union[str, Path, TextIO]) -> Any.

Why Any and not dict ?

I thought that a TOML would always load into a dict in Python.
Am I missing something ?

in docs it says that load takes either str, path or TextIO, but doesn't specify which Path type it takes.

This is a great project, I like it!
It would be nice if the releases can contain amd64 wheels for windows too.
Otherwise setuptools-rust must be present to continue the installation.

When I read in a TOML file and then dump it the order is not preserved. Is this intentional? It would be nice if order could be preserved either by default or by specifying an option.

Example:
I read

[section]
z = 'some'
a = 1

[other]
b = 41
a = "aval"

Dump it gives:

[other]
a = 1
b = 41

[section]
a = 1
z = 1

I'm trying to read a poetry toml file (which I'd think should not matter since it's a regular toml file) like this:

import rtoml
obj = rtoml.load("pyproject.toml")
print(obj)

and I'm getting

_rtoml.TomlParsingError: expected an equals, found eof at line 1 column 15

Here are few first lines from that file

[tool.poetry]
name = "datascience"
version = "0.1.0"

So it looks like it can't read the section [tool.poetry] line?
But in the example it seems to read it without problems?

rtoml version: 0.7.0

Running this

import rtoml
rtoml.dumps(rtoml.loads("time = 23:30:59"))

Errors with

_rtoml.TomlSerializationError: time is not serializable to TOML: datetime.time(23, 30, 59)

Is there any chance for us to build the wheel for Windows for python 3.11 while publishing it?

In order to create it for the user, the installation of rust and Microsoft C++ Build Tools is necessary, which can be quite burdensome.

Related: pwwang/datar#180

It would be nice if the releases can contain arm64 wheels for macos too.
Otherwise setuptools-rust must be present to continue the installation.

Version: rtoml = 0.8.0, python = 3.8.6
system: win11

example toml file:

# pyproject.toml
[project]
dependencies=["fire", "rtoml"]

[[project.authors]]
name = "Dragon-GCS"
email = "[email protected]"

Code which can work correctly, call it code1

import rtoml

with open("pyproject.toml") as f:
    tom = rtoml.loads(f.read())

with open("regenerate.toml", "w") as f:
    rtoml.dump(tom, f)

When I print tom, it's a dict like that:

{'project': 
  {
    'authors': [{'email': '[email protected]', 'name': 'Dragon-GCS'}],
    'dependencies': ['fire', 'rtoml']
  }
}

So I try to dump this dict to file with followed code:

tom = {
    'project': {
            'authors': [{'email': '[email protected]', 'name': 'Dragon-GCS'}],
            'dependencies': ['fire', 'rtoml']
        }
    }
with open("regenerate.toml", "w") as f:
    rtoml.dump(tom, f)

There was an Exception:

Traceback (most recent call last):
  File "e:\ProjectFiles\Python\Start\test_rtoml.py", line 12, in <module>
    rtoml.dump(tom, f)
  File "e:\ProjectFiles\Python\Start\.venv\lib\site-packages\rtoml\__init__.py", line 57, in dump
    s = dumps(obj, pretty=pretty)
  File "e:\ProjectFiles\Python\Start\.venv\lib\site-packages\rtoml\__init__.py", line 48, in dumps
    return serialize(obj)
_rtoml.TomlSerializationError: values must be emitted before tables

Another, when switch [project] and [[project.authors]], like that:

[[project.authors]]
name = "Dragon-GCS"
email = "[email protected]"
[project]
dependencies=["fire", "rtoml"]

code1 will raise TomlSerializationError too.

All the code was run correctly with toml(another toml parser package)

Hi @samuelcolvin. Do you know if the toml-rs library supports preservation of formatting and comments? Recently, I created a shared library wrapper around the rust toml_edit library (which does support comments and formatting, which was a requirement for my application of TOML).

The library I built is called toml-edit-labview since it's intended to be called from LabVIEW, but it builds the toml_edit rust library into a set of dll/so/dylib/.framework files for cross platform support).

I was curious how this mapped into this (rtoml) project and your use case for TOML.

The following code, exporting and reimporting a timestamp with millisecond precision fails:

from datetime import datetime
import rtoml

t = datetime.fromisoformat("2020-05-25T12:00:01.123450")
d = {"t": t}
dumped = rtoml.dumps(d)
loaded = rtoml.loads(dumped)
assert loaded == d

This is because the last subsecond digit is not dumped to string and load parsing fails with only 5 fractional digits.
If the last digit is not zero, it works.
Tested on macOS and Windows Python3.9.

Here are three unit tests covering passing and failing scenarios, including a passing test using the toml package instead.

from datetime import datetime
import rtoml
    
def test_date_milliseconds_passing():
    """passes, for 6 significant subsecond digits
    """
    t = datetime.fromisoformat("2020-05-25T12:00:01.123456")
    d = {"t": t}
    dumped = rtoml.dumps(d)
    loaded = rtoml.loads(dumped)
    assert loaded == d

def test_date_milliseconds_failing():
    """ 
    fails because last subsecond digit is not dumped to string and load parsing fails with only 5 fractional digits. 
    """
    t = datetime.fromisoformat("2020-05-25T12:00:01.123450")
    d = {"t": t}
    dumped = rtoml.dumps(d)
    loaded = rtoml.loads(dumped)
    assert loaded == d


def test_date_milliseconds_passing_toml_package():
    """ passes using toml package    
    """
    import toml
    t = datetime.fromisoformat("2020-05-25T12:00:01.123450")
    d = {"t": t}
    dumped = toml.dumps(d)
    loaded = toml.loads(dumped)
    assert loaded == d

I've had typing issues with rtoml.load accepting TextIO twice by now. Can't fully recreate the first situation, but the second is loading a TOML config file in Flask.

Flask's flask.Config.from_file has a load parameter whose type is documented as

Callable[[Reader], Mapping] where Reader implements a read method

and implemented as Callable[[IO[Any]], Mapping].

Trying to use rtoml with it like so:

import rtoml
app.config.from_file(config_filename, load=rtoml.load)

makes mypy unhappy:

Argument "load" to "from_file" of "Config" has incompatible type "Callable[[Union[str, Path, TextIO]], Dict[str, Any]]"; expected "Callable[[IO[Any]], Mapping[Any, Any]]"

Since type TextIO is a subclass of IO[str] with a few additional methods that rtoml does not use I suggest to change the signature to use IO[str] (or maybe even IO[Any] plus encoding handling?) instead of TextIO.

For comparison,

• tomli expects IO[bytes], and
• tomlkit extects IO.

What do you think?

hi, i maintain a package for this project in nixpkgs
(though i'm not very familiar with python)

i'm trying to update it from v0.8 to v0.9
this mostly worked on our 23.05 channel
(though there's two problems with the benchmarks)

but when i tried to build again on the latest nixpkgs channel it fails to install the rtoml output
i suspect this is due to nixpkgs having switched to using pypa's build instead of pip

an incomplete and misnamed lib/python3.10/site-packages/_rtoml gets created
but not the site-packages/rtoml i got on 23.05

>>> import tomli
>>> tomli.loads('"" = "abc"')
{'': 'abc'}

>>> import rtoml
>>> rtoml.loads('"" = "abc"')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    rtoml.loads('"" = "abc"')
  File "/path/to/python3.9/site-packages/rtoml/__init__.py", line 34, in
 loads
    return _rtoml.deserialize(toml)
_rtoml.TomlParsingError: empty table key found at line 1 column 1

Reference:

https://github.com/pwwang/toml-bench#testcompliancevalid

If passing a byte string or a rb-mode opened file, I think this library should be able to handle that input properly.

Actually, when parsing files, IMHO it's a better practice, because then the underlying library can handle any possible byte-to-string conversions that might be specific to the format.

Building wheel for rtoml (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [21 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-s390x-cpython-310
creating build/lib.linux-s390x-cpython-310/rtoml
copying rtoml/init.py -> build/lib.linux-s390x-cpython-310/rtoml
copying rtoml/py.typed -> build/lib.linux-s390x-cpython-310/rtoml
running build_ext
running build_rust
error: can't find Rust compiler

Could you add support for s390x?

When a list that is dumped inline is defined in a dict before a list of tables, dumping fails. This is duplicate of #45, which was already closed.

Minimal working example

>>>import rtoml
>>>rtoml.dumps({'bookmarks': [{'ch': 12.0}],'auto_download': ['ja']})
File .venv/lib/python3.11/site-packages/rtoml/__init__.py:49, in dumps(obj, pretty)
     46 else:
     47     serialize = _rtoml.serialize
---> 49 return serialize(obj)

TomlSerializationError: values must be emitted before tables

This does not occur if the order of the keys is switched:

>>>import rtoml
>>>print(rtoml.dumps({'auto_download': ['ja'],'bookmarks': [{'ch': 12.0}]}))
auto_download = ["ja"]

[[bookmarks]]
ch = 12.0

Taking care to define the items in the right order is for now a doable workaround.

rtoml Version: 0.9.0
Python Version: 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]

Hi Samuel,

would you mind providing wheels for Python 3.9?

I encountered an error message while working within my python:alpine container:

Traceback (most recent call last):
  File "//example.py", line 2, in <module>
    import rtoml
  File "/usr/local/lib/python3.11/site-packages/rtoml/__init__.py", line 5, in <module>
    from . import _rtoml
ImportError: cannot import name '_rtoml' from partially initialized module 'rtoml' (most likely due to a circular import) (/usr/local/lib/python3.11/site-packages/rtoml/__init__.py)

It seems that this error is caused by using wrong wheel, if I compile rtoml from source then the error gone. To reproduce, you can run the following command sequence:

Create container:

docker run --rm -it python:alpine sh

Inside the container:

wget https://raw.githubusercontent.com/samuelcolvin/rtoml/main/example.py

# Install using the wheel
pip install rtoml
# This will get error
python example.py

# Alternatively, build from source
pip uninstall rtoml
apk add cargo
pip install rtoml --no-binary :all:

# This can be executed successfully
python example.py

Currently the conda-forge Python 3.12 Linux-64 distribution is broken. The package installs but none of the package functions work, e.g. running the example code:

import rtoml

tomlstr = """\
title = "TOML Example"

[owner]
dob = 1979-05-27T07:32:00-08:00
name = "Tom Preston-Werner"

[database]
connection_max = 5000
enabled = true
server = "192.168.1.1"
ports = [8001, 8001, 8002]
"""

rtoml.load(tomlstr)

throws an error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 1
----> 1 rtoml.load(tomlstr)

AttributeError: module 'rtoml' has no attribute 'load'

inspecting the imported module shows that none of the load/dump functions are available:

In [6]: dir(rtoml)
Out[6]:
['__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__']

The dir output should look like this:

In [1]: import rtoml

In [2]: dir(rtoml)
Out[2]:
['Any',
 'Dict',
 'Path',
 'TextIO',
 'TextIOBase',
 'TomlParsingError',
 'TomlSerializationError',
 'Union',
 'VERSION',
 '__all__',
 '__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 '__version__',
 '_rtoml',
 'dump',
 'dumps',
 'load',
 'loads']

Between this issue and #74 users and dependent packages are very limited on Python 3.12