Which toml package to use in python?
MIT License
Which toml package to use in python?
See also: toml-lang and PEP 680
The verions of the packages tested in this report.
Version | |
---|---|
toml | 0.10.2 |
tomli/tomli_w | 2.0.1; tomli_w: 1.0.0 |
tomlkit | 0.12.5 |
rtoml | 0.11.0 |
qtoml | 0.3.1 |
tomllib | (Python 3.12.2) |
None
valueHow the package dumps None
value in python
Literally <package>.dumps(None)
Dumped value or error | |
---|---|
toml | 'NoneType' object is not iterable |
tomli/tomli_w | 'NoneType' object has no attribute 'items' |
tomlkit | Expecting Mapping or TOML Container, <class 'NoneType'> given |
rtoml | "null"---rtoml v0.11+ supports dumping None to a desired string:rtoml.dumps(data, none_value='@None') :"@None" |
qtoml | 'NoneType' object has no attribute 'items' |
tomllib | module 'tomllib' has no attribute 'dumps' |
None
pairHow the package dumps key-value pair with value None
Literally <package>.dumps({"key": None})
Dumped value or error | |
---|---|
toml | |
tomli/tomli_w | Object of type <class 'NoneType'> is not TOML serializable |
tomlkit | Invalid type <class 'NoneType'> |
rtoml | key = "null"---rtoml v0.11+ supports dumping None to a desired string:rtoml.dumps(data, none_value='@None') :key = "@None"
|
qtoml | TOML cannot encode None |
tomllib | module 'tomllib' has no attribute 'dumps' |
None
valueHow the package dumps a list with None
value in it.
Literally <package>.dumps({"key": [1, 2, 3, None, 5]})
Dumped value or error | |
---|---|
toml | key = [ 1, 2, 3, "None", 5,] |
tomli/tomli_w | Object of type <class 'NoneType'> is not TOML serializable |
tomlkit | Invalid type <class 'NoneType'> |
rtoml | key = [1, 2, 3, "null", 5]---rtoml v0.11+ supports dumping None to a desired string:rtoml.dumps(data, none_value='@None') :key = [1, 2, 3, "@None", 5]
|
qtoml | bad type '<class 'NoneType'>' for dump_value |
tomllib | module 'tomllib' has no attribute 'dumps' |
None
-like valuesHow the package loads None
-like value in string
Literally <package>.loads('v1 = "null" v2 = "None"')
Loaded as | |
---|---|
toml | {'v1': 'null', 'v2': 'None'} |
tomli/tomli_w | module 'tomli_w' has no attribute 'loads' |
tomlkit | {'v1': 'null', 'v2': 'None'} |
rtoml | {'v1': 'null', 'v2': 'None'}---rtoml v0.11+ supports loading custom None values:rtoml.loads(data, none_value='None') :{'v1': 'null', 'v2': None}rtoml.loads(data, none_value='null') :{'v1': None, 'v2': 'None'} |
qtoml | {'v1': 'null', 'v2': 'None'} |
tomllib | {'v1': 'null', 'v2': 'None'} |
How the package dumps a python dictionary with a heterogenous array.
Literally <package>.dumps({"v": [1, 1.2, True, "string"]})
Dumped value or error | |
---|---|
toml | v = [ 1, 1.2, true, "string",] |
tomli/tomli_w | v = [ 1, 1.2, true, "string",] |
tomlkit | v = [1, 1.2, true, "string"] |
rtoml | v = [1, 1.2, true, "string"] |
qtoml | v = [1, 1.2, true, 'string'] |
tomllib | Dumping not supported |
How the package loads a toml string with a heterogenous array.
Literally <package>.loads('v = [1, 1.2, True, "string"]')
Loaded as | |
---|---|
toml | Not a homogeneous array (line 2 column 1 char 1) |
tomli/tomli_w | {'v': [1, 1.2, True, 'string']} |
tomlkit | {'v': [1, 1.2, True, 'string']} |
rtoml | {'v': [1, 1.2, True, 'string']} |
qtoml | {'v': [1, 1.2, True, 'string']} |
tomllib | {'v': [1, 1.2, True, 'string']} |
How the package dumps a python dictionary with a nested array.
Literally <package>.dumps({"v": [[1], [1, 2]]})
Dumped value or error | |
---|---|
toml | v = [ [ 1,], [ 1, 2,],] |
tomli/tomli_w | v = [ [ 1, ], [ 1, 2, ],] |
tomlkit | v = [[1], [1, 2]] |
rtoml | v = [[1], [1, 2]] |
qtoml | v = [[1], [1, 2]] |
tomllib | Dumping not supported |
How the package loads a toml string with a nested array.
Literally <package>.loads('v = [[1], [1, 2]]')
Loaded as | |
---|---|
toml | {'v': [[1], [1, 2]]} |
tomli/tomli_w | {'v': [[1], [1, 2]]} |
tomlkit | {'v': [[1], [1, 2]]} |
rtoml | {'v': [[1], [1, 2]]} |
qtoml | {'v': [[1], [1, 2]]} |
tomllib | {'v': [[1], [1, 2]]} |
Whether the package preserves the order of the keys while dumps a python dictionary.
Thus, whether <package>.dumps({"c": 1, "a": 2, "b": 3})
yields a string
like c = 1\na = 2\nb = 3\n
.
Order kept? | |
---|---|
toml | Kept |
tomli/tomli_w | Kept |
tomlkit | Kept |
rtoml | Kept |
qtoml | Kept |
tomllib | Dumping not supported |
Whether the package preserves the order of the keys while loads a TOML string.
Thus, whether <package>.loads('c = 1\na = 2\nb = 3\n')
yields
a dictionary with keys in the order of ['c', 'a', 'b']
.
Order kept? | |
---|---|
toml | Kept |
tomli/tomli_w | Kept |
tomlkit | Kept |
rtoml | Kept |
qtoml | Kept |
tomllib | Kept |
How the package dumps Unicode in python
Literally, <package>.dumps({"你好": "世界"})
Dumped value | |
---|---|
toml | "你好" = "世界" |
tomli/tomli_w | "你好" = "世界" |
tomlkit | "你好" = "世界" |
rtoml | "你好" = "世界" |
qtoml | '你好' = '世界' |
tomllib | Dumping not supported |
How the package loads a file with unicode.
The file was created by:
## Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
f.write('"你好" = "世界"\n')
## Use `<package>.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
loaded = <package>.load(f)
Loaded as | |
---|---|
toml | {'你好': '世界'} |
tomli/tomli_w | File must be opened in binary mode, e.g. use open('foo.toml', 'rb') When loaded with rb :{'你好': '世界'} |
tomlkit | {'你好': '世界'} |
rtoml | {'你好': '世界'} |
qtoml | {'你好': '世界'} |
tomllib | File must be opened in binary mode, e.g. use open('foo.toml', 'rb') When loaded with rb :{'你好': '世界'} |
Test the compliance with the standard test suites for valid toml files here:
The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.
Result (toml-test v1.5.0) | |
---|---|
toml | spec/array-0.toml Not a homogeneous array (line 8 column 1 char 261)spec/keys-4.toml Found invalid character in key name: 'c'. Try quoting the key name. (line 2 column 8 char 57)spec/local-time-0.toml Parsed as unexpected data.datetime/no-seconds.toml invalid literal for int() with base 0: '13:37' (line 2 column 1 char 46)datetime/local-time.toml Parsed as unexpected data.datetime/datetime.toml Parsed as unexpected data.comment/tricky.toml Parsed as unexpected data.key/dotted-1.toml Parsed as unexpected data.key/unicode.toml Found invalid character in key name: ''. Try quoting the key name. (line 5 column 2 char 67)key/dotted-2.toml Found invalid character in key name: '"'. Try quoting the key name. (line 7 column 11 char 166)key/quoted-unicode.toml Duplicate keys! (line 3 column 1 char 19)key/dotted-empty.toml Duplicate keys! (line 2 column 1 char 17)key/escapes.toml Parsed as unexpected data.table/empty-name.toml Can't have a keygroup with an empty name (line 1 column 1 char 0)string/raw-multiline.toml Unbalanced quotes (line 20 column 50 char 532)string/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28)string/hex-escape.toml Reserved escape sequence used (line 3 column 1 char 35)string/escape-esc.toml Reserved escape sequence used (line 1 column 1 char 0)string/multiline-quotes.toml Unterminated string found. Reached end of file. (line 27 column 1 char 664)float/zero.toml Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47)array/mixed-int-string.toml Not a homogeneous array (line 1 column 1 char 0)array/nested-double.toml Not a homogeneous array (line 1 column 1 char 0)array/string-with-comma-2.toml string index out of rangearray/mixed-int-float.toml Not a homogeneous array (line 1 column 1 char 0)array/mixed-string-table.toml list index out of rangearray/mixed-int-array.toml Not a homogeneous array (line 1 column 1 char 0)inline-table/multiline.toml Invalid inline table value encountered (line 1 column 1 char 0)inline-table/key-dotted-1.toml Parsed as unexpected data.inline-table/key-dotted-5.toml Not a homogeneous array (line 2 column 1 char 20)inline-table/newline.toml Key name found without value. Reached end of line. (line 5 column 2 char 98)157/187 (83.96%) passed |
tomli/tomli_w | datetime/no-seconds.toml Expected newline or end of document after a statement (at line 2, column 23)key/unicode.toml Invalid statement (at line 3, column 1)string/hex-escape.toml Unescaped '' in a string (at line 3, column 22)string/escape-esc.toml Unescaped '' in a string (at line 1, column 10)inline-table/newline.toml Invalid initial character for a key part (at line 3, column 21)182/187 (97.33%) passed |
tomlkit | datetime/no-seconds.toml Invalid number at line 2 col 25key/unicode.toml Empty key at line 3 col 0string/hex-escape.toml Invalid character 'x' in string at line 3 col 20string/escape-esc.toml Invalid character 'e' in string at line 1 col 8inline-table/newline.toml Empty key at line 3 col 20182/187 (97.33%) passed |
rtoml |
spec/table-9.toml duplicate key: apple for key fruit at line 8 column 1datetime/no-seconds.toml expected a colon, found a newline at line 2 column 26key/unicode.toml unexpected character found: \u{20ac} at line 3 column 1table/array-within-dotted.toml duplicate key: apple for key fruit at line 4 column 1string/hex-escape.toml invalid escape character in string: x at line 3 column 21string/escape-esc.toml invalid escape character in string: e at line 1 column 9inline-table/newline.toml expected a table key, found a newline at line 3 column 21180/187 (96.26%) passed
|
qtoml | spec/string-4.toml Didn't find expected newline (line 7, column 62)spec/string-7.toml Didn't find expected newline (line 7, column 50)datetime/no-seconds.toml can't parse type (line 2, column 20)datetime/milliseconds.toml Didn't find expected newline (line 2, column 27)datetime/datetime.toml Didn't find expected newline (line 4, column 18)comment/after-literal-no-ws.toml can't parse type (line 1, column 4)comment/tricky.toml can't parse type (line 11, column 7)key/unicode.toml '€' cannot begin key (line 3, column 0)string/raw-multiline.toml Didn't find expected newline (line 22, column 3)string/hex-escape.toml \x not a valid escape (line 3, column 43)string/escape-esc.toml \e not a valid escape (line 1, column 33)string/multiline-quotes.toml Didn't find expected newline (line 4, column 26)inline-table/newline.toml ' ' cannot begin key (line 3, column 20)174/187 (93.05%) passed |
tomllib | datetime/no-seconds.toml Expected newline or end of document after a statement (at line 2, column 23)key/unicode.toml Invalid statement (at line 3, column 1)string/hex-escape.toml Unescaped '' in a string (at line 3, column 22)string/escape-esc.toml Unescaped '' in a string (at line 1, column 10)inline-table/newline.toml Invalid initial character for a key part (at line 3, column 21)182/187 (97.33%) passed |
Test the compliance with the standard test suites for invalid toml files here:
Not OK
: The toml file is parsed without error, but expected to fail.OK
: All files are failed to parse, as expected. Showing the lastResult (toml-test v1.5.0) | |
---|---|
toml | Not OK: integer/double-sign-plus.toml incorrectly parsed.Not OK: integer/us-after-bin.toml incorrectly parsed.Not OK: integer/double-sign-nex.toml incorrectly parsed.Not OK: integer/us-after-hex.toml incorrectly parsed.Not OK: integer/us-after-oct.toml incorrectly parsed.Not OK: spec/inline-table-2-0.toml incorrectly parsed.Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.Not OK: datetime/offset-overflow-hour.toml incorrectly parsed.Not OK: control/comment-del.toml incorrectly parsed.Not OK: control/string-del.toml incorrectly parsed.Not OK: 63 more items incorrectly parsed.298/371 (80.32%) passed |
tomli/tomli_w | OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18) 371/371 (100%) passed |
tomlkit | Not OK: control/comment-cr.toml incorrectly parsed.Not OK: control/multi-cr.toml incorrectly parsed.Not OK: control/rawmulti-cd.toml incorrectly parsed.Not OK: control/bare-cr.toml incorrectly parsed.Not OK: table/append-with-dotted-keys-1.toml incorrectly parsed.Not OK: table/overwrite-array-in-parent.toml incorrectly parsed.Not OK: table/append-to-array-with-dotted-keys.toml incorrectly parsed.Not OK: table/append-with-dotted-keys-2.toml incorrectly parsed.Not OK: array/extend-defined-aot.toml incorrectly parsed.Not OK: inline-table/overwrite-09.toml incorrectly parsed.361/371 (97.30%) passed |
rtoml | Not OK: integer/positive-hex.toml incorrectly parsed.Not OK: integer/positive-bin.toml incorrectly parsed.Not OK: integer/positive-oct.toml incorrectly parsed.Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.Not OK: datetime/offset-overflow-hour.toml incorrectly parsed.Not OK: control/comment-del.toml incorrectly parsed.Not OK: control/comment-cr.toml incorrectly parsed.Not OK: control/multi-cr.toml incorrectly parsed.Not OK: control/rawmulti-cd.toml incorrectly parsed.Not OK: control/bare-cr.toml incorrectly parsed.361/371 (97.30%) passed |
qtoml | Not OK: spec/inline-table-2-0.toml incorrectly parsed.Not OK: spec/table-9-1.toml incorrectly parsed.Not OK: spec/table-9-0.toml incorrectly parsed.Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.Not OK: control/comment-del.toml incorrectly parsed.Not OK: control/comment-lf.toml incorrectly parsed.Not OK: control/comment-null.toml incorrectly parsed.Not OK: control/comment-ff.toml incorrectly parsed.Not OK: control/comment-cr.toml incorrectly parsed.Not OK: control/multi-cr.toml incorrectly parsed.Not OK: 14 more items incorrectly parsed.347/371 (93.53%) passed |
tomllib | OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18) 371/371 (100%) passed |
Test the compliance with python tomllib test data (since python 3.11) for valid toml files here:
https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid
The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.
Result (cpython tag 3.12.4) | |
---|---|
toml | apostrophes-in-literal-string.toml Unbalanced quotes (line 1 column 50 char 49)five-quotes.toml Unterminated string found. Reached end of file. (line 7 column 1 char 97)dates-and-times/datetimes.toml Parsed as unexpected data.multiline-basic-str/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28)8/12 (66.67%) passed |
tomli/tomli_w | OK, 12/12 (100%) passed |
tomlkit | OK, 12/12 (100%) passed |
rtoml | OK, 12/12 (100%) passed |
qtoml | apostrophes-in-literal-string.toml Didn't find expected newline (line 3, column 3)five-quotes.toml Didn't find expected newline (line 3, column 3)dates-and-times/datetimes.toml Didn't find expected newline (line 1, column 19)9/12 (75.00%) passed |
tomllib | OK, 12/12 (100%) passed |
Test the compliance with python tomllib test data (since python 3.11) for invalid toml files here:
https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid
Not OK
: The toml file is parsed without error, but expected to fail.OK
: All files are failed to parse, as expected. Showing the lastResult (cpython tag 3.12.4) | |
---|---|
toml | Not OK: invalid-comment-char.toml incorrectly parsed.Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.Not OK: array/unclosed-empty.toml incorrectly parsed.Not OK: array/file-end-after-val.toml incorrectly parsed.Not OK: array/unclosed-after-item.toml incorrectly parsed.Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed.Not OK: inline-table/unclosed-empty.toml incorrectly parsed.41/50 (82.00%) passed |
tomli/tomli_w | OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21) 50/50 (100%) passed |
tomlkit | Not OK: array-of-tables/overwrite-array-in-parent.toml incorrectly parsed.Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.Not OK: dotted-keys/extend-defined-aot.toml incorrectly parsed.Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.Not OK: inline-table/override-val-in-table.toml incorrectly parsed.44/50 (88.00%) passed |
rtoml | Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.49/50 (98.00%) passed |
qtoml | Not OK: non-scalar-escaped.toml incorrectly parsed.Not OK: invalid-comment-char.toml incorrectly parsed.Not OK: table/redefine-2.toml incorrectly parsed.Not OK: table/redefine-1.toml incorrectly parsed.Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed.Not OK: inline-table/override-val-with-table.toml incorrectly parsed.41/50 (82.00%) passed |
tomllib | OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21) 50/50 (100%) passed |
rtoml
Test the speed of loading and dumping the loaded using data
provided by rtoml
https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml
Loading speed | Dumping speed | |
---|---|---|
toml | Excluded (heterogeneous arrays not supported) | Excluded (heterogeneous arrays not supported) |
tomli/tomli_w | 2.14s (5000 iterations) | 0.73s (5000 iterations) |
tomlkit | 39.78s (5000 iterations) | 0.98s (5000 iterations) |
rtoml | 0.37s (5000 iterations) | 0.08s (5000 iterations) |
qtoml | 4.99s (5000 iterations) | 1.87s (5000 iterations) |
tomllib | 2.04s (5000 iterations) | Dumping not supported |
tomli
Test the speed of loading and dumping the loaded using data
provided by tomli
https://github.com/hukkin/tomli/raw/master/benchmark/data.toml
Loading speed | Dumping speed | |
---|---|---|
toml | Excluded (heterogeneous arrays not supported) | Excluded (heterogeneous arrays not supported) |
tomli/tomli_w | 1.41s (5000 iterations) | 0.46s (5000 iterations) |
tomlkit | 24.55s (5000 iterations) | 0.51s (5000 iterations) |
rtoml | 0.32s (5000 iterations) | 0.16s (5000 iterations) |
qtoml | 3.63s (5000 iterations) | 1.25s (5000 iterations) |
tomllib | 1.44s (5000 iterations) | Dumping not supported |
pip install -U toml-bench
toml-bench
toml-bench --datadir /tmp/toml-bench
toml-bench --report ./README.md
BurntSushi/toml-test
)toml-bench --comver 1.0.0
toml-bench --iter 5000
git clone https://github.com/pwwang/toml-bench.git
cd toml-bench
# See https://python-poetry.org/docs/cli/#add
# for how to specify a version constraint
poetry add "tomli=2.0.0"
poetry update
poetry install
poetry run toml-bench