Feature request: vocab.json: serialize it in order of token index about tokenizers HOT 2 CLOSED

This could be possible by having the vocab field be a shallow wrapping of HashMap<String, u32> and then implementing the Serialize trait on the wrapper much like this: https://docs.serde.rs/src/serde/ser/impls.rs.html#353-366, except we'd pass an iterator to serializer.collect_map (https://docs.serde.rs/src/serde/ser/impls.rs.html#364) that iterates in order of token index.

from tokenizers.

Btw as a workaround this is how I am currently doing it, as a post-processing step:

const fs = require("fs");

const o = JSON.parse(fs.readFileSync("./vocab.json", "utf8"));

const entries = Object.entries(o);
entries.sort((a, b) => a[1] - b[1]);

const out = JSON.stringify(
	Object.fromEntries(entries),
	entries.map(x => x[0]),
	"\t"
);
/// ^^ second arg is to order by value. (and not default object order)

fs.writeFileSync("sorted-vocab.json", out);

from tokenizers.