Tokenizer.json compability with ai.djl.huggingface tokenizers broken for sentencepiece based models about vespa HOT 5 OPEN

The older tokenizer file saved on tokenizer.json on the model hub imports fine.

diff tokenizer-works.json tokenizer-fails.json 
71c71,72
<     "add_prefix_space": true
---
>     "prepend_scheme": "always",
>     "split": true
157c158,159
<     "add_prefix_space": true
---
>     "prepend_scheme": "always",
>     "split": true
1000171c1000173,1000174
<     ]
---
>     ],
>     "byte_fallback": false

There is one new introduced parameter byte_fallback in the file that fails to import, which could match up with

data did not match any variant of untagged enum PreTokenizerWrapper

from vespa.

Have they fixed it on HEAD, if so I guess 0.28 will be out soon.

from vespa.

Created deepjavalibrary/djl#3141

from vespa.

Workaround to patch the tokenizer.json file to remove the key vespa-engine/sample-apps#1421

from vespa.