When using the relatively recent "Functions" feature of the ChatGPT API, it seems like tiktoken-rs underestimates the total number of tokens in the request. Here's a minimal example request:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a friendly chatbot.\n"
    },
    {
      "role": "assistant",
      "content": "Hello, I am a friendly chatbot!\n"
    },
    {
      "role": "user",
      "content": "What is the weather in New York?"
    },
    {
      "content": "",
      "function_call": {
        "arguments": "{\n  \"city\": \"New York\"\n}",
        "name": "get_weather"
      },
      "role": "assistant"
    },
    {
      "role": "function",
      "name": "get_weather",
      "content": "{\"temperature\": 72, \"conditions\": \"partly_cloudy\"}"
    }
  ],
  "model": "gpt-4-0613",
  "temperature": 0,
  "stream": false
}

I get this response from OpenAI:

{
    // ...
    "usage": {
        "prompt_tokens": 78,
        "completion_tokens": 19,
        "total_tokens": 97
    }
}

...indicating the request consumed 78 tokens for the prompt. However, tiktoken_rs::num_tokens_from_messages returns a value of 66 tokens.

Is this tokenizer library limited to certain models? The reason I ask is that I've been trying to add a new local LLM (codellama:7b-code) to the Zed editor, and while I can get it to go through the motions of working...

...under the surface there are signs that it's not as simple as I'd hoped...

It's been suggested to me that the problem is because the Zed project uses your tiktoken library and that the library would need updating. I'm operating at the very limits of my abilities here, so any thoughts on this would be really appreciated 😉

In particular async-openai is fairly heavy in terms of dependencies for what it's used for here and it blocked compiling my attempted binding for ruby tiktoken_ruby on some platforms because of some sort of difficulty with openssl.

Also side note, thanks for putting together this little library, it made a ruby wrapper much easier gave hints about how it all needs to fit together

OpenAI's spec introduced some new capabilities that led to some larger changes in the async-openai crate. I'm more than happy to contribute this work to tiktoken-rs but I wanted to open the dialogue first about one of the larger structural changes:

The biggest thing (imo) is the structure of ChatCompletionRequestMessage was changed to support a different data type per role.

This makes getting message content less trivial, but seemed like a necessity since OpenAI now supports user messages including images when invoking certain models, so this will require a data structure change. I'm not very familiar with this space outside of general application, so looking for input in terms of how these new user messages should be handled in terms of token counting.

I dont know if im using it wrong but when creating a new BPE it creates around 20MB of memory and never releases it, on top of that the async_openai::get_max_tokens_chat_message function creates a new bpe in it so big memory usage that never releases after every call

Greetings.

How can we use cl100k with this library? I can see that the file for it is in the source directory, but there is no function to load it. Would it be the same as loading the others? If so I can do it myself, I'm just new to this topic and I don't know if there's something more complex that would have to be done.

Thanks.

Following the recent API update of ChatGPT, Async OpenAI has undergone numerous modifications in its codebase, necessitating significant adjustments to align this library with the updated version.

Candle does not support usize, and it is quite troublesome to do type conversion every time.

tiktoken-rs v0.5.3 cannot be built because it requires rustc 1.72.0 or newer, while the currently active rustc version is 1.71.1

There's nothing here that warrants enforcing the latest version (literally):
https://blog.rust-lang.org/2023/08/24/Rust-1.72.0.html

On line 140-141 of lib.rs, there is a comment where the author mentions he tried threading with rayon but noticed it wasn't much faster than python threads.

Currently the python version gets me the token length in ~0.26 seconds while this crate takes ~1.8 seconds so I propose we should add back threading to speed up the process.

Now I am still a bit new to Rust so this post is more to bring suggestions on how would we go integrating threading?

Hey, thanks for the library.

My use case is that I want to split text into tokens, and then split it into chunks based on that. Splitting naively may result in invalid utf-8 (if the token is not at a unicode boundary).

There are different ways to deal with this, in perfect world, I would know which tokens end a unicode character, and only consider those, but that's probably not feasible without major changes.

Alternatively, if I have access to the output of _decode_native, I can for example trim the last token(s) or use from_utf8_lossy.

What are your thoughts?

Hi @zurawiki

Thank you for creating this wonderful project.
I wrote a thin wrapper for tiktoken-rs to make some functions available from the C language.
https://github.com/kojix2/tiktoken-c

I plan to use this in tiktoken-cr.
https://github.com/kojix2/tiktoken-cr

Please let me know if you notice anything.

Thanks.

The code is:

#[test]
fn test_token() {
    let input = "🍌This is a sentence   with spaces, hahhahah haha ha";
    let rke = r50k_base()?;
    let _ = rke.split_by_token(input, true).unwrap();
}

Error:

called `Result::unwrap()` on an `Err` value: incomplete utf-8 byte sequence from index 0

Caused by this line:

It should be String::from_utf8_lossy?

use version 0.1

fn main() {
    let bpe = tiktoken_rs::tiktoken::cl100k_base().unwrap();
    println!("{:?}", bpe.encode_with_special_tokens("hello world"))
}

it ouputs

[15339, 1917]

use version 0.2

fn main() {
    let bpe = tiktoken_rs::cl100k_base().unwrap();
    println!("{:?}", bpe.encode_with_special_tokens("hello world"))
}

it throws error:

dyld[65262]: Library not loaded: @rpath/libpython3.8.dylib
  Referenced from: /Projects/rust/test/target/debug/test
  Reason: tried: '/Projects/rust/test/target/debug/deps/libpython3.8.dylib' (no such file), '/Projects/rust/test/target/debug/libpython3.8.dylib' (no such file), '/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libpython3.8.dylib' (no such file), '/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/libpython3.8.dylib' (no such file), '/lib/libpython3.8.dylib' (no such file), '/usr/local/lib/libpython3.8.dylib' (no such file), '/usr/lib/libpython3.8.dylib' (no such file)
fish: Job 1, 'cargo run' terminated by signal SIGABRT (Abort)

As far as I am able to tell, tiktoken-rs is able to encode into a vector of token references, and then you can decode those token references back into your original text. However my use case is to split my text into decoded tokens. I don't see a way to do this via the api. Having a method like "split_by_tokens" would be super helpful.

Hi @zurawiki

I have a question today.

• encode_with_special_tokens
• encode_ordinary

Could you tell me the difference between the two functions?
They both seem to allow special tokens.

The preview of the gpt-4 turbo is out:

https://help.openai.com/en/articles/8555510-gpt-4-turbo

It would be great to have support for this model!

The crate seem to use async_openai only for messages, but this a pretty small thing that should not require to bring in a whole crate to its dependencies. I just want to use this crate with other implementation of openai api wrapper, not async_openai, but it's the only things that comes in the way