Displaying the traceback isn't useful for this tool.

would be nice

After setting up the OPENAI_API_KEY and llm init-db when I now write llm logs I get an API response to the prompt "logs" (explanation of logs).

What I expected is to see the logfiles.

Enables this kind of pattern:

curl -s 'https://simonwillison.net/2023/May/15/per-interpreter-gils/' | \
  llm --system 'A hilarious joke about this post' --stream

Output just now:

Why did the Python developer compile Python themselves just to test the Per-Interpreter GIL feature?

Because they wanted to thread the needle!

Options for an interactive chat mode:

• It's a new command - llm chat -m gpt-4 for example. This feels a bit odd since the current default command is actually llm chatgpt ... and llm chat feels confusing.
• It's part of the default command: llm --chat -4 starts one running.

Maybe the llm chatgpt command is mis-named, especially since it can be used to work with GPT-4.

I named llm chatgpt that because I thought I'd have a separate command for bard and for llama and so-on, and because I thought the other OpenAI complete APIs (the non-chat ones, like GPT-3) may end up with a separate command.

Originally posted by @simonw in #6 (comment)

Having to write templates in YAML is a bit nasty.

How about this as an option?

llm 'Summarize this: $input' --system "You are GlaDOS" -m gpt4 --save glados

Since this is a CLI tool it's nice to be able to > file.py to save generated Python code.

Problem: it usually comes wrapped in triple backticks.

Solution: a --code option which sets a system prompt to avoid that happening.

The XDG base directory spec has a directory that would be perfect for this tool's database: ~/.local/share/llm.

There are two big advantages to using XDG_DATA_HOME instead of ~/.llm:

• Users can easily configure the directory to use by setting XDG_DATA_HOME
• Limits the explosion of dot directories in the home directory

here's how I implement it in my similar tool, for reference - use XDG_DATA_HOME if present, otherwise default to ~/.local/share/<app_name>

I want to make streaming mode the default - I'm fed up of forgetting to add -s to everything. I don't see any harm in it as a default, people can turn it off with --no-stream if they really want to.

Originally posted by @simonw in #17 (comment)

Following:

• #42

% llm templates list
bad       : this is bad
joke      : Tell a really funny and short joke, surprise me
long      : This is a really long prompt. It's long long long. This is a really long prompt. It's long long long. This is a really long prompt....
recipe    : Suggest a recipe using ingredients: $ingredients  It should be based on cuisine from this country: $country
roast     : 
steampunk : Summarize the following text. Insert frequent satirical steampunk-themed illustrative anecdotes. Really go wild with that. Text to ...
summarize : 
summary   : Summarize this: $input

Some thoughts on that migration:

Should I have the tool perform a one-off migration when you upgrade it, to move ~/.llm/log.db to the new location?

I think not - instead, I'll have llm init-db take an optional argument for starting the database by copying an existing one, then mention that upgrade path in the release notes.

Originally posted by @simonw in #7 (comment)

Inspired by Datasette's: https://docs.datasette.io/en/stable/plugins.html

Refs:

• #32
• #33
• #20

Three initial plugin hooks:

• register_commands(cli) - to register extra commands
• #52
• #53

I'm not going to implement the same one-off plugin mechanism as Datasette, so I'll have to instead teach people how to develop package plugins locally with pip install -e and show them how to create wheel files they can install elsewhere in case they don't want to push packages to PyPI.

I need to mock the OpenAI API calls so I can write tests against the llm "prompt" command.

This is the feature I built llm for. I want to log my interactions to a SQLite database.

Best way to do this will be with Datasette or sqlite-utils but it would be neat to have a basic history command built into llm itself.

I want to make a few schema changes in time for 0.4:

• Add an explicit id integer column so I don't have to remember to select rowid
• chat_id should be an integer, and a foreign key to id
• Ditch the concept of a provider and just use the model column

One thing I'm torn on right now: should I keep the system prompt as a separate column, even though most models other than OpenAI don't have that as a concept?

Following https://til.simonwillison.net/homebrew/packaging-python-cli-for-homebrew

For consistency with the llm logs command.

Related:

• #55
• #57

Related code:

--param should not be a bad option any more.

Blocks:

• #32
• #33
• #20

The ChatGPT endpoints only work for chatting if you manually send back your previous questions and responses:

https://til.simonwillison.net/gpt3/chatgpt-api

This tool could help with that, maybe through a llm chat command?

UPDATE: Or a -c/--continue option for continuing the most recent conversation.

Right now I'm just recording the model that was requested, e.g. gpt-3.5-turbo in the model column.

But... it turns out the response from OpenAI includes this - "model": "gpt-3.5-turbo-0301" - and there are meaningful differences between those model versions, e.g. the latest is gpt-3.5-turbo-0613 but you have to opt into it.

I'd like to record the model that was actually used. Not sure how best to put this in the schema though, since it may only make sense for OpenAI models.

Similar to https://sqlite-utils.datasette.io/en/stable/cli-reference.html

I'm not sure this is actually an issue, as I've developed a workaround, but I thought it was worth bringing up for discussion.

I prefer to keep keys like this in my password manager. Among other things, it allows secure access and consistent sync across machines. I already had a function to access the key, but I don't want to call it on every new shell session as it pops up a prompt in my password manager. I'd prefer to only do that when using the tool, and only the first time in the shell session.

So, I wrote a wrapper function to do that:

llm() {
  if [ -z "$OPENAI_API_KEY" ]; then
    export OPENAI_API_KEY=`open_ai_key`
  fi
  command llm "$@"
}

I use zsh on macOS.

I'm not aware of other patterns that llm could use to look for a key, and you already provide two reasonable ones...but if there was a third way that would obviate the need for my little wrapper, that would be cool! Otherwise, maybe someone else finds this helpful.

would be interesting to add markdown output preview built in.

for now i'm using it with glow https://github.com/charmbracelet/glow to do the same

Inspired by https://twitter.com/goodside/status/1669613516402089984

Idea came from here:

Or... maybe it takes an llm argument which is similar to datasette in that it's an object offering a documented API for various useful things, like looking up configuraiton settings and loading templates and suchlike.

@hookspec
def register_models(llm):
    """Return a list of Models"""

Originally posted by @simonw in #53 (comment)

Made me realize that this tool could work like sqlite-utils in that most features could be available both as Python API methods and as CLI commands.

A command that launches a web server to allow you to both browse your logs and interact with APIs through your browser.

This will allow users to store templates for complex prompts (both as system prompts and regular prompts that have strings interpolated into them) so they can use them in the future.

llm templates edit summary
# An editor opens to edit that prompt
llm --template summary "$(curl -s https://www.example.com/)"

With a shortcut so llm -t summary works too.

So you can default to something other than 3.5 turbo.

Related:

• #17
• #20

Lists available models.

Use cog for this.

If any templates have newlines the output gets very confusing:

% llm templates list
bad       : this is bad
joke      : Tell a really funny and short joke, surprise me
long      : This is a really long prompt. It's long long long. This is a really long prompt. It's long long long. This is a really long prompt. It's long long long. This is a really long prompt....
recipe    : Suggest a recipe using ingredients: $ingredients

It should be based on cuisine from this country: $country
roast     : 
steampunk : Summarize the following text.
Insert frequent satirical steampunk-themed illustrative anecdotes. Really go wild with that.
Text to summarize: $input

summarize : 
summary   : Summarize this: $input

Refs:

• #13 (comment)_

Count tokens with tiktoken and switch to the 16k or 32k models if necessary.

Initially this tool will let you run things against ChatGPT and GPT-4 from the command-line.

Over time I want to introduce Pluggy plugins to allow you to hook it up to all sorts of other language models, including ones that run locally.

But for starters it will do this:

llm "Prompt goes here"

And a streaming variant:

llm "Ten ideas for cheesecakes" -s

Plus use -4 to run against GPT-4, or --model X to specify another model.

It will still be pip install llm and llm prompt ... but I want to call it LLM when I write about it.

As seen on https://sqlite-utils.datasette.io/en/stable/cli.html#running-sql-queries - use sphinx-copybutton.

Do this just before the 0.4 release which will make the /stable/ pages start working.

Being able to paste in ChatGPT JSON transcripts and save them to the database would be really useful - similar to the method described in https://simonwillison.net/2023/Mar/27/ai-enhanced-development/

Would love to see gpt-3.5-turbo-16k support, along with the 0613 models.

See: https://community.openai.com/t/gpt-3-5-turbo-0613-function-calling-16k-context-window-and-lower-prices/263263

Refs:

• #6

In order to find the chat ID that I should use to continue a conversation, llm logs needs to include the rowid in the output.

Since prompt responses can be really long, it would be useful to provide an optional -t/--truncate option for truncating them.

To ensure plugins are installed in the correct virtual environment, similar to datasette install and sqlite-utils install.

Here's where I did it for sqlite-utils:

• simonw/sqlite-utils#483

Needed by plugins:

• #49

This can be handled using the new templates feature instead:

• #23

I just noticed Click already has functionality for app directories, which I can use instead of the extra platformdirs dependency: https://click.palletsprojects.com/en/8.1.x/utils/#finding-application-folders

cfg = os.path.join(click.get_app_dir(APP_NAME), 'config.ini')

It looks like I can pass this io.datasette.llm:

>>> import click
>>> click.get_app_dir("hello")
'/Users/simon/Library/Application Support/hello'
>>> click.get_app_dir("io.datasette.llm")
'/Users/simon/Library/Application Support/io.datasette.llm'