simonw / llm Goto Github PK
View Code? Open in Web Editor NEWAccess large language models from the command-line
Home Page: https://llm.datasette.io
License: Apache License 2.0
Access large language models from the command-line
Home Page: https://llm.datasette.io
License: Apache License 2.0
This will allow users to store templates for complex prompts (both as system prompts and regular prompts that have strings interpolated into them) so they can use them in the future.
llm templates edit summary
# An editor opens to edit that prompt
llm --template summary "$(curl -s https://www.example.com/)"
With a shortcut so llm -t summary
works too.
Being able to paste in ChatGPT JSON transcripts and save them to the database would be really useful - similar to the method described in https://simonwillison.net/2023/Mar/27/ai-enhanced-development/
This is the feature I built llm
for. I want to log my interactions to a SQLite database.
After setting up the OPENAI_API_KEY
and llm init-db
when I now write llm logs
I get an API response to the prompt "logs" (explanation of logs).
What I expected is to see the logfiles.
I need to mock the OpenAI API calls so I can write tests against the llm "prompt"
command.
Refs:
I just noticed Click already has functionality for app directories, which I can use instead of the extra platformdirs
dependency: https://click.palletsprojects.com/en/8.1.x/utils/#finding-application-folders
cfg = os.path.join(click.get_app_dir(APP_NAME), 'config.ini')
It looks like I can pass this io.datasette.llm
:
>>> import click
>>> click.get_app_dir("hello")
'/Users/simon/Library/Application Support/hello'
>>> click.get_app_dir("io.datasette.llm")
'/Users/simon/Library/Application Support/io.datasette.llm'
Some thoughts on that migration:
Should I have the tool perform a one-off migration when you upgrade it, to move
~/.llm/log.db
to the new location?I think not - instead, I'll have
llm init-db
take an optional argument for starting the database by copying an existing one, then mention that upgrade path in the release notes.
Originally posted by @simonw in #7 (comment)
I want to make a few schema changes in time for 0.4:
id
integer column so I don't have to remember to select rowid
chat_id
should be an integer, and a foreign key to id
provider
and just use the model
columnOne thing I'm torn on right now: should I keep the system
prompt as a separate column, even though most models other than OpenAI don't have that as a concept?
Count tokens with tiktoken and switch to the 16k or 32k models if necessary.
It will still be pip install llm
and llm prompt ...
but I want to call it LLM when I write about it.
Refs:
In order to find the chat ID that I should use to continue a conversation, llm logs
needs to include the rowid
in the output.
Since prompt responses can be really long, it would be useful to provide an optional -t/--truncate
option for truncating them.
Options for an interactive chat mode:
- It's a new command -
llm chat -m gpt-4
for example. This feels a bit odd since the current default command is actuallyllm chatgpt ...
andllm chat
feels confusing.- It's part of the default command:
llm --chat -4
starts one running.Maybe the
llm chatgpt
command is mis-named, especially since it can be used to work with GPT-4.I named
llm chatgpt
that because I thought I'd have a separate command forbard
and forllama
and so-on, and because I thought the other OpenAI complete APIs (the non-chat ones, like GPT-3) may end up with a separate command.
Originally posted by @simonw in #6 (comment)
Use cog for this.
Lists available models.
The XDG base directory spec has a directory that would be perfect for this tool's database: ~/.local/share/llm
.
There are two big advantages to using XDG_DATA_HOME
instead of ~/.llm
:
XDG_DATA_HOME
here's how I implement it in my similar tool, for reference - use XDG_DATA_HOME
if present, otherwise default to ~/.local/share/<app_name>
A command that launches a web server to allow you to both browse your logs and interact with APIs through your browser.
Do this just before the 0.4 release which will make the /stable/ pages start working.
If any templates have newlines the output gets very confusing:
% llm templates list
bad : this is bad
joke : Tell a really funny and short joke, surprise me
long : This is a really long prompt. It's long long long. This is a really long prompt. It's long long long. This is a really long prompt. It's long long long. This is a really long prompt....
recipe : Suggest a recipe using ingredients: $ingredients
It should be based on cuisine from this country: $country
roast :
steampunk : Summarize the following text.
Insert frequent satirical steampunk-themed illustrative anecdotes. Really go wild with that.
Text to summarize: $input
summarize :
summary : Summarize this: $input
To ensure plugins are installed in the correct virtual environment, similar to datasette install
and sqlite-utils install
.
Here's where I did it for sqlite-utils
:
Needed by plugins:
Right now I'm just recording the model that was requested, e.g. gpt-3.5-turbo
in the model
column.
But... it turns out the response from OpenAI includes this - "model": "gpt-3.5-turbo-0301"
- and there are meaningful differences between those model versions, e.g. the latest is gpt-3.5-turbo-0613
but you have to opt into it.
I'd like to record the model that was actually used. Not sure how best to put this in the schema though, since it may only make sense for OpenAI models.
This can be handled using the new templates feature instead:
Having to write templates in YAML is a bit nasty.
How about this as an option?
llm 'Summarize this: $input' --system "You are GlaDOS" -m gpt4 --save glados
cat myfile.py | llm --system 'Explain this code'
would be interesting to add markdown output preview built in.
for now i'm using it with glow
https://github.com/charmbracelet/glow to do the same
The ChatGPT endpoints only work for chatting if you manually send back your previous questions and responses:
https://til.simonwillison.net/gpt3/chatgpt-api
This tool could help with that, maybe through a llm chat
command?
UPDATE: Or a -c/--continue
option for continuing the most recent conversation.
I'm not sure this is actually an issue, as I've developed a workaround, but I thought it was worth bringing up for discussion.
I prefer to keep keys like this in my password manager. Among other things, it allows secure access and consistent sync across machines. I already had a function to access the key, but I don't want to call it on every new shell session as it pops up a prompt in my password manager. I'd prefer to only do that when using the tool, and only the first time in the shell session.
So, I wrote a wrapper function to do that:
llm() {
if [ -z "$OPENAI_API_KEY" ]; then
export OPENAI_API_KEY=`open_ai_key`
fi
command llm "$@"
}
I use zsh on macOS.
I'm not aware of other patterns that llm
could use to look for a key, and you already provide two reasonable ones...but if there was a third way that would obviate the need for my little wrapper, that would be cool! Otherwise, maybe someone else finds this helpful.
Would love to see gpt-3.5-turbo-16k support, along with the 0613 models.
would be nice
Enables this kind of pattern:
curl -s 'https://simonwillison.net/2023/May/15/per-interpreter-gils/' | \
llm --system 'A hilarious joke about this post' --stream
Output just now:
Why did the Python developer compile Python themselves just to test the Per-Interpreter GIL feature?
Because they wanted to thread the needle!
Best way to do this will be with Datasette or sqlite-utils
but it would be neat to have a basic history command built into llm
itself.
Following:
% llm templates list
bad : this is bad
joke : Tell a really funny and short joke, surprise me
long : This is a really long prompt. It's long long long. This is a really long prompt. It's long long long. This is a really long prompt....
recipe : Suggest a recipe using ingredients: $ingredients It should be based on cuisine from this country: $country
roast :
steampunk : Summarize the following text. Insert frequent satirical steampunk-themed illustrative anecdotes. Really go wild with that. Text to ...
summarize :
summary : Summarize this: $input
I want to make streaming mode the default - I'm fed up of forgetting to add
-s
to everything. I don't see any harm in it as a default, people can turn it off with--no-stream
if they really want to.
Originally posted by @simonw in #17 (comment)
I'm not going to implement the same one-off plugin mechanism as Datasette, so I'll have to instead teach people how to develop package plugins locally with pip install -e
and show them how to create wheel files they can install elsewhere in case they don't want to push packages to PyPI.
Since this is a CLI tool it's nice to be able to > file.py
to save generated Python code.
Problem: it usually comes wrapped in triple backticks.
Solution: a --code
option which sets a system prompt to avoid that happening.
Initially this tool will let you run things against ChatGPT and GPT-4 from the command-line.
Over time I want to introduce Pluggy plugins to allow you to hook it up to all sorts of other language models, including ones that run locally.
But for starters it will do this:
llm "Prompt goes here"
And a streaming variant:
llm "Ten ideas for cheesecakes" -s
Plus use -4
to run against GPT-4, or --model X
to specify another model.
For consistency with the llm logs
command.
Idea came from here:
Or... maybe it takes an
llm
argument which is similar todatasette
in that it's an object offering a documented API for various useful things, like looking up configuraiton settings and loading templates and suchlike.@hookspec def register_models(llm): """Return a list of Models"""
Originally posted by @simonw in #53 (comment)
Made me realize that this tool could work like sqlite-utils in that most features could be available both as Python API methods and as CLI commands.
As seen on https://sqlite-utils.datasette.io/en/stable/cli.html#running-sql-queries - use sphinx-copybutton
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.