Comments (18)
What do you think of using
tokens
/-t/--tokens
for calculating the number of tokens in a prompt?
Yes! That's a great reason not to use "tokens" to mean authentication tokens, since tokens already means something different in LLM space.
from llm.
This is implemented, next step is to write the docs for it - which is blocked on:
from llm.
Another pattern I'm seeing for configuring API keys is in the (excellent and free) DeepLearning.AI Short Courses. They rely on the dotenv module, and allow more flexible configuration, allowing other API keys to be set.
from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv()) # read local .env file
openai.api_key = os.environ['OPENAI_API_KEY']
That reads the API keys from the .env
file, and requires that the file contents look like OPENAI_API_KEY=sk...
So unfortunately you can't use the same file for both mechanisms. Note if you try to reuse the .env
file, you'll get this rather confusing error:
openai.error.AuthenticationError: <empty message>
I hope they provide a better error message via the discussion at openai/openai-python#464
Back to llm: I suggest supporting (or even switching to) the dotenv approach, since I dare say that people will want to specify multiple API keys for the burgeoning variety of LLMs and plugins out there.
from llm.
This is now blocking this issue, because I need to solve it for other API key providers too:
from llm.
I'm going to start storing keys in user_data_dir
from #7. I'll add a command people can use to manage their keys (if they don't want to use environment variables).
I think I'll leave people who want to use .env
to set that up themselves - it looks like the pip install "python-dotenv[cli]"
tool provides options for managing those already, so I don't need to add it as a dependency to this tool. https://github.com/theskumar/python-dotenv#command-line-interface
You can do dotenv run llm ...
to run the llm
command with environment variables loaded from your .env
.
from llm.
I'm taking inspiration from how Vercel does this: https://til.simonwillison.net/macos/fs-usage
$ cat ~/Library/Application\ Support/com.vercel.cli/auth.json
{
"// Note": "This is your Vercel credentials file. DO NOT SHARE!",
"// Docs": "https://vercel.com/docs/project-configuration#global-configuration/auth-json",
"token": "... redacted ..."
}
I'm going to use the same filename - auth.json
- in my user_data_dir
.
I'll use different keys for the tokens though. Each command/provider will have a default key - "openai" for OpenAI, something else for the PaLM ones.
When using llm ...
the following will happen:
- If the command was called with
--token xyz
andxyz
appears to be a token, it will be used. - If
--token xyz
matches a key in theauth.json
dictionary, the value of that key will be used. This supports having multiple keys so you can do things likellm "some prompt" --token personal
. - Now check for the environment variable -
$OPENAI_API_KEY
or whatever. - If no environment variable, use the default key e.g.
"openai"
inauth.json
.
This design ensures people can over-ride their current environment variable if they want to by specifying --token openai
.
from llm.
I'm not going to have a -t
shortcut for --token
because I want to discourage people from using the --token
option - it feels less secure to me than using environment variables or tokens stored in a file. I also may want to use -t
for something else in the future.
from llm.
How do users get their tokens into the system? They can edit auth.json
directly, but I'll also provide a llm tokens
set of commands for this:
$ llm tokens set openai
Enter token:
$ llm tokens path
~/.share/llm/auth.json
from llm.
Now I'm torn on naming. It would be good if the filename and the commands and the concepts were all consistent with each other.
Some options:
llm/tokens.json
$ llm tokens set openai2
$ llm "pelican names" --token openai2
I worry that "tokens" are less obviously secret than the others.
llm/auth.json
$ llm auth set openai2
$ llm "pelican names" --auth openai2
llm/keys.json
$ llm keys set openai2
$ llm "pelican names" --key openai2
Keys do at least have a clear implication that they should be protected.
llm/secrets.json
$ llm secrets set openai2
$ llm "pelican names" --secret openai2
I feel like secrets could be misunderstood to mean some other concept.
from llm.
How do users get their tokens into the system? They can edit
auth.json
directly, but I'll also provide allm tokens
set of commands for this:$ llm tokens set openai Enter token: $ llm tokens path ~/.share/llm/auth.json
What do you think of using tokens
/ -t/--tokens
for calculating the number of tokens in a prompt?
I know you have ttok
for this usage; but I believe it could be better to integrate this feature in llm
, for it could count the number of tokens on its own and automatically choose the best model based on its context length.
And then having key
or something to set up the API keys.
from llm.
OpenAI call them "API keys". Google call them all sorts of things - gcloud auth print-access-token
and ?key=xxx
in URLs and API keys on https://cloud.google.com/docs/authentication/api-keys and I'm sure I could find more if I kept looking.
I'm going to call them keys, and go with this:
llm/keys.json
$ llm keys set openai2
$ llm "pelican names" --key openai2
from llm.
I'm going with:
/Users/simon/Library/Application Support/io.datasette.llm/keys.json
On macOS at least - that's generated using:
def keys_path():
return os.path.join(user_dir(), "keys.json")
def user_dir():
return user_data_dir("io.datasette.llm", "Datasette")
I plan to move the docs for this tool to llm.datasette.io
shortly.
from llm.
Partly for unit testing convenience I'm going to allow the path to io.datasette.llm/keys.json
to be over-ridden by an environment variable.
from llm.
Prototype - this seems to work well:
diff --git a/llm/cli.py b/llm/cli.py
index 37dd9ed..b7d7112 100644
--- a/llm/cli.py
+++ b/llm/cli.py
@@ -1,9 +1,13 @@
import click
from click_default_group import DefaultGroup
import datetime
+import getpass
import json
import openai
import os
+import pathlib
+from platformdirs import user_data_dir
+import requests
import sqlite_utils
import sys
import warnings
@@ -124,6 +128,50 @@ def init_db():
db.vacuum()
+@cli.group()
+def keys():
+ "Manage API keys for different models"
+
+
+@keys.command()
+def path():
+ "Output path to keys.json file"
+ click.echo(keys_path())
+
+
+def keys_path():
+ return os.path.join(user_dir(), "keys.json")
+
+
+def user_dir():
+ return user_data_dir("io.datasette.llm", "Datasette")
+
+
+@keys.command(name="set")
+@click.argument("name")
+def set_(name):
+ """
+ Save a key in keys.json
+
+ Example usage:
+
+ $ llm keys set openai
+ Enter key: ...
+ """
+ default = {"// Note": "This file stores secret API credentials. Do not share!"}
+ path = pathlib.Path(keys_path())
+ path.parent.mkdir(parents=True, exist_ok=True)
+ value = getpass.getpass("Enter key: ").strip()
+ if not path.exists():
+ path.write_text(json.dumps(default))
+ try:
+ current = json.loads(path.read_text())
+ except json.decoder.JSONDecodeError:
+ current = default
+ current[name] = value
+ path.write_text(json.dumps(current, indent=2) + "\n")
+
+
@cli.command()
@click.option(
"-n",
diff --git a/setup.py b/setup.py
index dc75536..b36718b 100644
--- a/setup.py
+++ b/setup.py
@@ -31,7 +31,13 @@ setup(
[console_scripts]
llm=llm.cli:cli
""",
- install_requires=["click", "openai", "click-default-group-wheel", "sqlite-utils"],
+ install_requires=[
+ "click",
+ "openai",
+ "click-default-group-wheel",
+ "sqlite-utils",
+ "platformdirs",
+ ],
extras_require={"test": ["pytest", "requests-mock"]},
python_requires=">=3.7",
)
from llm.
def keys_path():
return os.path.join(user_dir(), "keys.json")
Since you're using pathlib later in the code, what do you think of making pathlib the default lib to handle it?
The same result can be achieved with return Path.home() / "keys.json"
I suggested the generalization of pathlib in #19 😊.
from llm.
OK, I implemented these two commands:
$ llm keys --help
Usage: llm keys [OPTIONS] COMMAND [ARGS]...
Manage API keys for different models
Options:
--help Show this message and exit.
Commands:
path Output path to keys.json file
set Save a key in keys.json
$ llm keys path --help
Usage: llm keys path [OPTIONS]
Output path to keys.json file
Options:
--help Show this message and exit.
$ llm keys set --help
Usage: llm keys set [OPTIONS] NAME
Save a key in keys.json
Example usage:
$ llm keys set openai
Enter key: ...
Options:
--value TEXT Value to set
--help Show this message and exit.
from llm.
Still needed:
- Update
get_openai_api_key()
to use the new keys mechanism (including checking forOPENAI_API_KEY
) - Documentation, including upgrade documentation for the changelog
- Move
/.llm/log.db
to the new location
from llm.
Since you're using pathlib later in the code, what do you think of making pathlib the default lib to handle it?
Good call, I'll do that as part of cleaning up this code.
from llm.
Related Issues (20)
- Add named chats HOT 1
- Support prefill HOT 8
- Mechanism for recording a different model ID from the one requested HOT 1
- How to handle fake messages that were not part of real coversations? HOT 3
- llm-llamafile is missing from plugin directory
- How to cut off the LLM in chat mode
- IndexError on Windows for llm chat HOT 1
- llm-groq does not support llama 3 HOT 2
- some plugins fail to install with "Connection refused" error HOT 1
- UI around chat history HOT 1
- [plugin] add IBM watsonx
- A rapidly convert Files to Prompts Using Rust
- Enhancement idea: implement a self help
- Asynchronous API support HOT 1
- Add API documentation on how to import and use this tool as a Python library
- Support for GPT-4o HOT 3
- Fix for latest mypy
- Rename the gpt-4-turbo aliases HOT 1
- All I ever get is "insufficient_quota"
- llm 0.14: Can't run <<llm chat>> on Windows 11 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llm.