We want the ability to get aggregated cost metric with filters.

ex:
"is there any way to add openAI keys using helicone API and get the usage costs for each key ?" - Ivo

https://discord.com/channels/1020597994703310878/1021699161831251968/1095740857090592768

Consider adding to https://github.com/radi-cho/awesome-gpt4 if you plan to work with the GPT-4 API.

From OpenAI's API data usage policy:

Any data sent through the API will be retained for abuse and misuse monitoring purposes for a maximum of 30 days, after which it will be deleted (unless otherwise required by law).

Helicone retains all request data at the moment, and this is totally understandable if you wish to debug issues or tune your model. However, if you are developing an application where end-users could potentially send sensitive data to the Helicone API, it would be best to be able to tell users "your request data will be deleted automatically in x time" or immediately, with no TTL.

This setting would be fine to set account-wide, or on messages with a certain header such as sensitive: true.

Helicone has a distinct advantage in being able to track costs per user, model, key etc, but tracking the actual messages themselves may not be necessary. This could also be implemented as a "metadata only mode" where only the following fields are kept:

• time
• total tokens
• user
• model

Automatically detect Anomalies

Example anomalies

• Empty responses
• Repeat offensive (how closely does the response match the input)
• Maybe integrate with Gaurd Rails

We can flag them on the UI or send a push notification.

Copied from #146

Consider a way to invite teammates and other users to a shared account or dashboard. E.g. creating an 'organization' or 'team' and adding users.

https://discord.com/channels/1020597994703310878/1021699161831251968/1085940954877407313

This looks like a cool project! I'd love to try it out quickly, and the easiest way to do that (instead of setting up NPM, Supabase, and the Cloudflare workers) would be to run a docker image. I'd also suggest putting screenshots or a demo gif on the Github README so people can get a sense of the UI without going to the website. Thanks for your work on this and open-sourcing it!

Taken from #139

Azure has the ability for users to deploy their own OpenAI service as part of their cognitive services offerings. By the looks of it, Microsoft will quickly mirror functionality and models as they're made available by OpenAI (e.g. they added chat-turbo the day after it was available).

See below example from the OpenAI cookbook on how the Azure OpenAI endpoints look. Users give their deployment a name and end up with https://my-service-name.openai.azure.com. No indication that this url pattern will change, but not sure.

https://github.com/openai/openai-python/blob/main/README.md#microsoft-azure-endpoints

Potential options:

Pass in the Azure endpoint along with the rest of the Helicone params
Or setup something in the UI for users to specify which API key should go where (either create an Azure service record or send to OpenAI)

updated documentation link, need to provide api_type, api_base and api_version

Credit: @aavetis

error shows as below:

Application error: a client-side exception has occurred (see the browser console for more information).

Something like this?

query {
  heliconeRequest(filter: {response: {id: {eq: "chat-asdfasdf"}}}) {
    id
    response {
      id
    }
  }
}

Thanks!

The authentication spec allows for the "Authorization" header to accept multiple keys. Change the integration from requiring Helicone-Auth to only use the Authorization header.

We want to UX to be like this..

import openai

openai.api_base = "https://oai.hconeai.com/v1"
opeani.api_key = "Bearer <OpenAI API Key>, Bearer helicone-sk-<KEY>"
openai.Completion.create(
  # ...other parameters
)

Make the changes in RequestWrapper.ts and HeliconeHeaders.ts.

Make sure that you take the OpenAI API Key and map it to the correct "Authorization" header so that we don't send OpenAI our Helicone key

Only make changes to the /worker

Streamed requests are currently not showing the price, we need to design our backend to accept this and display the price.

Let's kick off the discussion about how we can best support the ability to track prompt formats and how prompts were constructed.

Our main focus is ensuring the UX is good and there are no major code changes/workflow disruptions to add this logging.

We currently have formatted templates https://docs.helicone.ai/advanced-usage/templated-prompts. But as @ianbicking from HN mentioned, they might want to add formatting details and not have helicone format the prompt for them.

Here is one idea we can do

template = {
    "prompt": "Write an amazing poem about Helicone",
    "promptFormat": "Write an amazing poem about Helicone"
}

serialized_template = json.dumps(template)

openai.Completion.create(
    model="text-davinci-003",
    prompt=serialized_template,
    headers={
        'Helicone-Prompt-Format': 'on',
    }
)

• Get email notification notifications when your app errors or OpenAI is down.

Docs show 2 methods for passing in a user.
This method, tested does not work (user is not stored in the request)

const response = await openai.createCompletion({
  model: "text-davinci-003",
  prompt: "How do I log users?",
  user: "[email protected]",
});

This method works 👍 (user is stored in request)

const configuration = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
  basePath: "https://oai.hconeai.com/v1",
  baseOptions: {
    headers: {
      Helicone-User-Id: "[email protected]",
    },
  },
});

We can do a similarity score between the response and the initial prompt to detect % likelyhood that there was a prompt injection and flag over some default threshold (80%?)

We need to change the average response time metric on the dashboard to use the delay_ms column and not the time diff of response and request

allow CORS so that Helicone can be called within a browser

Hello,

I'm using open ai api like that :

const completion = await this.openaiApi.createChatCompletion({ model: this.model, messages: messages, functions: this.functions, stream: true, temperature: 1 , function_call: "auto" }, { responseType: 'stream' })

After setting basePath and baseOptions following the documentation.

A lot of chunks (not all) are truncated the chunk looks like that : data:

{"id":"chatcmpl-7jOVWL

Instead of that :

data: {"id":"chatcmpl-7jOVWLdYdXUnf2omQlYBBZPOawnRR","object":"chat.completion.chunk","created":1691053322,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"?"},"finish_reason":null}]}

just like helicone, it has its own base and api_key as below:

import os
os.environ['OPENAI_API_KEY'] = 'fk-xxxxxxxxxxxxxxxxxxxxxxxx'
import openai

openai.api_base = "https://openai.api2d.net/v1"

We want to wrap our entire worker in a try catch that will fall back on the catch, where the catch will log the error and then do a best effort to forward the request to OpenAI.

Description: A team member accidentally published the SUPABASE_SERVICE_ROLE_KEY on GitHub, which is a secret key used by the Supabase service to authenticate requests and perform operations on behalf of a service role. This key is used to grant permissions to specific resources within the Supabase project. As a result, the key is now accessible to anyone who has access to the repository, which can lead to potential security breaches and unauthorized access to team resources.

Actions Taken:

• The team member immediately removed the key from the repository to prevent unauthorized access.
• The team member revoked the exposed key from the Supabase service to prevent anyone from using it to access team resources.
• The team member generated a new key and updated the service to use the new key.
• The team reviewed the code to ensure that no other sensitive information is being exposed.
• The team member notified the team about the issue and the actions taken to fix it.

It is important for the team to be aware of the potential risks associated with accidentally publishing sensitive information on public repositories. It is recommended to implement security measures such as:

• Use environment variables to store sensitive information instead of hardcoding them in the code.
• Implement code reviews and automated tools to detect and prevent the publishing of sensitive information.
• Provide training and education to team members on the importance of keeping sensitive information secure.

By taking these steps, the team can help prevent potential security breaches and protect sensitive information from unauthorized access.

Add support for streaming in Anthropic

This is requests per week, we should maybe state per week somewhere

Currently you can only choose 3 months, 1 months, 7 days, 24 hours and 1 hour.

We want to allow our users to select dynamic time ranges

What I'd expect

• truncated text of what was being embedded

What did I observe

• In the Request column, it prints "Invalid Prompt" for whenever the model is text-embedding-ada-002
• On clicking the request and selecting json, the input text and model appear to be working as normal

What's the problem?

• It's confusing so had me verifying if there was a real issue or whether an end user was trying to jail break the model or something

We want to allow users to encrypt their prompts. We will do this by passing in an extra header called helicone-encrypt-prompt and then encrypt the actual prompt that is stored

It would be great to get an overview of token usage by users without exporting the data and doing the calculations elsewhere. This is the main thing I'm using Helicone to keep track of.

Hey!

I was trying to follow the set-up instructions that are currently described in README.md, however, I receive the following errors when trying to execute wrangler dev:

✘ [ERROR] Could not resolve "@supabase/supabase-js"

    src/index.ts:1:45:
      1 │ import { createClient, SupabaseClient } from "@supabase/supabase-js";
        ╵                                              ~~~~~~~~~~~~~~~~~~~~~~~

  You can mark the path "@supabase/supabase-js" as external to exclude it from the bundle, which will remove this error.


✘ [ERROR] Could not resolve "gpt3-tokenizer"

    src/index.ts:11:26:
      11 │ import GPT3Tokenizer from "gpt3-tokenizer";
         ╵                           ~~~~~~~~~~~~~~~~

  You can mark the path "gpt3-tokenizer" as external to exclude it from the bundle, which will remove this error.


✘ [ERROR] Could not resolve "events"

    src/index.ts:12:29:
      12 │ import { EventEmitter } from "events";
         ╵                              ~~~~~~~~

  The package "events" wasn't found on the file system but is built into node.
  Add "node_compat = true" to your wrangler.toml file to enable Node compatibility.


✘ [ERROR] Could not resolve "@supabase/supabase-js"

    src/properties.ts:1:29:
      1 │ import { createClient } from "@supabase/supabase-js";
        ╵                              ~~~~~~~~~~~~~~~~~~~~~~~

  You can mark the path "@supabase/supabase-js" as external to exclude it from the bundle, which will remove this error.


✘ [ERROR] Could not resolve "async-retry"

    src/retry.ts:2:18:
      2 │ import retry from 'async-retry';
        ╵                   ~~~~~~~~~~~~~

  You can mark the path "async-retry" as external to exclude it from the bundle, which will remove this error.


✘ [ERROR] Build failed with 5 errors:

  src/index.ts:1:45: ERROR: Could not resolve "@supabase/supabase-js"
  src/index.ts:11:26: ERROR: Could not resolve "gpt3-tokenizer"
  src/index.ts:12:29: ERROR: Could not resolve "events"
  src/properties.ts:1:29: ERROR: Could not resolve "@supabase/supabase-js"
  src/retry.ts:2:18: ERROR: Could not resolve "async-retry"

How do I best fix these? I might unintentionally be doing something completely stupid as well to get this error.

Any help is appreciated!

Love the caching feature! Makes the request so much quicker when it's seen before, and more predictable too.

However, I noticed an issue when using it together with user properties. Say I have the following headers:

const headers =  {
  "Helicone-Cache-Enabled": "true",
  "Helicone-Property-App": "my-application",
  "Helicone-Property-Session-": "12345689",
},

Helicone-Property-App is static so that's fine, but when the session ID Helicone-Property-Session changes, it no longer uses the cache.

Would love to be able to have the cache ignore certain headers.

We want to have a flag that checks only for 429s

We should also check the response header for the x-ratelimit-reset-request and wait for that amount of time before exponential back off.
https://platform.openai.com/docs/guides/rate-limits/overview

See this thread for more information
https://discord.com/channels/1020597994703310878/1021699161831251968/1128947380134359070

Can we please remove these types of print statements??

print("logging", request, provider)
print("logging", async_log, Provider.OPENAI)

Used a way around to hide these but it would be better to remove these.

When trying to add the helicone proxy to AzureOpenAI with langchain wrapper, getting a consistent resource not found error from openai

from langchain.chat_models import AzureChatOpenAI
helicone_headers = {
        "Helicone-Auth": f"Bearer {helicone_api_key}",
        "Helicone-Property-Env": helicone_env,
        "Helicone-Cache-Enabled": "true",
        "Helicone-OpenAI-Api-Base":
            "https://<model_name>.openai.azure.com/"
    }
self.model = AzureChatOpenAI(
        openai_api_base="https://oai.hconeai.com/v1",
        deployment_name="gpt-35-turbo",
        openai_api_key=<AZURE_OPENAI_API_KEY>,
        openai_api_version="2023-05-15",
        openai_api_type="azure",
        max_retries=max_retries,
        headers=helicone_headers,
        **kwargs,
    )

'error': 'Resource not found'

Calling the model without the wrapper works fine

import openai
openai.api_base = 'https://oai.hconeai.com/v1'
response = openai.Completion.create(
   engine='gpt-35-turbo', 
   prompt='Write a tagline for an ice cream shop. ', 
   max_tokens=10, 
   headers={
        "Helicone-Auth": f"Bearer {os.environ.get('HELICONE_API_KEY')}", 
        "Helicone-Cache-Enabled": "true", 
        "Helicone-OpenAI-Api-Base": "https://<MODEL_NAME>.openai.azure.com"
    }, 
    api_version="2023-05-15", 
    api_type='azure', 
    api_key=<AZURE_OPENAI_API_KEY>, 
    api_base="https://<MODEL_NAME>.openai.azure.com")

<OpenAIObject text_completion id=cmpl-7e5iIQeevfdKYouYoyu2kFHi2xWgK at 0x10593fbf0> JSON: {
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "logprobs": null,
      "text": "2 days left\n\n...a tagline we can"
    }
  ],
  "created": 1689789438,
  "helicone_meta": {},
  "id": "cmpl-7e5iIQeevfdKYouYoyu2kFHi2xWgK",
  "model": "gpt-35-turbo",
  "object": "text_completion",
  "usage": {
    "completion_tokens": 10,
    "prompt_tokens": 11,
    "total_tokens": 21
  }
}

Taken from #140

We should have the ability to easily rate limit a user for X # of requests per day or something like that.

Rate limiting by user. See this super simple implementation of a user ID rate limiter implemented with Upstash

Something like this

{
  `helicone-enable-user-rate-limit': 'true'
  `helicone-user-requests`: "# of requests per cadence"
  `helicone-user-cadence`: "# of seconds" 
}

From @aavetis

Well those 2 params are pretty much the only thing people seem to use at the moment with redis caches :)

Allow users to see p75 and p90 for their metrics like costs

Currently, Helicone only allows people to proxy to the following services:

However there are many other OpenAI compatible-services, and people are building OpenAI interfaces to open-source models like LLama and company, so Helicone could provide metrics without any code modifications.

We should have the ability to give feedback for specific requests.

For a given request we should be able to provide feedback and be able to report human feedback and compare across prompts and models

Taken from #140

We should be able to block bad actors or users. We need some user management section with in the UI to say "restrict traffic for this user"

We should add weekly reports on app performance and cost analytics for that week.

For some reason the cost on user table is populating sometimes...

Request Table is using requestbody for the model but we should use responsebody, this is an issue with the engine key being there and not themodel key

Thanks for the awesome work in this service! Have found it immensely helpful as we start testing our application with users.

I noticed that #309 introduced increased limits, but the gauge is still counting against the old limit. I'm sure this is going to be fixed but just wanted you to be aware 😁

I'm getting the error Invalid API base when using the Helicone-OpenAI-Api-Base header via CURL and Python. I'm confused on how this is happening, so it would be great if the Helicone UI included failed requests too, with the full outgoing request and headers to the external API, and its response. Or maybe just adding a more detailed error message for this problem.

We should have a main search box on the request page

Trying to use the helicone.openai module out of the box, but I get the error:

With an HTML page with the following paragraph:

"Check your DNS Settings. A 523 error means that Cloudflare could not reach your host web server. The most common cause is that your DNS settings are incorrect. Please contact your hosting provider to confirm your origin IP and then make sure the correct IP is listed for your A record in your Cloudflare DNS Settings page."