hendrik2319 / nfcapstone-chatgpt-promptoptimizer Goto Github PK
View Code? Open in Web Editor NEWA tool to test and optimize prompts for ChatGPT API
Home Page: https://chatgpt-promptoptimizer.onrender.com/
A tool to test and optimize prompts for ChatGPT API
Home Page: https://chatgpt-promptoptimizer.onrender.com/
As a User
I need variables in prompts
to be able to test the prompts with different inputs.
The expected use case of such ChatGPT prompts is to get answers / texts to specific input texts. These input texts have to be inserted into the prompt (... template) and the final prompt will be send to ChatGPT to get the answer. To define the position for insert, a variable or placeholder in the prompt is needed. It would be useful to define more than one variable. The default type of inputs should be text / string, but it could be also a list of texts / strings.
alert
messages with FloatingDialogs
As a User
I need a kind of cost control
to be able to limit the charges caused by using the API.
Each call to the API of OpenAI causes charges to the owner of the used organization key.
To limit these costs while using the app, some sort of cost control and documentation of these costs is required.
These charges are based on the amount of language tokens the AI has processed. Each call responses the amount of tokens causes by processing the prompt, by processing the answer (completion) and in total. These tokens should be documented for each call and aggregated up to the Test Scenarios and overall for each user or the entire app.
Usage
)As a User
I need a minimal solution of the project
to be able to get a first impression of the tool.
Build a minimal solution with access to ChatGPT API. A simple frontend with on input field for the prompt, a send button and an output field for the answer of ChatGPT is quite enough.
As a User
I need a sort of rating and evaluation of the API answer
to be able to see easily, if the answer meets my expectations or not.
The app should do some simple rating or evaluation of the answers of the API. For example, it could check, if an answer contains of more than one word or similar things. I should be able to switch these checks on and off.
If all checks of an answer are passed, this answer should displayed with a green color (text or background or an icon). if not, a red mark should be displayed.
As a User
I need logging of test
to be able to see the evolution of a prompt and document the progress.
Each passed test run should produce a log entry with all data of the test run. This data should contain everything needed for the test run, so that the user can change all settings (variable definitions, test cases, ...) at any time and these changes are documented for each test run. Each log entry should contain date and time of the run.
If there are some benchmark values for a test run, a graph of these values should be displayable.
All these log entries together build a Test Scenario.
A Test Scenario should have a name and an author / owner / assigned user.
As a User
I need this app
to be able to optimize my prompts
The base idea of this task is to clearify the base testing procedure and to determine the needed features to do this.
At first I want to test a prompt multiple times (Series of Tests), that means:
The second feature is the ability to define Variables. Each Variable can be a single string or a list of strings. Variables have placeholders in the prompt to define the position, where to insert the value of the Variable. Multiple Variables can be defined and each Variable can have multiple placeholders in the prompt.
The third feature is the definition of Test Cases. Each Test Case is a set of values of Variables, one value for each Variable. As said before, a value can be single string or a list of strings. If a test case contains a list of string for a Variable, multiple answers will be generated for this Test Case.
These Series of Tests, Variable definitions and list of Test Cases build a Test Scenario, that will be stored as a document in a database.
To ensure that no unauthorized access to ChatGPT API can be performed a User Management ist needed too. Otherwise unrestricted access to the app and so to the API will cause uncontrolled raise of costs caused by the API usage. The User Managment should base on widely used accounts like GitHub or Google. Each user should log in with his/her account and have to be granted access by an admin. So three roles are needed: Admin, User and Unknown Account.
To keep costs in perspective, the tokens caused by each API call should be documented. Theses tokens should be documented for each call and aggregated up to the Test Scenarios and overall for each user or the entire app.
We need optimization criteria for prompts and also some kind of evaluation of API responses.
As an Administrative Person
I need a user management
to be able to restrict the access to the app.
A user management should be based on Github accounts. There should be Admins and Users. A newly signed up account is initially an Unknown Account, that should be raised to an User by an Admin. Only Users and Admins can have acccess to the app.
As a User
I need Test Cases
to be able to define the values of variables for a single test run.
In a single test run, a prompt should be tested with multiple values for each variable to obtain a wide range of possible responses from ChatGPT and to determine whether a prompt is stable or not. A stable prompt is one that provides usable responses for all expected inputs.
These values for the variables are summarized in test cases. Each variable can be set with a single string or a list of strings.
As a User
I need a some optimization criteria
to be able to evolve a prompt.
The main purpose of the app is to optimize a ChatGPT prompt. So its essential to have some optimization criteria for each prompt. It could be a numeric value or a rating or a boolean criteria.
A obvious cirteria would be the charges per API call.
Each call to the API of OpenAI causes charges to the owner of the used organization key. These charges are based on the amount of language tokens the AI has processed. Each call responses the amount of tokens causes by processing the prompt, by processing the answer (completion) and in total.
An average value for charges per API call can be computed for each test run. This value can be stored for the corresponding prompt. A chart of this value over all prompt versions in a Test Scenario could be useful.
If more criteria are found in the future, they can be added here.
As a User
I need an API State Display
to be able to see if the API is disabled or not.
Next to the page head line I need a small display if the API is disabled or not.
As a User
I need a dark mode switch
to be able to adapt the look of the app to my browser or system settings
This component should switch between Dartk, Light and System mode.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.