scaleapi / llm-engine Goto Github PK
View Code? Open in Web Editor NEWScale LLM Engine public repository
Home Page: https://llm-engine.scale.com
License: Apache License 2.0
Scale LLM Engine public repository
Home Page: https://llm-engine.scale.com
License: Apache License 2.0
Right now, if you don't have prompt,response
as the CSV headers to your input, the fine-tuning job will run, but fail later and not give a clear message. Ideally we'd fail fast.
Hi, can you please add documentation to self-host on runpod? it offers GPU prices cheaper than aws.
in the inference script for a finetune model, there's a model parameter:
from llmengine import Completion
response = Completion.create(
model="llama-2-7b",
prompt="Do you offer in-flight Wi-fi?",
max_new_tokens=100,
temperature=0.2,
)
print(response.json())
inserting the name of the finetune from the get() method as described in the documentation throws an error "NotFoundError: The specified endpoint could not be found.".
what should the model name be?
This is a feature request to add token log probabilities to the response of a Completion.create
call.
cc: @yunfeng-scale
Context: https://twitter.com/mikewadhera/status/1684730279234355200?s=46&t=4hgFpDqJMyzRI1wfZC2dAQ
User currently sees Unable to read dataset from s3.
; we should be surfacing Error tokenizing data. C error: Expected 2 fields in line 277, saw 15
Requested by Biju on Twitter.
What is the problem you're currently running into?
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Llama 3 is now available, let's add to the Model Zoo!
Why do you want this feature?
A clear and concise description of why you want the feature.
Internal user demand.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Does this feature block you from using the project?
How many users will benefit from this feature?
Complexity
Thank you for your contribution to llm-engine
. Please ensure you've given the feature considerable thought before submitting it. Once your feature request is accepted, and you're interested in building it, please mention it so that the maintainers can guide you!
What is the problem you're currently running into?
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
We don't yet have mistral models in the FineTune API.
Why do you want this feature?
A clear and concise description of why you want the feature.
Mistral outperforms llama-2-7b
, so it should be at least equally popular for finetuning.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Add Mistral models (mistral-7b
and mistral-7b-instruct
) to the FineTune API so that users can call
FineTune.create(model="mistral-7b", ...)
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Does this feature block you from using the project?
How many users will benefit from this feature?
Complexity
Thank you for your contribution to llm-engine
. Please ensure you've given the feature considerable thought before submitting it. Once your feature request is accepted, and you're interested in building it, please mention it so that the maintainers can guide you!
Hi Team,
Can you provide a quick solution to the internal server error?
I was trying to fine tune and get this error using my code as well as the code provided in the llm engine documents.
create_fine_tune_response = FineTune.create(
model="llama-2-7b",
training_file="https://scale-demo-datasets.s3.us-west-2.amazonaws.com/sports/sports_training_dataset.csv",
validation_file=None,
hyperparameters={"epochs": "1", "lr": "0.0002"},
suffix="my-first-fine-tune"
)
fine_tune_id = create_fine_tune_response.fine_tune_id
UnknownError Traceback (most recent call last)
in <cell line: 1>()
----> 1 create_fine_tune_response = FineTune.create(
2 model="llama-2-7b",
3 training_file="https://scale-demo-datasets.s3.us-west-2.amazonaws.com/sports/sports_training_dataset.csv",
4 validation_file=None,
5 hyperparameters={"epochs": "1", "lr": "0.0002"},
1 frames
/usr/local/lib/python3.10/dist-packages/llmengine/api_engine.py in post_sync(cls, resource_name, data, timeout)
105 )
106 if response.status_code != 200:
--> 107 raise parse_error(response.status_code, response.content)
108 payload = response.json()
109 return payload
UnknownError: Internal Server Error: <class 'AttributeError'>: 'CreateFineTuneResponse' object has no attribute 'artifact_id'
Dear llm-engine developer,
Greetings! I am Jimmy, a community developer and volunteer at InternLM. Your work has been immensely beneficial to me, and I believe it can be effectively utilized in InternLM as well. Welcome to add Discord https://discord.gg/gF9ezcmtM3 . I hope to get in touch with you.
Best regards,
Jimmy
potentially put into vLLM vllm-project/vllm#914
Why do you want this feature?
TensorRT-LLM boasted some good performance gain https://github.com/NVIDIA/TensorRT-LLM
Need to compare RetNet and llama 2 models at comparable parameter #
Potentially implement in vLLM. Implementation would be tricky but I think doable.
Hi,
Is there a way to change the frequency_penality or logit bias when sending a completion request?
from llmengine import Completion
ImportError: cannot import name 'Completion' from partially initialized module 'llmengine' (most likely due to a circular import)
pip list:
scale-llm-engine 0.0.0b7
When calling FineTune.create with an arbitrary model and CSV a "NotFoundError" is thrown. When investigating, the Scale AI Spellbook API endpoint at https://api.spellbook.scale.com/v1/llm/fine-tune throws an 404.
Used version: 0.0.0b5
Code:
from llmengine import FineTune
import os
import yaml
if __name__ == "__main__":
with open("environments/spellbook_api_key.yaml", "r") as stream:
os.environ.update(yaml.safe_load(stream))
response = FineTune.create(
model="llama-2-7b",
training_file="storage.googleapis.com/llama2-finetune-test/finetune_dataset.csv",
)
print(response.json())
I believe we could do < 20s for llama 2 7b models.
I installed the scale-llm engine using pip for windows and am currently running it on python. To test the engine, I ran the example code shown on the website for the engine. However, I ran into the following error.
Below is the code that was used, along with the code that raised the error
Any help would be appreciated
Great job with this repo. I was able to finetune Llama-2 and it certainly seems to have an effect.
Unfortunately the finetune silently accepts all inputs and the documentation states that you simply truncate inputs to max length. But it's not specified anywhere what's LLama-2's max length. Originally Meta released it with a bug that caused max length to be 2048 while the native max length seems to be 4096. So which is it?
Also, I tested my finetune model's completion code with inputs as big as 12,000 tokens and it still makes a completion. So I assume you truncate there as well? Only taking the tail of the prompt, presumably?
tldr:
This is a feature request to add an on_inference_ready
callback to Model.create
, which sends a GET request to the provided endpoint when the deployed model is ready for inference.
Currently, you need the SCALE_API_KEY
env var set. This is generally fine, but is annoying for say notebook users.
We should allow setting a module-level API key that, if present, supersedes the env var value, e.g.:
import llmengine
llmengine.api_key = "abc"
Current Model Zoo page doesn't support llama 2 70b:
Hi,
Thanks for open-sourcing the code.
I was wondering how it compares in terms of throughput with existing inference frameworks like https://github.com/huggingface/text-generation-inference and https://github.com/vllm-project/vllm , do we have any benchmarks?
Are we downloading the data from s3 immediately? I am creating presigned URLs that should last long enough, but my model run failed after a while...
2023-07-24 21:34:49,770 INFO [train_llm_engine] [train_llm_engine.py:113] - run_id: ft-civestsdoujg03t1akcg, job_status: RUNNING
...
2023-07-24 22:39:14,368 INFO [train_llm_engine] [train_llm_engine.py:113] - run_id: ft-civestsdoujg03t1akcg, job_status: RUNNING
2023-07-24 22:40:14,750 INFO [train_llm_engine] [train_llm_engine.py:113] - run_id: ft-civestsdoujg03t1akcg, job_status: FAILURE
Get events:
(Pdb) FineTune.get_events('ft-civestsdoujg03t1akcg')
GetFineTuneEventsResponse(events=[LLMFineTuneEvent(timestamp=1690238353.925053, message='Unable to read dataset from s3.', level='error')])
If we're continuously reading from s3 would be better to download up front
What is the problem you're currently running into?
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Add Mistral-7B to LLM Engine.
Why do you want this feature?
A clear and concise description of why you want the feature.
https://mistral.ai/news/announcing-mistral-7b/
Describe the solution you'd like
A clear and concise description of what you want to happen.
Mistral-7B should appear in the model zoo.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
N/A
Additional context
Add any other context or screenshots about the feature request here.
Does this feature block you from using the project?
How many users will benefit from this feature?
Complexity
Thank you for your contribution to llm-engine
. Please ensure you've given the feature considerable thought before submitting it. Once your feature request is accepted, and you're interested in building it, please mention it so that the maintainers can guide you!
we currently use on-demand instances and spot instances are cheaper with lower availability https://aws.amazon.com/ec2/spot/instance-advisor/ so with lower cold start time we could give it a try
a big thanks for the LLM engine, i don't know if there's a provision for fine-tuning LLM models using local GPUs/ infrastructure . kindly help me with that.
`--------------------------------------------------------------------------
UnauthorizedError Traceback (most recent call last)
/data/MBBS_pharma/fineTuningLLAMAScienceQA.ipynb Cell 15 line 1
----> 1 response = FineTune.create(
2 model="llama-2-7b",
3 training_file=r"/data/MBBS_pharma/trainScience.csv",
4 validation_file=r"/data/MBBS_pharma/valScience.csv",
5 hyperparameters={
6 'lr':2e-4,
7 },
8 suffix='science-qa-llama'
9 )
10 run_id = response.id
File /data/MBBS_pharma/mbbsPharma/lib64/python3.9/site-packages/llmengine/fine_tuning.py:151, in FineTune.create(cls, model, training_file, validation_file, hyperparameters, wandb_config, suffix)
35 """
36 Creates a job that fine-tunes a specified model with a given dataset.
37
ref='/data/MBBS_pharma/mbbsPharma/lib64/python3.9/site-packages/llmengine/fine_tuning.py:0'>0;32m (...)
141
142 """
143 request = CreateFineTuneRequest(
144 model=model,
145 training_file=training_file,
ref='/data/MBBS_pharma/mbbsPharma/lib64/python3.9/site-packages/llmengine/fine_tuning.py:0'>0;32m (...)
...
--> 107 raise parse_error(response.status_code, response.content)
108 payload = response.json()
109 return payload
UnauthorizedError: Invalid API Key.
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...`
Add CI step to master that runs mkdocs gh-deploy
Allow users to checkpoint during training and then use various checkpoints during inference
We are doing some maintenance for LLM Engine fine-tuning events. See email text below:
Dear Spellbook users,
We are performing some maintenance of LLM Engine this week. If you are not an LLM Engine user, you can disregard this email. Otherwise, please read on.
How Will You Be Affected?
During the maintenance window, any fine-tuned models that you’ve created will be unavailable to serve traffic. Requests sent to those models will return with a HTTP 503 error. You will be able to create fine-tunes throughout the maintenance window, but the resulting models will be subject to the maintenance schedule downtime.
Note that the Completions API for base models is unaffected. Also, any files uploaded through the Files API will be unaffected. There is no loss of any of your metadata or data as a result of this maintenance.
Maintenance Schedule
LLM Engine fine-tuned models will be unavailable between Monday August 28th 22:00 PDT and Thursday August 31st 09:00 PDT. After 09:00 PDT on August 31st, fine-tuned models will be available again.
Why is this Maintenance Necessary?
This maintenance is necessary to roll out performance enhancements that will increase throughput and reduce cost/token for fine-tuned endpoints. As we prepare to transition LLM Engine out of evaluation mode and into production, these enhancements are necessary to scale to your production workloads.
What Do You Need to Do?
We kindly request that you take the following steps:
Thank you for using LLM Engine. We look forward to delivering an even better experience after this maintenance.
Thanks,
-Yi
We removed this from our internal version (we'll be moving to public repo-first development soon!), we just need to remove from the public repo too:
llm-engine/server/llm_engine_server/common/dtos/llms.py
Lines 117 to 119 in b2588ea
For LLaMa-13b (non-chat version) documentation states that data must be formatted as a CSV file that includes two columns: prompt and response.
How do we have to prepare data for the chat-version?
Hey everyone. This stems from an older, closed issue: #170.
The post_sync method in api_engine.py concatenates parts of the URL to make the API call against using the os.path.join Python method.
See:
@classmethod
def post_sync(cls, resource_name: str, data: Dict[str, Any], timeout: int) -> Dict[str, Any]:
api_key = get_api_key()
response = requests.post(
os.path.join(LLM_ENGINE_BASE_PATH, resource_name),
json=data,
timeout=timeout,
headers={"x-api-key": api_key},
auth=(api_key, ""),
)
if response.status_code != 200:
raise parse_error(response.status_code, response.content)
payload = response.json()
return payload
However, that method is designed to join paths on file systems depending on the OS Python is being executed on. When on Windows, this results in the path being joined into a URL where one slash is actually a backslash since that is the "divider" used on the Windows file system.
Long story short: bug that breaks all API calls when using package on Windows since os.path.join is used which puts a backslash instead of a slash into the endpoint URL
Is there any thought/desire to setup a helm chart for gke deployment as well?
Describe the bug
I'm unable to pass in PEFT configs when doing FineTune.create()
. The documentation states that it should be a "dict of parameters" but when I pass in a dictionary I get the following error:
llmengine.errors.UnknownError: Internal Server Error: <class 'launch.api_client.exceptions.ApiValueError'>: Invalid inputs given to generate an instance of <class 'launch.api_client.model.create_fine_tune_request.CreateFineTuneRequest.MetaOapg.properties.hyperparameters.MetaOapg.additional_properties'>. None of the anyOf schemas matched the input data.
Looking through the code it seems like only strings, ints, or floats are accepted, so I tried to convert the dictionary to a string but that results in this error later on in the pipeline when the finetuning is actually being done:
'{"events": [{"timestamp": 1697248409.2646654, "message": "\'str\' object has no attribute \'get\'", "level": "error"}]}'
I also tried passing in a LoraConfig
object from HuggingFace but that wasn't accepted either. Am I missing something very obvious? How do I pass in my PEFT configs?
LLM Engine Version
System Version
Minimal Reproducible Example
Running the code snippet as is should do it. I made some dummy test files for this example.
from llmengine import FineTune
response = FineTune.create(
model = "llama-2-7b",
training_file = "file-963KxGaPu8WET25",
validation_file = "file-qU5_mPTaRlmIuLK",
hyperparameters = {
"lr": 2e-3,
"epochs": 5,
"peft_config": {
"r": 8,
"lora_alpha": 8,
"lora_dropout": 0.0
}
},
suffix = "foofoobarbar"
)
Please advise, thank you!
Currently Completion.create
only supports the option to adjust sampling via temperature - and doesn't support greedy decoding.
This is a feature request to support the following options - pulled from the TGI text_generation client.generate docstring.
"""
do_sample (`bool`):
Activate logits sampling
max_new_tokens (`int`):
Maximum number of generated tokens
best_of (`int`):
Generate best_of sequences and return the one if the highest token logprobs
repetition_penalty (`float`):
The parameter for repetition penalty. 1.0 means no penalty. See [this
paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
return_full_text (`bool`):
Whether to prepend the prompt to the generated text
seed (`int`):
Random sampling seed
stop_sequences (`List[str]`):
Stop generating tokens if a member of `stop_sequences` is generated
temperature (`float`):
The value used to module the logits distribution.
top_k (`int`):
The number of highest probability vocabulary tokens to keep for top-k-filtering.
top_p (`float`):
If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or
higher are kept for generation.
"""
The docs state that the training_file
parameter of the FineTune.create()
method should be a "publicly accessible URL to a CSV file for training".
Does that mean that we can only fine-tune a model on data hosted publicly and not locally? It kind of defeats the purpose of experimenting with data we may not want to make public just yet.
If not, then it would be great if the fine tuning jobs failed earlier because right now, I am testing it with a local CSV file and it is simply running for a long time, and it is unclear if the failure is caused by a bad input or not.
Allow r
and alpha
to be set in Create
request
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.