Optimized the Mixtral model by using ipex_llm.optimize_model() to transform it to low-

Issue with loading 4bit ipex-llm Mixtral saved model with generation length about bigdl HOT 3 OPEN

tsantra commented on August 15, 2024

Issue with loading 4bit ipex-llm Mixtral saved model with generation length

from bigdl.

Comments (3)

shane-huang commented on August 15, 2024

what is the version of langchain and ipex-llm are you using?

from bigdl.

shane-huang commented on August 15, 2024

Seems you're running this example on CPU? maybe you can try follow this example to load low bit model. https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/example/CPU/LangChain/low_bit.py

from bigdl.

lzivan commented on August 15, 2024

Hi,

We have reviewed your query and used our sample code to address the problem.

Solution:

We utilized our sample code to resolve the issue. Here is the relevant section of our sample code that addresses your issue:

import argparse
from ipex_llm.langchain.llms import TransformersLLM, TransformersPipelineLLM
from langchain import PromptTemplate, LLMChain
from langchain import HuggingFacePipeline


def main(args):
    question = args.question
    model_path = args.model_path
    low_bit_model_path = args.target_path
    template ="""{question}"""

    prompt = PromptTemplate(template=template, input_variables=["question"])

    llm = TransformersLLM.from_model_id(
        model_id=model_path,
        model_kwargs={"temperature": 0, "max_length": 1024, "trust_remote_code": True},
    )
    llm.model.save_low_bit(low_bit_model_path)
    del llm
    low_bit_llm = TransformersLLM.from_model_id_low_bit(
        model_id=low_bit_model_path,
        tokenizer_id=model_path,
        model_kwargs={"temperature": 0, "max_length": 1024, "trust_remote_code": True}
    )
    llm_chain = LLMChain(prompt=prompt, llm=low_bit_llm)

    output = llm_chain.run(question)
    print("====output=====")
    print(output)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='TransformersLLM Langchain Chat Example')
    parser.add_argument('-m','--model-path', type=str, required=True,
                        help='the path to transformers model')
    parser.add_argument('-t','--target-path',type=str,required=True,
                        help='the path to save the low bit model')
    parser.add_argument('-q', '--question', type=str, default='What is AI?',
                        help='qustion you want to ask.')
    args = parser.parse_args()

    main(args)

And then use this command to run it.

python low_bit.py -m <path_to_model> -t <path_to_target> [-q <your question>]

Runtime Arguments Explained:

-m MODEL_PATH: Required, the path to the model
-t TARGET_PATH: Required, the path to save the low_bit model
-q QUESTION: the question

Output:

After implementing the solution, we obtained the following output:

Artificial Intelligence (AI) refers to the development of computer systems that can perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. AI systems use algorithms and machine learning techniques to analyze data, identify patterns, and make decisions based on that data.

There are different types of AI, including:

1. Narrow AI: Also known as weak AI, this type of AI is designed to perform a specific task, such as voice recognition or image recognition.
2. General AI: Also known as strong AI, this type of AI is designed to perform any intellectual task that a human can do.
3. Superintelligent AI: This is a hypothetical type of AI that is much smarter than the best human minds in virtually all spheres, including scientific creativity, general wisdom, and social skills.

AI is used in various industries, including healthcare, finance, manufacturing, transportation, and education, to improve efficiency, accuracy, and productivity. It can also be used to develop new products and services, enhance customer experiences, and solve complex problems.

However, AI also raises ethical concerns, such as privacy, bias, and job displacement. It is important for organizations and individuals to use AI responsibly and ethically, and to consider the potential impact on society as a whole.
====output=====
What is AI?

Artificial Intelligence (AI) refers to the development of computer systems that can perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. AI systems use algorithms and machine learning techniques to analyze data, identify patterns, and make decisions based on that data.

There are different types of AI, including:

1. Narrow AI: Also known as weak AI, this type of AI is designed to perform a specific task, such as voice recognition or image recognition.
2. General AI: Also known as strong AI, this type of AI is designed to perform any intellectual task that a human can do.
3. Superintelligent AI: This is a hypothetical type of AI that is much smarter than the best human minds in virtually all spheres, including scientific creativity, general wisdom, and social skills.

AI is used in various industries, including healthcare, finance, manufacturing, transportation, and education, to improve efficiency, accuracy, and productivity. It can also be used to develop new products and services, enhance customer experiences, and solve complex problems.

However, AI also raises ethical concerns, such as privacy, bias, and job displacement. It is important for organizations and individuals to use AI responsibly and ethically, and to consider the potential impact on society as a whole.

from bigdl.

Issue with loading 4bit ipex-llm Mixtral saved model with generation length about bigdl HOT 3 OPEN

Comments (3)

Solution:

Output:

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent