iinemo / lm-polygraph Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Hi, I have successfully run the information-based and density-based methods.
May I know how to run the ensemble-based uncertainty estimation method?
Hello,
I am not 100% sure, but I believe the entropy calculation is wrong here: https://github.com/IINemo/lm-polygraph/blob/main/src/lm_polygraph/stat_calculators/entropy.py
On line 43, shouldn't you compute the sum instead of the mean?
Also, the entropy should be calculated with base 2. The log probabilities (logprobs) returned by the HuggingFace language models typically use the natural logarithm (base e).
Good day! Is there any way to use the LM-polygraph in an LLM pipeline, created with the Langchain framework?
Thank you for providing the codes for the previously generated text! They have been very helpful, and I've successfully used them for Lexical Similarity analysis. I'm planning to test them for other measurements, including NumSets, Degree matrix (Deg), and Eccentricity.
I noticed that these measurements require two additional statistics: semantic_matrix_entail
and semantic_matrix_contra
. According to the original paper, I know that these are calculated using DeBERTa over generated samples. I'm wondering if there are any short code snippets available to compute these matrices and feed them into the estimator function.
Thanks!
It's now setting both top_k=50 and top_p if we just set top_p, if i understand code correctly. Having both top_k and top_p is kind of weird to me.
Also there seems there is no do_sample; from the hf docs, we could accidentaly get contrastive_search() instead of sample()
Would be nice to test this
Is there a reason why the max length is equal to 256?
In models.py
:
AutoModelForCausalLM.from_pretrained(max_length=256)
Thank you for the amazing framework! Today when I was trying the following codes (simplest demo), I got the error massage saying that:
return UncertaintyOutput(ue[0], input_text, texts[0], model.model_path, estimator.level)
TypeError: UncertaintyOutput.__init__() takes 5 positional arguments but 6 were given
I am wondering whether the framework is ready to use or you are still implementing them?
from lm_polygraph.utils.model import WhiteboxModel
from lm_polygraph.estimators import *
from lm_polygraph.utils.manager import estimate_uncertainty
ue_method = MeanPointwiseMutualInformation()
estimator = SemanticEntropy()
model = WhiteboxModel.from_pretrained(
"bigscience/bloom-560m",
device="cuda:0",
)
input_text = "Who is George Bush?"
estimate_uncertainty(model, ue_method, input_text=input_text)
RT
Thanks again for your work!
I noticed that in your framework, we need to first run the model then get the uncertainty scores. While it's perfectly fine when using free models, it could be expensive when working with charging APIs like ChatGPT.
Specifically, I'm curious if there's a way to obtain uncertainty measures for previously generated texts without having to rerun the model.
Any information or suggestions you can offer in this regard would be greatly appreciated. I look forward to hearing from you and learning more about this possibility.
I was running the polygraph_eval with this example config https://github.com/IINemo/lm-polygraph/blob/main/examples/configs/polygraph_eval_wmt14_ende.yaml
I got a warning about too long string
It didn't failed but i almost sure it would mess up with the results
My wild guess is that stat calculators max_lengths in not connected to max_generated_tokens in ue manager itself. But i didn't really look into it for now
The code
model = WhiteboxModel.from_pretrained(
"tiiuae/falcon-40b-instruct",
cache_dir="~/cache/",
device_map='auto',
offload_folder="offload_folder"
Throws the error You shouldn't move a model when it is dispatched on multiple devices.
While
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b-instruct",
trust_remote_code=True,
cache_dir="~/cache/",
device_map="auto",
offload_folder="offload_folder")
seems to work fine :/
Hi,
thanks for providing the community with this library. I believe uncertainties of LLM queries are an important topic. I tried to play around with this library and am a bit stuck. So I'd like to use a remote model that is accessible through the openai
library. For this, I have to provide a custom OPENAI_API_BASE
and my OPENAI_API_KEY
. However, the library tells me that it doesn't know how to query the remote model?
Here is the code that I drafted given your example:
def main():
print(f":: black box test, using Mistral-7B-Instruct-v0.2 from {os.environ["OPENAI_API_BASE"]}")
model = BlackboxModel(openai_api_key=os.environ["OPENAI_API_KEY"], model_path="Mistral-7B-Instruct-v0.2", parameters={"openai_api_base": os.environ["OPENAI_API_BASE"]})
print(model.parameters)
print(":: using estimator EigValLaplacian")
estimator = EigValLaplacian(verbose=True)
answer = estimate_uncertainty(
model, estimator, input_text="When did Albert Einstein die?"
)
print(">>",answer)
So I get the following error:
:: using estimator EigValLaplacian
Traceback (most recent call last):
File "/home/steinb95/development/lm-polygraph/lm-polygraph/examples/./black_box.py", line 23, in <module>
main()
File "/home/steinb95/development/lm-polygraph/lm-polygraph/examples/./black_box.py", line 16, in main
answer = estimate_uncertainty(
^^^^^^^^^^^^^^^^^^^^^
File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 166, in estimate_uncertainty
man()
File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 400, in __call__
batch_stats = self.calculate(batch_stats, self.stat_calculators, inp_texts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 534, in calculate
raise e
File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 518, in calculate
new_stats = stat_calculator(
^^^^^^^^^^^^^^^^
File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/stat_calculators/sample.py", line 46, in __call__
temperature=model.parameters.temperature,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'temperature'
I tried a couple of things, but I am simply unclear where to supply the temperature?
Best
P
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.