Predictions scores for the ZeroShotGPTClassifier about scikit-llm HOT 3 CLOSED

@Puturbold there is no way to obtain the prediction scores. HF ZeroShotClassifier is a true classifier, whereas we are using gen models to act as if they were classifiers, but in fact they are just generating a bunch of tokens. Of course we could "ask" the model to return the prediction scores, but that will unlikely represent the actual scores.

from scikit-llm.

I appreciate your quick response. Do you have any suggestions for evaluating labels without a pre-labeled reference?

from scikit-llm.

Probably manual evaluation would still be the best bet if no labeled data is available. Also having prediction scores would not necessarily help, as the model can be incorrect in a very confident way (a high score for a wrong label).

from scikit-llm.