Compatibility with text-generation-inference about huggingface.js HOT 5 CLOSED

Can you try textGeneration instead of textGenerationStream? Maybe the model itself doesn't support streaming?

from huggingface.js.

NVM

Path was hardcoded on /generate, using stream arg to let the api decide the correct path and it works now

from huggingface.js.

Is this documented somewhere?

I failed to get it woking for the streaming endpoint generate_stream of the https://github.com/huggingface/text-generation-inference , may be I did something wrong?

I may have done something wrong, but could not get it to work (the endpoint itself answers correctly as a stream when called by an API request)

const Hf = new HfInference();
const model = Hf.endpoint("endpointUrl/generate_stream");

  const response = await Hf.textGenerationStream({
    // model: model,   <- needed for custom endpoint?
    inputs: messages,
    parameters: {
      max_new_tokens: 200,
      typical_p: 0.2,
      repetition_penalty: 1,
      truncate: 1000,
      return_full_text: true
    }
  })

from huggingface.js.

If you call endpointUrl directly, does it work?

from huggingface.js.

If you call endpointUrl directly, does it work?

I got it to work, this is how the model endpoint needs to be included.

const Hf = new HfInference();
const model = Hf.endpoint("endpointUrl/generate_stream");

  const response = await model.textGenerationStream({
    inputs: someInput,
    parameters: {
  ...
    }
  })

from huggingface.js.