josejuanmartinez / mindcraft Goto Github PK

By recommendation of @ Chad Kesington

• ultralm
• xwin

Past conversations are stored, by a summary of what has been discussed for the last X interactions can be relevant to give more naturality to the answer.

• Change the prompt to include explicit instructions to not make up anything

I need to add non-AWQ support

https://github.com/microsoft/LLMLingua

No AWQ-quantized version available. Maybe we can lift the restriction of having AWQ-only quantized models to allow small, non-quantized ones

prompt_lookup_num_tokens=10

1. First provide examples manually, to finetune the answer style to match those styles
2. Combining the style with personalities and motivations as the feelings of an NPC can change over the time (imagine a character becoming angry, that personality feature should modify their default conversational style)
   3 Then it would be nice to have a mode which could extract from textual files conversations where a character participate. Ideas of things to train the model on:

• "...says/said [CHARACTER_NAME]"
• "[CHARACTER_NAME]: ..."

✅ vLLM
❌Optimum-Nvidia https://huggingface.co/blog/optimum-nvidia - Not installable through pip, only through Docker
❌ Flash Attention with BetterTransformers - better transformers still don't support Mistral or Zephyr
❌https://github.com/pytorch-labs/gpt-fast : It's not a library, they only released code for benchmarking

Serve with ´fastapi´ for remote connections via HTTP

By asking feedback after the interactions, we can save them as success or failure and train based on that data a custom model. Probably finetuning an LLM is too computationally expensive, so maybe training a smaller model like T5

Serve through RPC for interconnectivity between different programming languages using RPC protocol