xai-org / grok-1 Goto Github PK

Hi there. Anybody succeeded running the model in FP16 on a RPi? Thanks in advance.

No issue, just a comment on how incredible this is!

Fix torrent link on X

The tracker tracker.coppersurfer.tk should be removed from the magnet link in the README.

It is no longer a working bittorrent tracker, instead the domain redirects to a parking page.

X GOOOOOOOOD!!!!!!

Hi,
Thanks for releasing Grok! Is there any chance we could load the model in 4-bit given how large it is? Do you know if bitsandbytes support is planned (cc @TimDettmers)?
Thanks!

It seems as if the model is being loaded in FP16. I've also noticed how QuantizedWeight8bit is imported in run.py, but not actually used. Is that for runtime quantization with FP16 weights, or is it needed for the released 8bit weights but not actually utilized (due to an oversight?)?

Running python run.py on a single Nvidia GPU it fails with ValueError: Number of devices 1 must equal the product of mesh_shape (1, 8)

Can the nr of devices be adjusted to 1 only?

Sorry to spam you, but I don't know any other way I can reach him 🤷‍♂️
@ibab

遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先遥遥领先

Musk's goal is to liberate human productivity by sharing this LLM with everyone, not just for those experts.
I think the most important part is to let thousands of programmers to join in first.
But we don't even have a hello-world demo for now.
I'd like to help to write documents, and wish to have much more information.
Do you have other sites or tutorials?

Seriously, I would expect that some of the weights were trashed to limit performance.

... for doing the right thing.

Илон Маск, возьми на работу пожалуйста )))

Если надо будет,дообучаюсь. Готов контракт подписать лет на 20 хоть, что после обучения останусь)

Сфера айти, пишу код на с#. Могу другой язык выучить. Матем есть, физика есть)

Пиши :)

LFG

https://discord.gg/9JUT39r

Can we download the original (supposedly bfloat16) weights for fine-tuning? The checkpoint is int8 quantized.

Dear xAI team,

Thank you for this gift to the community! Pure awesomeness.

Huggingface support and a Model Card would be appreciated. Huggingface is like a super cool and very popular model hub. This will help the user with a standard workflow.

Thanks,
Vulcan

Trying to run this on Windows, getting tons of errors...
Also, what GPU card is the min requirement for this ?

I would find it quite useful if the Discussions tab could be enabled on this repository. Then users would have a place to discuss questions, configurations and new ideas, without adding noise here in the issues.

It can be enabled very easily, see Enabling GitHub Discussions on your repository.

Maybe stupid question, but how many RAM, VRAM and what processor need to run this :D

I as a hobbyist am not able to run a 300B model realistically, and fine tuning such a large model is likely even harder. I think Grok 0 weights should be released for people that want to still experiment with Grok but cannot run Grok 1

We are living in Egypt we can not have any device that can open this!!

Not quite sure whether huggingface's gpu match the requirements. If it works, could you provide a huggingface version? Thank you !

not an issue but would be nice if it was in the readme/model.py header:
314B parameters
Mixture of 8 Experts
2 experts used per token
64 layers
48 attention heads for queries
8 attention heads for keys/values
embeddings size: 6,144
rotary embeddings (RoPE)
SentencePiece tokenizer; 131,072 tokens
Supports activation sharding and 8-bit quantization
Max seq length (context): 8,192 tokens

Dear xAI team

Would xAI consider opensourcing the training pipleline or recipe to reproduce models? As I understand the weights and inference part is open. Arguably this is not fully opensource.

The benefits of open sourcing the training pipeline is mutual, researches could experiment and contribute to a more effective training pipeline or model improvements... xAI could benefit from that to get more market share and be the true Open AI!

https://github.com/xai-org/grok/blob/be76c959faa3ee0a6b5fa6770b793ab6e7c9abab/run.py#L66

Surprise! You can't run it on your average desktop or laptop!

Run grok-1 with less than 🔲 420 G VRAM

Run grok on an Mac Studio with an M2 Ultra and 192GB of unified ram: See: llama2.cpp / grok-1 support
@ibab_ml on X

You need a beefy machine to run grok-1

Grok-1 is a true mystical creature. Rumor has it that it lives in the cores of 8 GPU's and that the Model must fit in the VRAM.

This implies that you need a very beefy machine. Very very beefy machine. So beefy...

How do you know if your machine is beefy or not?

Your machine is not beefy if it is not big - the bigger the better, size matters! It has to make the sound of a jet engine when it thinks, also it has to be hot to the touch mostly.

It must also smell like burnt plastic at times. The more big iron, the more heavy the more beefy! If you didn't pay a heavy price for it, such as 100k$++, an arm and a leg, then it is not beefy.

What are some of the working setups?

llama2.cpp:

Mac

• Mac Studio with an M2 Ultra
• 192GB of unified ram.

AMD

• Threadripper 3955WX
• 256GB RAM
• 0.5 tokens per second.

This repo:

Intel + Nvidia

• GPU: 8 x A100 80G
• Total VRAM: 640G
• CPU: 2 x Xeon 8480+
• RAM: 1.5 TB

#168 (comment)

AMD

• GPU: 8 x Instinct MI300X GPU 190G
• Total VRAM: 1520G

#130 (comment)

Other / Container / Cloud

• GPU: 8 x A100 80G
• Total VRAM: 640G
• K8 cluster

#6 (comment)

What can you do about it?

Try: See: llama2.cpp / grok-1 support

What are the other options?

• Rent a GPU cloud instance with sufficient resources
• Subscribe to grok at X (twitter.com)
• Study the blade, save up money
• Get someone to cosplay as grok

What is the Answer to the Ultimate Question of Life, the Universe, and Everything?

#42

Ref:
#168 (comment)
#130 (comment)
#130 (comment)
#125 (comment)
#6 (comment)
ggerganov/llama.cpp#6204 (comment)
ggerganov/llama.cpp#6204 (comment)
ggerganov/llama.cpp#6204 (comment)

See: Discussion
Note: This issue has been edited totally to elevate Issue 42 to serve a much better cause. @xSetech Would you not be tempted to pin this?
Edit: Corrected llama2.cpp inaccuracies

how do fix this error i am facing an error on and i need perfect docs on it we will utilize it

1. How to train, evaluate, and provide the train script
2. Example train datasets
3. Docs of grok's technical details.

C:\Users\MARKELOX\Desktop\grok-main>py run.py
INFO:jax._src.xla_bridge:Unable to initialize backend 'cuda':
INFO:jax._src.xla_bridge:Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': UNIMPLEMENTED: LoadPjrtPlugin is not implemented on windows yet.
INFO:rank:Initializing mesh for self.local_mesh_config=(1, 8) self.between_hosts_config=(1, 1)...
INFO:rank:Detected 1 devices in mesh
Traceback (most recent call last):
File "C:\Users\MARKELOX\Desktop\grok-main\run.py", line 72, in
main()
File "C:\Users\MARKELOX\Desktop\grok-main\run.py", line 63, in main
inference_runner.initialize()
File "C:\Users\MARKELOX\Desktop\grok-main\runners.py", line 282, in initialize
runner.initialize(
File "C:\Users\MARKELOX\Desktop\grok-main\runners.py", line 181, in initialize
self.mesh = make_mesh(self.local_mesh_config, self.between_hosts_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\MARKELOX\Desktop\grok-main\runners.py", line 586, in make_mesh
device_mesh = mesh_utils.create_hybrid_device_mesh(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\MARKELOX\AppData\Local\Programs\Python\Python312\Lib\site-packages\jax\experimental\mesh_utils.py", line 373, in create_hybrid_device_mesh
per_granule_meshes = [create_device_mesh(mesh_shape, granule)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\MARKELOX\AppData\Local\Programs\Python\Python312\Lib\site-packages\jax\experimental\mesh_utils.py", line 302, in create_device_mesh
raise ValueError(f'Number of devices {len(devices)} must equal the product '
ValueError: Number of devices 1 must equal the product of mesh_shape (1, 8)

I tried to load this model in Google colab but it fails to load, is there any way that it will get optimized or something to load in Google colab or any other notebook like paperspace?

Hello everyone.

I've tried to run the pip install, but I'm facing the following error:

ERROR: Could not find a version that satisfies the requirement jaxlib==0.4.25+cuda12.cudnn89; extra == "cuda12_pip" (from jax[cuda12-pip]) (from versions: 0.4.17, 0.4.18, 0.4.19, 0.4.20, 0.4.21, 0.4.22, 0.4.23, 0.4.24, 0.4.25)
ERROR: No matching distribution found for jaxlib==0.4.25+cuda12.cudnn89; extra == "cuda12_pip"

I'm install on MacOsX and in a Ubuntu and faced the same issue.

Anyone else got the same error?

Hello! I am translating llm into different languages, can I come to you?

Here is my work: https://huggingface.co/Vikhrmodels

i have installed python 3.10 and venv. Trying to "pip install -r requirements.txt"

ERROR: Ignored the following versions that require a different python version: 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10
ERROR: Could not find a version that satisfies the requirement jaxlib==0.4.25+cuda12.cudnn89; extra == "cuda12_pip" (from jax[cuda12-pip]) (from versions: 0.4.13, 0.4.14, 0.4.16, 0.4.17, 0.4.18, 0.4.19, 0.4.20, 0.4.21, 0.4.22, 0.4.23, 0.4.24, 0.4.25)
ERROR: No matching distribution found for jaxlib==0.4.25+cuda12.cudnn89; extra == "cuda12_pip"

Is there a scientific paper accompanying this release? I've searched but couldn't find one. I find it odd that the weights would be released but not the research.