A local-first virtual world sandbox composing AI capabilities

This is a work in progress. Contributions welcome!

So far, this has been tested on Apple Silicon (M2 Max) but should work just as well (better even) with CUDA. At present, it requires around 20 GB VRAM at peak.

git submodule update --init
python3 -m virtualenv venv
venv/bin/pip install torch==2.1.2 setuptools==69.5.1
venv/bin/pip install -r requirements.txt

Download the models (~5GB?)

venv/bin/python scripts/preload.py

venv/bin/python -m latentclay.server

On a Mac:

PYTORCH_ENABLE_MPS_FALLBACK=1 venv/bin/python -m latentclay.server

To set a preferred GPU device (by default it will guess CUDA or Metal as available):

venv/bin/python -m latentclay.server --device ...

See the video at the top and follow along with these guides.

1. Open the web app in your browser (http://localhost:8000 by default)
2. Pick a spot on the floor grid and click to start defining a polygonal selection
3. Click a few more points to define a simple polygon (don't cross lines)
4. Step inside the shape
5. Type / to open the console. Enter "room " and then whatever style of room you'd like (e.g., "/room drawing room")
6. Hit Enter
7. Let the room generate; rooms (currently) take a minute or so
8. In the meantime, hit C to clear your selection

1. Once the room generates, click a point on the floor
2. Type / to open the console again; now it should prefill with "/create"
3. Describe your object, e.g. "lamp", and hit Enter
4. Type C to clear your selection
5. Wait for the object to generate

1. From the inside of a room, spot an area of wall that will be a bit wider than it is high to place a door
2. Click on the wall to start a wall-aligned selection and set the left edge of this region
3. Click to set the right edge of this region
4. Type / to open the console. Enter "door " and then whatever style of doorway you'd like, for instance, "/door doorway". Currently only open doorways/arches are supported (i.e., no hinged/sliding doors yet)
5. Type C to clear selection
6. Wait for the door to generate. This is the quickest of all generations.
7. If all went well, you should be able to see through the doorway to the exterior and walk through it.

1. After creating a room with a doorway, as described above, exit to the exterior
2. Turn around and define a polygonal shape selection, as above, with one edge very nearly lining up to the previous room
3. Create the new room as described above ("/room ...")
4. Once the room generates, type I to toggle door indicators. This will highlight a section of floor red corresponding to the doorway from the previous room (which is now covered by new wall)
5. Use the red door indicator as a guide to draw a corresponding doorway in the new room as described above
6. If all goes well, after generation, you should be able to move back and forth between the two rooms

• W A S D - movement
• / Open Console - to create an object, room, or door, typically after defining a selection with clicks
• C Clear Selection
• I Toggle Door Indicators

• /room <desc> --xz <coords>
• /create <desc> --x <x> --z <z> --ry [rotY] [--scale <scale>]
• /door <desc> --xz <coords>
• /undo - removes the most recently created entity

• improve inference performance - keep models in memory
• handle errors in UI
• removing objects
• tagging objects for reuse/defining classes
• hook up a VLM to autonomously create environments
• hook up an LLM for storytelling
• variable height rooms
• create stairs/ramps and define rooms/surfaces at current/arbitrary height
• exterior environments/terrain
• controllable weather and lighting
• load humanoids with animations
• create, define and launch projectiles
• VR client

Copyright © 2024 Evan Jones