omniscient deterministic engine

This library implements:

• a suite of experimental language models, including transformers, recurrent neural networks, state space models, and more
• a plethora of custom, composable layer types, for use in your own models
• custom optimizers, including AdamW, Lion, and Prodigy
• learning rate schedulers
• custom loss functions
• a custom training loop, with gradient checkpointing (broken), L2-normalized gradient clipping
• custom tokenizers, including support for all Huggingface Tokenizers models
• dataset management via generator functions and iterable sampling strategies
• support for one-label, multi-label, one-hot and integer-encoded datasets
• a number of text-decoding strategies, including greedy (argmax), temperature, top-k, top-p, and Mirostat sampling
• object-oriented, extensible design
• tons more

See cli.js for complete usage.

node cli.js \
  --version 6 \
  --batchSize 2 \
  --gradientAccumulationSteps 8 \
  --sampleLength 256 \
  --generateEvery 512 \
  --predictLength 512 \
  --saveEvery 250 \
  --action train

import ODE from 'ode'

const net = await ODE({
    backend: 'webgl', // available backends: ['tensorflow', 'webgl', 'webgpu']
    version: 6
})

await net.init()

const dataSampler = net.ode.samplers.CosmopediaSampler()
await net.train(dataSampler, {
    batchSize: 1,
    gradientAccumulationSteps: 64,
    sampleLength: 256,
    saveEvery: 100
})

const output = await net.generate({
    prompt: 'Once upon a time, ',
    doSample: true,
    temperature: 0.7,
    maxNewTokens: 64,
    repetitionPenalty: 1.2
})

console.log(output)

• [] in cross attention, queries are generated from one embedding, and keys/values from another (A "peer" is all you need)
• [] explore model parallelism, sharding, peer-to-peer capabilities
• [] reinforcement learning, to teach the models "when" to respond, and when to remain quiet
• [] cross-modal models (i.e. a picture of text predicts a character)