npm install

We use mostly models provided by transformers.js.

The steps to download the models we use:

Set ORT_PERF_DIR is set to root of this project.

Set TJS_DIR to the root of the transformers.js repo.

git clone https://github.com/xenova/transformers.js/

pip install optimum

cd $TJS_DIR
$ORT_PERF_DIR/download-models.sh

ort-perf assumes all models are in the models directory. Copy or link the transformers.js models to that models directory:

cd $ORT_PERF_DIR
mkdir models/
ln -s $TJS_DIR/models models/tjs

npx light-server -s . -p 8888

point your browser to http://localhost:8888/ort-perf.html

npx light-server -s . -p 8888
npx playwright test

The playwright configuration currently is somewhat hard coded. Take a look at playwright.config.js. Also take a look at ort-perf.spec.js and change settings to your needs.

ort-perf is configured by arguments on the url. Currently supported:

the model path, ie. tjs/t5-small/onnx/encoder_model.onnx

the model name.

filter pre-configured models to the given set. Currently supported is default and tjs-demo. The later has all models used by https://xenova.github.io/transformers.js/ with similar parameters.

wasm|webgpu|webnn

Only valid for webnn: cpu|gpu

number of threads to use

1 = capture onnxruntime profile output for 20 runs

How the input data for the model is generated. See ort-perf.html for supported types.

1 = verbose onnxruntime output

Mimimum numbers of queries to run (default=30).

Minimum time to run (default=10sec)

Sequence length if model supports it (default=128)

Encoder sequence length if models supports it (default=128)

1 = start benchmark directly

1 = generate csv output