This repository contains an experimental demo application that shows how you can add client-side auto-generated captions to Amazon IVS Real-time and Low-latency streams using transformers.js and WebGPU.
MIT-0 License
A demo web application that showcases state-of-the-art client-side transcription, with everything running directly in your browser. By leveraging Transformers.js and ONNX Runtime Web, this demo enables WebGPU-accelerated real-time in-browser transcription for Amazon IVS Low-latency and Real-time streams.
[!WARNING] This is an experimental demo designed exclusively for educational purposes. By using this solution, you understand and accept its risks and limitations.
v20.10.0
and Node package manager (npm).
nvm use
to sync your node version with this project.API_URL
from the deployed serverless infrastructure for this demo.Follow these instructions to run the demo:
npm run deploy:init
npm run deploy:backend
For more details, review the Amazon IVS WebGPU Captions Demo Serverless Infrastructure
npm ci
npm run dev
The following command will deploy the client website to a public cloudfront url.
npm run deploy:website
Replace the PLAYBACK_URL
in src/constants.js with your IVS Playback URL.
Modify the SAMPLE_MODELS
in src/constants.js to add or remove the models shown in the UI. Additional models may be found in the Hugging Face ONNX Community.
{
label: 'Model name'
description: 'A short description of the model.',
value: 'huggingface_model_name', // for example, 'onnx-community/whisper-tiny.en'
sizeInBytes: Model size,
modelOptions: {
dtype: {
encoder_model: 'q4', // 'q4' or 'q8' or 'fp16 or 'fp32' (values may not work with all models)
decoder_model_merged: 'q4', // 'q4' or 'q8' or 'fp16 or 'fp32' (values may not work with all models)
},
device: 'webgpu', // or 'wasm'
},
},
Amazon Interactive Video Service (Amazon IVS) is a managed livestreaming and stream chat solution that is quick and easy to set up, and ideal for creating interactive video experiences. Learn more.