Turbo Upload Service that facilitates robust delivery of clients' fiat- or crypto-funded ANS-104 data items to Arweave.
AGPL-3.0 License
Turbo is a robust, data bundling service that packages ANS-104 "data items" for reliable delivery to Arweave. It is architected to run at scale in AWS, but can be run in smaller scale, Docker-enabled environments via integrations with LocalStack. Additionally, local-development-oriented use cases are supported via integrations with ArLocal.
Turbo is powered by two primary services:
They are composed atop a common set of service dependencies including but not limited to:
Data items accepted by the service can be signed with Arweave, Ethereum, or Solana private keys.
For a compatible development environment, we require the following packages installed on the system:
nvm
yarn
husky
docker
aws
localstack
(optional)docker compose --env-file ./.env.localstack up upload-service
Once all of its dependencies are healthy, the Upload Service will start on port 3000. Visit its /api-docs
endpoint for more information on supported HTTP routes.
NOTE: Database and queue state persistence across service runs are the responsibility of the operator.
With a compatible system, follow these steps to start the Upload Service on its own on your local system:
cp .env.sample .env
(and update values)yarn
yarn build
yarn db:up && yarn db:migrate:latest
yarn start
Developers can alternatively use yarn start:watch
to run the app in development mode with hot reloading provided by nodemon
yarn db:up
: Starts a local docker PostgreSQL container on port 5432yarn db:down
: Tears down local docker PostgreSQL container and deletes the db volumeyarn db:migrate:list
- lists all the migrations applied to the databaseyarn db:migrate:up MIGRATION_NAME
: Runs a specific migration on a local PostgreSQL databaseyarn db:migrate:latest
: Runs migrations on a local PostgreSQL databaseyarn db:migrate:new MIGRATION_NAME
: Generates a new migration fileyarn dotenv -e .env.dev ...
: run any of the above commands against a specific environment fileKnex is used to create and run migrations. To make a migration follow these steps:
Add migration function and logic to schema.ts
Run the yarn command to stage the migration, which generates a new migration script in migrations/
directory
yarn db:migrate:new MIGRATION_NAME
(e.g. yarn db:migrate:new add_id_to_table
)Construct the migration queries in src/db/arch/migrator.ts
Update the generated migration file to call the proper migration script.
Run the migration:
yarn db:migration:latest
or yarn knex migration:up MIGRATION_NAME.TS
Alternatively, run the migration against a specific environment file:
yarn dotenv -e .env.dev yarn db:migrate:latest
You can rollback knex migrations using the following command:
yarn db:migrate:rollback
- rolls back the most recent migrationyarn db:migrate:rollback --all
- rolls back all migrationsyarn db:migrate:down MIGRATION_NAME
- rolls back a specific migrationAdditional knex
documentation can be found here.
To build the container:
docker build --build-arg NODE_VERSION=$(cat .nvmrc |cut -c2-8) --build-arg NODE_VERSION_SHORT=$(cat .nvmrc |cut -c2-3) .
Runs this service, against the most recent version of payment-service
and arlocal
, and local postgres instances.
docker compose up -d
Run just the upload service against migrated local postgres instance.
docker compose up upload-service --build
Unit and integration tests can be run locally or via docker. For either, you can set environment variables for the service via a .env
file:
yarn test:local
- runs unit and integration tests locally against postgres and arlocal docker containersyarn test:unit
- runs unit tests locallyyarn test:docker
- runs integration tests (and unit tests) in an isolated docker container (RECOMMENDED)yarn test:integration:local
- runs the integration tests locally against postgres and arlocal docker containersyarn test:integration:local -g "Router"
- runs targeted integration tests against postgres and arlocal docker containers
watch -n 30 'yarn test:integration:local -g "Router'
- runs targeted integration tests on an interval (helpful when actively writing tests)