river

Fixed

• Fixed release script that caused CLI to become uninstallable because its reference to rivershared wasn't updated. PR #541.

Changed

• Producer's logs are quieter unless jobs are actively being worked. PR #529.

Fixed

• River CLI now accepts postgresql:// URL schemes in addition to postgres://. PR #532.

Fixed

• Derive all internal contexts from user-provided Client context. This includes the job fetch context, notifier unlisten, and completer. PR #514.
• Lowered the go directives in go.mod to Go 1.21, which River aims to support. A more modern version of Go is specified with the toolchain directive. This should provide more flexibility on the minimum required Go version for programs importing River. PR #522.

Fixed

• database/sql driver: fix default value of scheduled_at for InsertManyTx when it is not specified in InsertOpts. PR #504.
• Change ColumnExists query to respect search_path, thereby allowing migrations to be runnable outside of default schema. PR #505.

Added

• Expose Driver on Client for additional River Pro integrations. This is not a stable API and should generally not be used by others. PR #497.

Fixed

• Include pending state in JobListParams by default so pending jobs are included in JobList / JobListTx results. PR #477.
• Quote strings when using Client.JobList functions with the database/sql driver. PR #481.
• Remove use of filepath for interacting with embedded migration files, fixing the migration CLI for Windows. PR #485.
• Respect ScheduledAt if set to a non-zero value by JobArgsWithInsertOpts. This allows for job arg definitions to utilize custom logic at the args level for determining when the job should be scheduled. PR #487.

Fixed

• Migration version 005 has been altered so that it can run even if the river_migration table isn't present, making it more friendly for projects that aren't using River's internal migration system. PR #465.

⚠️ Version 0.10.0 contains a new database migration, version 5. See documentation on running River migrations. If migrating with the CLI, make sure to update it to its latest version:

go install github.com/riverqueue/river/cmd/river@latest
river migrate-up --database-url "$DATABASE_URL"

The migration includes a new index. Users with a very large job table may want to consider raising the index separately using CONCURRENTLY (which must be run outside of a transaction), then run river migrate-up to finalize the process (it will tolerate an index that already exists):

ALTER TABLE river_job
    ADD COLUMN unique_key bytea;

CREATE UNIQUE INDEX CONCURRENTLY river_job_kind_unique_key_idx ON river_job (kind, unique_key) WHERE unique_key IS NOT NULL;

go install github.com/riverqueue/river/cmd/river@latest
river migrate-up --database-url "$DATABASE_URL"

Added

• Fully functional driver for database/sql for use with packages like Bun and GORM. PR #351.
• Queues can be added after a client is initialized using client.Queues().Add(queueName string, queueConfig QueueConfig). PR #410.
• Migration that adds a line column to the river_migration table so that it can support multiple migration lines. PR #435.
• --line flag added to the River CLI. PR #454.

Changed

• Tags are now limited to 255 characters in length, and should match the regex \A[\w][\w\-]+[\w]\z (importantly, they can't contain commas). PR #351.
• Many info logging statements have been demoted to debug level. PR #452.
• pending is now part of the default set of unique job states. PR #461.

Added

• Config.TestOnly has been added. It disables various features in the River client like staggered maintenance service start that are useful in production, but may be somewhat harmful in tests because they make start/stop slower. PR #414.

Changed

⚠️ Version 0.9.0 has a small breaking change in ErrorHandler. As before, we try never to make breaking changes, but this one was deemed quite important because ErrorHandler was fundamentally lacking important functionality.

• Breaking change: Add stack trace to ErrorHandler.HandlePanicFunc. Fixing code only requires adding a new trace string argument to HandlePanicFunc. PR #423.

  # before
  HandlePanic(ctx context.Context, job *rivertype.JobRow, panicVal any) *ErrorHandlerResult
  
  # after
  HandlePanic(ctx context.Context, job *rivertype.JobRow, panicVal any, trace string) *ErrorHandlerResult
  

Fixed

• Pausing or resuming a queue that was already paused or not paused respectively no longer returns rivertype.ErrNotFound. The same goes for pausing or resuming using the all queues string (*) when no queues are in the database (previously that also returned rivertype.ErrNotFound). PR #408.
• Fix a bug where periodic job constructors were only called once when adding the periodic job rather than being invoked every time the periodic job is scheduled. PR #420.

Added

• Add transaction variants for queue-related client functions: QueueGetTx, QueueListTx, QueuePauseTx, and QueueResumeTx. PR #402.

Fixed

• Fix possible Client shutdown panics if the user-provided context is cancelled while jobs are still running. PR #401.

Added

• The default max attempts of 25 can now be customized on a per-client basis using Config.MaxAttempts. This is in addition to the ability to customize at the job type level with JobArgs, or on a per-job basis using InsertOpts. PR #383.
• Add JobDelete / JobDeleteTx APIs on Client to allow permanently deleting any job that's not currently running. PR #390.

Fixed

• Fix StopAndCancel to not hang if called in parallel to an ongoing Stop call. PR #376.

Fixed

• River now considers per-worker timeout overrides when rescuing jobs so that jobs with a long custom timeout won't be rescued prematurely. PR #350.
• River CLI now exits with status 1 in the case of a problem with commands or flags, like an unknown command or missing required flag. PR #363.
• Fix migration version 4 (from 0.5.0) so that the up migration can be re-run after it was originally rolled back. PR #364.

Added

• RequireNotInserted test helper (in addition to the existing RequireInserted) that verifies that a job with matching conditions was not inserted. PR #237.

Changed

• The periodic job enqueuer now sets scheduled_at of inserted jobs to the more precise time of when they were scheduled to run, as opposed to when they were inserted. PR #341.

Fixed

• Remove use of github.com/lib/pq, making it once again a test-only dependency. PR #337.

⚠️ Version 0.5.0 contains a new database migration, version 4. This migration is backward compatible with any River installation running the v3 migration. Be sure to run the v4 migration prior to deploying the code from this release.

Added

• Add pending job state. This is currently unused, but will be used to build higher level functionality for staging jobs that are not yet ready to run (for some reason other than their scheduled time being in the future). Pending jobs will never be run or deleted and must first be moved to another state by external code. PR #301.

• Queue status tracking, pause and resume. PR #301.

  A useful operational lever is the ability to pause and resume a queue without shutting down clients. In addition to pause/resume being a feature request from #54, as part of the work on River's UI it's been useful to list out the active queues so that they can be displayed and manipulated.

  A new river_queue table is introduced in the v4 migration for this purpose. Upon startup, every producer in each River Client will make an UPSERT query to the database to either register the queue as being active, or if it already exists it will instead bump the timestamp to keep it active. This query will be run periodically in each producer as long as the Client is alive, even if the queue is paused. A separate query will delete/purge any queues which have not been active in awhile (currently fixed to 24 hours).

  QueuePause and QueueResume APIs have been introduced to Client pause and resume a single queue by name, or all queues using the special * value. Each producer will watch for notifications on the relevant LISTEN/NOTIFY topic unless operating in poll-only mode, in which case they will periodically poll for changes to their queue record in the database.

Changed

• Job insert notifications are now handled within application code rather than within the database using triggers. PR #301.

  The initial design for River utilized a trigger on job insert that issued notifications (NOTIFY) so that listening clients could quickly pick up the work if they were idle. While this is good for lowering latency, it does have the side effect of emitting a large amount of notifications any time there are lots of jobs being inserted. This adds overhead, particularly to high-throughput installations.

  To improve this situation and reduce overhead in high-throughput installations, the notifications have been refactored to be emitted at the application level. A client-level debouncer ensures that these notifications are not emitted more often than they could be useful. If a queue is due for an insert notification (on a particular Postgres schema), the notification is piggy-backed onto the insert query within the transaction. While this has the impact of increasing insert latency for a certain percentage of cases, the effect should be small.

  Additionally, initial releases of River did not properly scope notification topics within the global LISTEN/NOTIFY namespace. If two River installations were operating on the same Postgres database but within different schemas (search paths), their notifications would be emitted on a shared topic name. This is no longer the case and all notifications are prefixed with a {schema_name}. string.

• Add NOT NULL constraints to the database for river_job.args and river_job.metadata. Normal code paths should never have allowed for null values any way, but this constraint further strengthens the guarantee. PR #301.

• Stricter constraint on river_job.finalized_at to ensure it is only set when paired with a finalized state (completed, discarded, cancelled). Normal code paths should never have allowed for invalid values any way, but this constraint further strengthens the guarantee. PR #301.

Fixed

• Update job state references in ./cmd/river and some documentation to rivertype. Thanks Danny Hermes (@dhermes)! 🙏🏻 PR #315.

⚠️ Version 0.4.0 has a number of small breaking changes which we've decided to release all as part of a single version. More breaking changes in one release is inconvenient, but we've tried to coordinate them in hopes that any future breaking changes will be non-existent or very rare. All changes will get picked up by the Go compiler, and each one should be quite easy to fix. The changes don't apply to any of the most common core APIs, and likely many projects won't have to change any code.

• Breaking change: There are a number of small breaking changes in the job list API using JobList/JobListTx:
  • Now support querying jobs by a list of Job Kinds and States. Also allows for filtering by specific timestamp values. Thank you Jos Kraaijeveld (@thatjos)! 🙏🏻 PR #236.
  • Job listing now defaults to ordering by job ID (JobListOrderByID) instead of a job timestamp dependent on on requested job state. The previous ordering behavior is still available with NewJobListParams().OrderBy(JobListOrderByTime, SortOrderAsc). PR #307.
  • The function JobListCursorFromJob no longer needs a sort order parameter. Instead, sort order is determined based on the job list parameters that the cursor is subsequently used with. PR #307.
• Breaking change: Client Insert and InsertTx functions now return a JobInsertResult struct instead of a JobRow. This allows the result to include metadata like the new UniqueSkippedAsDuplicate property, so callers can tell whether an inserted job was skipped due to unique constraint. PR #292.
• Breaking change: Client InsertMany and InsertManyTx now return number of jobs inserted as int instead of int64. This change was made to make the type in use a little more idiomatic. PR #293.
• Breaking change: river.JobState* type aliases have been removed. All job state constants should be accessed through rivertype.JobState* instead. PR #300.

See also the 0.4.0 release blog post with code samples and rationale behind various changes.

Added

• The River client now supports "poll only" mode with Config.PollOnly which makes it avoid issuing LISTEN statements to wait for new events like a leadership resignation or new job available. The program instead polls periodically to look for changes. A leader resigning or a new job being available will be noticed less quickly, but PollOnly potentially makes River operable on systems without listen/notify support, like PgBouncer operating in transaction pooling mode. PR #281.
• Added rivertype.JobStates() that returns the full list of possible job states. PR #297.

Added

• New periodic jobs can now be added after a client's already started using Client.PeriodicJobs().Add() and removed with Remove(). PR #288.

Changed

• The level of some of River's common log statements has changed, most often demoting info statements to debug so that info-level logging is overall less verbose. PR #275.

Fixed

• Fixed a bug in the (log-only for now) reindexer service in which it might repeat its work loop multiple times unexpectedly while stopping. PR #280.
• Periodic job enqueuer now bases next run times on each periodic job's last target run time, instead of the time at which the enqueuer is currently running. This is a small difference that will be unnoticeable for most purposes, but makes scheduling of jobs with short cron frequencies a little more accurate. PR #284.
• Fixed a bug in the elector in which it was possible for a resigning, but not completely stopped, elector to reelect despite having just resigned. PR #286.

Although it comes with a number of improvements, there's nothing particularly notable about version 0.1.0. Until now we've only been incrementing the patch version given the project's nascent nature, but from here on we'll try to adhere more closely to semantic versioning, using the patch version for bug fixes, and incrementing the minor version when new functionality is added.

Added

• The River CLI now supports river bench to benchmark River's job throughput against a database. PR #254.
• The River CLI now has a river migrate-get command to dump SQL for River migrations for use in alternative migration frameworks. Use it like river migrate-get --up --version 3 > version3.up.sql. PR #273.
• The River CLI's migrate-down and migrate-up options get two new options for --dry-run and --show-sql. They can be combined to easily run a preflight check on a River upgrade to see which migration commands would be run on a database, but without actually running them. PR #273.
• The River client gets a new Client.SubscribeConfig function that lets a subscriber specify the maximum size of their subscription channel. PR #258.

Changed

• River uses a new job completer that batches up completion work so that large numbers of them can be performed more efficiently. In a purely synthetic (i.e. mostly unrealistic) benchmark, River's job throughput increases ~4.5x. PR #258.
• Changed default client IDs to be a combination of hostname and the time which the client started. This can still be changed by specifying Config.ID. PR #255.
• Notifier refactored for better robustness and testability. PR #253.

Fixed

• Fixed a problem in riverpgxv5's Listener where it wouldn't unset an internal connection if Close returned an error, making the listener not reusable. Thanks @mfrister for pointing this one out! PR #246.