Queueing jobs in Postgres from Node.js like a boss
MIT License
Bot releases are visible (Hide)
Published by timgit over 3 years ago
onComplete
to opt out of creating a completion job in the queue once a job is completed. This defaults to true for backwards compatibility.Published by timgit over 3 years ago
archived
, deleted
, and expired
.Published by timgit about 4 years ago
Published by timgit about 4 years ago
Published by timgit about 4 years ago
Published by timgit about 4 years ago
Published by timgit about 4 years ago
Reintroduced archive delay for completed jobs to restore reliable throttling and debouncing.
publish()
will now throw if you use a throttling or debouncing interval set higher than the archive delay.archiveCompletedAfterSeconds
constructor option to allow overriding the default of 12 hours. If you set this lower than 60s, a warning will be emitted and cron processing will be disabled, as this feature relies on debouncing once a minute to operate properly.Published by timgit over 4 years ago
Published by timgit over 4 years ago
The pg-boss team hired a timekeeper and now has distributed cron-based scheduling! This works across all instances based on the database server's time as a central clock.
New functions:
schedule(name, cron, data, options)
unschedule(name)
getSchedules()
New constructor configuration properties:
clockMonitorIntervalSeconds
clockMonitorIntervalMinutes
noScheduling
connect()
and disconnect()
to simplify usage since these functions became obsolete in v4. If you had relied on secondary instances running with connect()
, you should switch to start()
. Since start()
is multi-master, it's safe to let it monitor and submit maintenance work, but if you need to opt out of this for whatever reason on a particular instance, set the noSupervisor
and noScheduling
constructor options to true
.poolSize
in constructor database config to standardize on max
property used in the pg package.Removed properties related to moving completed jobs to the archive table. Completed jobs will be moved to the archive table based on the maintenance interval going forward.
Old | New |
---|---|
archiveIntervalSeconds |
** REMOVED ** |
archiveIntervalMinutes |
** REMOVED ** |
archiveIntervalHours |
** REMOVED ** |
archiveIntervalDays |
** REMOVED ** |
Renamed properties for controlling when to delete jobs from the archive table
Old | New |
---|---|
deleteIntervalSeconds |
deleteAfterSeconds |
deleteIntervalMinutes |
deleteAfterMinutes |
deleteIntervalHours |
deleteAfterHours |
deleteIntervalDays |
deleteAfterDays |
Published by timgit over 7 years ago
While job expiration has always been a thing, reacting to them was not as easy. boss.onExpire(jobName, callback)
was added to this release to add a listener explicitly for jobs that have expired.
failed
was added as a job status.done()
in suscribe()
callbacks now support passing an error (the popular node convention) to automatically mark the job as failed as well as emitting failed. For exampe, if you are processing a job and you want to explicitly mark it as failed, you can just call done(error)
at any time.boss.fail(jobId)
was also added for external failure reporting for the fetch()
and complete()
crew.Thanks to @jordansexton for the suggestion on this one. The world is now a better place. 👍
boss.unsubscribe(jobName)
was added to allow unsubscribing a previous subscribe()
. I don't expect this to be very popular, but it does make the API a bit more flexible and aligns subscriptions with the pattern already in place with other methods. For example, stop()
reverses start()
and disconnect()
reverses connect()
.
Published by timgit over 7 years ago
pg-boss turns 1 this week, and after using it in production for the last year it felt fitting to use the latest round of enhancements, fixes and optimizations as an opportunity to bump the version up to 1.0.0 and finally embrace true semver.
A year ago node 0.10 and 0.12 were still very well supported, but i decided to go ahead and drop support for them with 1.0.0 as they are no longer supported node releases. I know with 100% confidence that I broke 0.10 support with this release because of the way 0.10 handled event emitters. While pg-boss should technically still work on 0.12, I no longer run any automated tests on it.
Even though this is major version change, it's not something that will make your current 0.x code break as only additions were made to the API. However, given there were actually database schema changes in 1.0, a migration script needs to be run against any 0.x instances. If you are one of these awesome early pioneers, don't worry too much about the migration because this will be automatic after you upgrade and call start()
.
As always, if you don't want this to occur automatically, or if your user account doesn't have the required permissions, run
getMigrationsPlans()
and it will produce the SQL scripts required for migration based on your current version. I added a scripts folder to the repo as an example of how one might go about exporting said SQL if needed.
On 0.x I used minor releases for semver, but now that I'm on 1.x, I hope this removes all doubt regarding versioning in the future as I fully plan on using semver.
Singleon jobs - "I only want 1 job of this type to be running at a time"
A new publish option called singletonKey
was added in order to make sure only 1 job of a certain type is active, queued or in a retry state. Thanks to @jede for the suggestion and feedback on this.
Throttling "next slot" - "I like throttling, but if my publish request is throttled, please at least queue it up for the next throttling interval"
A new publish option called singletonNextSlot
was added in order to make sure a job is processed eventually, even if it was throttled down (not accepted).
This is actually something I had way back in 0.3.x, but removed it because it wasn't explicit in the API and made things a little unpredictable.
Per-subscription new job check interval - "I like the default polling of every second, but only check for this job every 10 seconds".
The same newJobCheckInterval
and newJobCheckIntervalSeconds
are now available on subscribe()
. They behave the same as the global default used in the constructor and control how often each job type is fetched on a subscription by subscription basis if needed.
This may not seem all too exciting, but if you use it in combination with
teamSize
, you can start to realize how this could be helpful in regards to fine tuning your job processing.
Connection pooling - "I need to ensure that pg-boss will not use more than this number of connections against my database."
A new option poolSize
has been added to the constructor that enforces the maximum number of connections that can be used against the specified database. Thanks to @phips28 for pointing this out.
Technically connection pooling was configured behind the scenes in 0.x, but since a pool wasn't explicitly set up during pg-boss initialization, it wasn't a guarantee on a maximum number of connections that could be opened against the database.
0.x had a data management bug which caused expired jobs to not be archived and remain in the job table. After this was fixed I also added a fix to the migration script so if you had any old expired jobs they should be automatically archived.
Error handling in subscriber functions! Previously I've encouraged folks to handle their own errors with try catch and be as defensive as possible in callback functions passed to subscribe()
. However, it was too easy to miss that at times and if an error occurred that wasn't caught, it had the pretty lousy side effect of halting all job processing. 1.0.0 now wraps all subscriber functions in try catch blocks and emits the 'error' event if one is encountered.
Also, please remember that pg-boss is an instance of node's EventEmitter. If an EventEmitter emits 'error' and you haven't written a handler for it, it's kind of a big deal. I updated the primary readme to add this handler because it really should just always be there.
Published by timgit over 7 years ago
fetch()
and complete()
were added in 0.5.0 in order to provide a way to request and process jobs externally in contrast to the typical managed polling style in subscribe()
. These methods were originally internally facing, but having them publically facing would allow you to build other entry points into the job system such as a web API.
Published by timgit over 8 years ago
Removed ready
event during start()
and cancel()
in favor of a promise API. The readme was updated with the latest API.