pg_timetable

Breaking Changes

This v4 release is backward-incompatible with previous v3 versions.

• Do not try to run this release against the old-version timetable schema.
• The best way to run it against the clean database or rename schema, e.g.

ALTER SCHEMA timetable RENAME TO timetable_old;

Please, use our new detailed manual to know more about new features and settings.

Changelog 💔🌄🧪

[!] add configuration file support with Viper, closes #177 bebab44
[!] add CopyFromFile built-in task f87d6fc
[!] add Readthedocs documentation (#254) 004b31f
[!] merge timetable.command table with timetable.task, closes #261 8604b18
[!] reimplement logging, closes #158 (#231) 4313948
[!] remove jmoiron/sqlx and DATA-DOG/go-sqlmock dependencies, closes #187 #202 6542b71
[!] remove old migrations and start from scratch, closes #258 020563d
[!] rewrite cron handling from scratch 35a8cc8 fbfbfb2
[!] rewrite pgengine and scheduler without global variables as classes fa37167
[!] use Go 1.16 to build releases 729ef31
[!] use new consistent terminology: command -> task -> chain f59fdda
[+] add $PGTT_CLIENTNAME env var 31faae6
[+] add --cronworkers and --intervalworkers options under "Resource" group 91f5c0e
[+] add .pgpass support, closes #247 d3a317f
[+] add --chain-timeout command-line parameter, closes #270 7f27a50
[+] add --log-database-level command-line parameter, closes #274 338c28c
[+] add --task-timeout command line parameter 80428a7
[+] add all release badge 0b1ae61
[+] add chain timeout, closes #267 05b9736
[+] add config.example.yaml file bebab44
[+] add config_test 5df8386
[+] add database comments for objects f59fdda
[+] add docs badge e325ff5
[+] add high load skip timeout to LogHook 0513ba8
[+] add log hook for PostgreSQL using COPY machinery 93d51cc
[+] add LogHook tests 0513ba8
[+] add output for built-in and SQL tasks to the timetable.execution_log, closes #185 (#224) 681caf3
[+] add pgengine.NewDB function 330cb62
[+] add pgxpoolIface 33fa7a4
[+] add support for logging to file, closes #272 6a73a80
[+] add supported cloud environments to the readme, #256 70c9f49
[+] add supported PostgreSQL versions and operating systems to the readme, closes #256 5695742
[+] add task timeout, closes #271 80428a7
[+] add TASK_STARTED and TASK_DONE statuses, rename STARTED to CHAIN_STARTED 660e32b
[+] add TestMigratorOptions() and increase TestMigrateTxError() coverage 466c909
[+] add TestSchedulerExclusiveLocking() 08e7ff9
[+] add TestSelectChains() 3696f01
[+] add time zone information to the manual 77c0237
[+] add version number to all release files, closes #228 cf72721
[+] allow specify content-type for SendMail built-in task, closes #225 (#226) 100bedd
[+] bump github.com/pashagolub/pgxmock 1.2.0 af98bfd
[+] bump github.com/spf13/viper to 1.8.1 e7b30fd
[+] bump jackc/pgconn version to 1.9.0 7f2d671
[+] bump jackc/pgtype version to 1.8.0 7f2d671
[+] bump jackc/pgx version to 4.12.0 7f2d671
[+] bump jessevdk/go-flags version to 1.5.0 7f2d671
[*] bump georgysavva/scany to 0.2.9 c1f9529
[+] create Dependabot config file 37729d7
[+] delete only succeeded self-destructive chains, closes #265 613a945
[+] increase TestMigrations() coverage 975d68c
[+] Increase v4 tests coverage (#222) 9689e50
[+] insert run status immediately during max_instance check, closes #223 5765662
[+] introduce PgxIface, PgxConnIface, PgxPoolIface b028eaa
[+] move cache settings to LogHook 0513ba8
[+] set client name during LogHook creation 0513ba8
[+] specify password for tests explicitly 524046f
[+] use //go:embed for migration .sql files aaee11d
[+] use //go:embed for pgengine .sql files b453937
[+] use retcode and deferred functions instead of os.Exit() 7a1cdfa
[*] change "--port" command-line option type to integer bebab44
[*] decrease run_status rows usage by using only task related information 660e32b
[*] improve and rename get_running_jobs() to get_chain_running_statuses() 7a1cdfa
[*] improve TestExecuteSQLTask() 396cc88
[*] improve timetable.run_status table 7a1cdfa
[*] make go test fail fast in the build action 35a8cc8
[*] make pgengine.NewDB() and config.NewCmdOptions() use variadic string params 524046f
[*] move health_check() function to job_functions.sql 7a1cdfa
[*] move Logger to appropriate file b5bcece
[*] move PgURL parsing to the pgengine bebab44
[*] move SetupCloseHandler to main.go 7a1cdfa
[*] remove sensitive information from logs, closes #286 aba954d
[*] remove unused chain.excluded_execution_configs column f59fdda
[*] remove unused PgEngine.CanProceedChainExecution() 4cf2323
[*] remove unused timetable.trig_chain_fixer(), closes #255 5b033d7
[*] rename pgengine.UpdateChainRunStatus to AddChainRunStatus 660e32b
[*] rename rus_status.current_execution_element column to command_id 660e32b
[*] replace "--verbose" command-line option with "--loglevel" bebab44
[*] return immediately from pgengine.CanProceedChainExecution if context expired 34946b8
[*] simplify pgengine.CanProceedChainExecution() function 7a1cdfa
[*] simplify readme.md, #256 b7cc5bf
[*] split options into groups: Connection, Logging, Start, etc. bebab44
[*] store remote database connection strings in chain table directly, closes #234 20f28f8
[*] support alpha-beta strings in tag name for Release action e7318a8
[*] switch to ory/mail from abandoned gomail, closes #248 21858fd
[*] update Golang version used in Github Actions 944b903
[*] update latest release badge by including pre-releases 8645ee0
[*] use channel for error instead of variable 0513ba8
[*] use dashes in long command-line parameters names 6a73a80
[*] uses error log level during tests by default 524046f
[-] fix 'date/time field value out of range' error in next_run(), fixes #237 35a8cc8
[-] fix --pgurl ignored during connection, closes #252 5d771df
[-] fix empty long dash separated command-line parameters, fixes #279 4e8016f
[-] fix ErrNoRows check in CanProceedChainExecution() f0701c4
[-] fix SelectChain() 8b802c3
[-] remove database/sql from import eeb3eb4
[-] remove STRICT option from add_job() function, fixes #291 2eff73a
[-] remove unneeded logging CheckNeedMigrateDb() function f59fdda

Changelog 🤜🔗⬆

[+] bump jackc/pgx version to 4.11.0 d0df012
[+] bump jackc/pgconn version to 1.8.1 d0df012
[+] bump jessevdk/go-flags version to 1.5.0 d0df012

Changelog 📪🌍❗

[+] add version number to all release files, closes #228 8912685
[+] allow specify content-type for SendMail built-in task, closes #225 59516ce

Changelog 🛠❗📰

[-] fix deadlock when a simultaneous chains number exceeds DB pool size, closes #218 1af4f34

Changelog 🐳🚢🛠

[+] add certificates to the Docker image 2c0d1cc
[*] add workflow_dispatch trigger to "Update Docker" action 67fdce7
[-] fix "x509: certificate signed by unknown authority" in SendMail task from Docker, fixes #216 2c0d1cc

Changelog 🚢🚀🔨

[!] Rewrite Dockerfile to support go modules, resolves #204 b6babaf
[+] Bump jmoiron/sqlx from 1.2.0 to 1.3.1 1642906
[+] Build & Test action: add fkirc/skip-duplicate-actions step e936561
[*] Build & Test action: remove push trigger, enable pull_request and workflow_dispatch only 6f97d21 44de57a
[-] Allow client name with dashes, fixes #214 20363f9
[-] Fix 'Scan error on column index 1, name "chain_id"...', fixed #207 6d22626

Changelog ⌚⚛🔗

[!] add notify_chain_start() and notify_chain_stop() functions, closes #195 62fbfb1
[!] add PGTT_CLIENTNAME env variable, closes #192 4d94de6
[+] add ChainSignal structure to handle START and STOP async commands 64a8f78
[+] add check to prevent session locking on a replica e7cd5f4
[+] add Dependabot Status badge f878ea5
[+] add on "pull_request" trigger to "Go Build & Test" action bac82ba
[+] add pgengine.TestSelectChain() test case af9d6c4
[+] add TestLogError test case, fix some regular expressions 2fdf7f5 4327436
[+] adds activeChains map so we can abort chain by request 76a11b3
[+] enhance "Build & Test" GitHub Action triggers 849535a
[+] enhance "Update Docker" GitHub Action triggers 9387b63
[+] push docker tags using the semver syntax by their git name 615ebd5
[*] allow SelectChain() return not only live chains for debug purposes 5dff98d
[*] bump github.com/jackc/pgconn from 1.6.4 to 1.8.0 11f671c
[*] bump github.com/jackc/pgx/v4 from 4.8.1 to 4.10.1 11f671c
[*] bump github.com/ory/dockertest/v3 from 3.6.0 to 3.6.3 5c5c3a5
[*] bump github.com/stretchr/testify from 1.6.1 to 1.7.0 884a842
[*] change error message in TryLockClientName to neutral form and make it log message dd5dd02
[*] clean cached NOTIFY messages after main loop period, closes #191 6c64aa6
[*] elevate adding async chain message from DEBUG to LOG 0091118
[*] improve "Release with goreleaser" GitHub Action triggers 01980e4
[*] obtaining chains logic moved to pgengine module from scheduler 65f4cd5
[*] remove duplicate GutHub Actions run for push and pull-request e514848
[*] remove duplicated PID from the timetable.active_sessions, closes #190 84c2a5d
[*] remove exact names for .mod and .sum files from actions triggers 570ec86
[*] remove ignored tasks for "Build & Test" action 0518b02
[*] remove obsolete comment 2636dba
[*] remove unused pgengine.GetLogPrefixLn() dd572a4
[*] rewrite notification testing with external notifier e95e556
[*] update golangci-lint-action to v2 d63404c
[*] usage examples – make them work immediately e7a4716
[*] use Log function instead of fmt.Printf 026d371
[-] add mutex to operations on activeChains map, fix race conditions 3ff246b
[-] change LogToDB to Log to avoid deadlock aa6d520
[-] do not pass chain configs less or equal 0 for async processing d8526e6
[-] fix "Cannot log to the database: context canceled" error f33ccdf
[-] fix "Cannot log to the database: context deadline exceeded" error for NOTICE log entries, closes #153 6b899c6
[-] fix assignment to entry in nil map b81663a
[-] fix chain status table if chain aborted b230491
[-] fix cleaning period for NOTIFY messages fe9c9e2
[-] fix deprecated set-env in GitHub Actions 615ebd5
[-] fix handling 2+ NOTIFY messages with the same payload, fixes #189 64a8f78
[-] fix TestHandleNotifications() 68a9f88

Changelog 🎄🎅❄

[+] add Dependabot support 1ff9fdf
[+] add information about docker image to README 06ddecd
[+] enhance "Build & Test" GitHub Action triggers 3dcb212
[+] enhance "Update Docker" GitHub Action triggers 026dfc6
[+] push docker tags using the semver syntax by their git name 72fc3d9
[*] bump github.com/jackc/pgconn from 1.6.4 to 1.8.0 28d3719
[*] bump github.com/jackc/pgx/v4 from 4.8.1 to 4.10.1 28d3719
[*] bump github.com/ory/dockertest/v3 from 3.6.0 to 3.6.3 4e9452c
[*] improve "Release with goreleaser" GitHub Action triggers b10b0b3
[*] update golangci-lint-action to v2 3b29778
[-] fix deprecated set-env in GitHub Actions 72fc3d9
[-] fix incorrect double quote escaping in task names, fixes #167 f2adea1

Changelog

[!] implement asynchronous chain execution, closes #71 a387c0f
[!] implement debug mode, closes #63 a387c0f
[!] implement exclusive execution mode for chains, resolves #50 c7a538d
[!] reimplement session locking, resolves #149 ec1f368
[!] use jackc/pgx PostgreSQL library instead of lib/pq, closes #112 f4c945c
[+] add Chain.Lock() and Chain.Unlock() methods 233d004
[+] add getPgxConnString() function 73021c4
[+] add LISTEN\NOTIFY support b8fd7c4
[+] add Log() function which performs logging to standard output 2b7d5a4
[+] add mutex to the notificationHandler df062a9
[+] add notification_test da11522
[+] add setupTestCaseEx() function df062a9
[+] add tests for IntervalChain class 80ffce9
[+] add tests for pgengine.ExecuteCustomScripts() function a82b995
[+] add tests for pgengine.ExecuteSchemaScripts() function bcfbada
[+] add tests for pgengine.FinalizeConfigDBConnection() function 6eeae0a
[+] add tests for pgengine.ReconnectDbAndFixLeftovers() function 01bebe6
[+] add tests for pgx logger 749f999
[+] add tests for retrieveAsyncChainsAndRun() function 82434d9
[+] add tests for scheduler.chainWorker() function 167664e
[+] add timetable.next_run(timetable.cron) function #105 91473f6
[+] check if the schema is available already first in TryLockClientName() 73021c4
[+] create SECURITY.md 0e779de
[+] explain the ignore_error flag on the scheduler working process. 4c0ca4a
[+] make chain channels buffered 82434d9
[+] make intervalChainsChan channel buffered 80ffce9
[+] make LOG lines green and DEBUG blue b8fd7c4
[+] prettify pgx logged object output b8fd7c4
[+] run chains retrieval in parallel b8fd7c4
[+] use built-in pgx logging for verbose mode 236bbce
[+] use golangci-lint-action instead of binary installation 85e5c74
[+] use pre-installed PostgreSQL in Github Actions (#163) 12c5ade
[*] add *.sql files as targets for github action 8a2a36e
[*] add PostgreSQL v13 to the manual 23d0acb
[*] awesome-go badge added 96f5958
[*] change SHELL task kind to PROGRAM, closes #155 ea06e3d
[*] move chain implementation into separate file ef729d4
[*] remove unused const 41dac5c
[*] update pgx and go-sqlmock dependencies 3d69873
[*] update schema 46f95a8
[-] "Cron syntax "/5 * * * *" don't execute job at 00 minutes" bug fixed, closes #156 15a178f
[-] call DB.Close() in tear down test function 90ac0c8
[-] fix --name option in --clientname. c4c9246
[-] fix engine tests 7ef5402
[-] fix golint warnings b468219
[-] fix migrator tests 5bac7af
[-] fix README default values docs, spelling c6f2369
[-] fix remote execution, fixes #164 a61ee58
[-] fix reschedule() function 38b77ba
[-] fix tests 36ecf8f
[-] fix tests for pgx library #112 d52ccb8
[-] fix TestSamplesScripts() f76bb74
[-] fix TestShellCommand ee4dbb2 a5ac83b
[-] fixes "Interval field value out of range", fixes #148 75bf0d6
[-] use client_name in job_add function, fixes #150 06a294b

Changelog

[!] update Go version to 1.14 b8d7d30
[!] add contexts to all low level engine functions, resolves #142 2c458a3
[!] add continuous code coverage #140 70f3e21
[!] add timetable.next_run(timetable.cron) 03cfbef
[+] add a separate tag to Docker Hub for every release 4495b73
[+] add bootstrap tests 5db1aef
[+] add ClientName argument to cmdparser.NewCmdOptions 16b3383
[+] add contexts to task and logging functions 56f00f0
[+] add continuous code coverage enhancement. Closes #140 d426785
[+] add Coveralls action for push and pull request 128d727
[+] add log tests 5429ed8
[+] add self destruct chain to scheduler test 088372d
[+] add tests for pgengine migrations b618547
[+] add tests for scheduler
[+] exit from workers immediately if context done 925fe37
[+] improve pgengine test coverage 4869958
[+] make access.go 100% covered by tests bea9739
[+] make chains channel buffered b491a35
[+] make cmdparser 100% covered by tests 292c03b
[+] make interval chain reschedule compatible with context 0389e5b
[+] make intervalChainsChan channel buffered
[+] make migrator 100% covered by tests 63a1ae8
[*] close database connection at the end of each test e481d9b
[*] log message prettify 6a61b03
[*] prettify code 4fd917c
[*] remove init() function from shell.go 8ec2e7e
[*] rewrite pgengine.TryLockClientName 06d9a6c
[-] disallow parallel execution of tests 325c0a8
[-] fix Docker Hub name d9a7633
[-] fix double sleep time for interval chains bb2b452
[-] fix engine tests 16ef999
[-] fix possible blocking in chain workers b491a35
[-] fix SelfDestruct.sql sample 5ef5586
[-] fix TryLockClientName to return True immediately without waiting 46c01b7
[-] remove Golang setup step. Latest is available automatically 61c207b
[-] remove unused variable 9626cd0

Changelog

[+] create workflow to push docker image bea8f55
[+] log output for sql and built-in tasks in case of error 086b261
[-] fix "sql and built-in tasks have returncode 0 even in case of error" 086b261
[-] if task output is empty, log error as output effdfea

Changelog

[!] add --init command line argument fixes #128, closes #129 f441bfc
[!] add --no-help command line argument to suppress help output 18768b0
[!] add Grafana dashboard to monitor chain execution stats dd8608c
[!] implement custom startup script support with --file, closes #131 e66ece0
[!] implement autonomous tasks
[+] add context argument to InitAndTestConfigDBConnection() 4acfaf5
[+] add context arguments to migrations routines 4acfaf5
[+] add context arguments to transactions, access and shell routines #118 9b599c2 4acfaf5
[+] add context to pgengine.ReconnectDbAndFixLeftovers() and use channels
[+] add context to scheduler.Run()
[+] add pgengine.IsAlive() method
[+] add reconnection routine to the main()
[+] add sample for autonomous task
[+] make cmdparser 100% covered with tests, resolves #126 8e787f8
[+] make built-in 100% tasks 5778939
[+] make download task 100% covered with tests 1f193e1
[+] make mail task 100% covered with tests febeb1f
[+] pass derived contexts to chain workers goroutines
[+] return ConnectionDropped after failed pgengine.IsAlive()
[+] rewrite reconnecting routing with channels and context #118 787628a c2cc25d
[+] set timeout for initial connection to 90 sec #118 19dbca7 9d4c5c4
[*] improve log messages for reconnection tries #118 4ee0cc3
[*] make InitAndTestConfigDBConnection() return boolean 4e1e2c8
[*] make MigrateDb() and CheckNeedMigrateDb() return boolean #118 4e1e2c8 828f8f2
[*] make pgengine.WaitTime public const 032ab3e a1c63ce
[*] pass context argument to InitAndTestConfigDBConnection() in tests #118 223f1d9 693e817
[*] prettify log strings 65afcb1
[*] refactor command line handling b87995f
[*] remove reconnection routine from pgengine.LogToDB()
[*] set cyclomatic complexity to 16 for linter; resolves #122 73d29fa
[-] fix "if block ends with a return statement, so drop this else and outdent its block" 16fa2ee 3f8c518
[-] fix "Cannot log to the database: pq: relation "timetable.log" does not exist" #118 25df3de 74ee9e6
[-] fix "exported func Run returns unexported type scheduler.runStatus, which can be annoying to use" 310fd5d 3363398
[-] fix "Host in --pgurl parameter ignored if port is not specified" f441bfc
[-] fix "interval chains are not rescheduled if a database error occurs" f6dcb97
[-] fix "return" to "continue" in scheduler.intervalChainWorker() 8226433
[-] fix "S1039: unnecessary use of fmt.Sprintf" 8226433
[-] fix tests connection string 3106fa4 e32cb76
[-] fix verbose level output
[-] make scheduler.Run() return runStatus type #118 20c08e0 d07a486
[-] reschedule interval chains in a separate goroutines acf02ec

Changelog

[+] add "output" column to the timetable.execution_log table; fixes #86 067e893
[+] add .yml files as action triggers b258934
[+] log client name and PID of worker at the start of a session cae0572
[+] make REPAIR pseudo log level red a3fccd6
[+] PostgreSQL license used 23f27ef
[+] resolves #108: add client_name to run_status and execution_log tables cae0572
[+] resolves #115: Implement color output cc1fb64
[+] test only if GoLang files were affected e843dbc
[+] unshallow step is required for the changelog to work correctly b40beea
[+] update golangci-lint version to 1.24.0 7fd8db4
[*] add link to PR's fdab874
[*] new pgengine/log.go file added cc1fb64
[*] remove setup-go action since Go is preinstalled 7fd8db4
[*] templated license used 4faf8ff
[*] update golangci-lint settings 7fd8db4
[-] do not produce db objects, use db.Ping() instead of sqlx.Connect() 8662f94
[-] fix "Cannot stop execution if connection lost" #119 #120 2eb6193
[-] fix "Cannot stop execution inside RECONNECTING cycle" 2eb6193
[-] fix "S1039: unnecessary use of fmt.Sprintf" 1bb1080
[-] fix pgengine test 581ad8a
[-] fix workflow with including paths e4b1d8c
[-] fixes #113: "STARTED deleted from run_status during crash" cae0572
[-] merge pull request #121 from cybertec-postgresql/120_connection_lost_terminate 19b4a93

Changelog

47faf73 - fix error date in the log #111

Changelog

fc150de Merge pull request #107 from cybertec-postgresql/100_fix_crash

The new major v2 release introduces:

• brand new cron-style schedules;
• @reboot, @every and @after jobs;
• disallow executing the shell tasks using --no-shell-tasks command line parameter;
• continuos testing implemented with GitHub Actions;
• release packages for different platforms available (.rpm, .deb, .tar.gz, .zip; Linux, macOS, Windows)
• no external .sql files for bootstrapping, only single binary shipped;
• upgrade machinery with database migrations implemented, see --upgrade option;
• notice processing added, e.g., RAISE NOTICE ... from tasks collected into the log

Changelog

6b99ed9 - fix insert statements in samples #70

Embedded SQL and migrations