Release Highlights

This release mitigates a problem where invalid UTF-8 data could be supplied to the history service, causing a denial of service

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use the tag 1.21.6)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Full Changelog: https://github.com/temporalio/temporal/compare/v1.21.5...v1.21.6

Release Highlights

This release mitigates a problem where invalid UTF-8 data could be supplied to the history service, causing a denial of service

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use the tag 1.20.5)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Full Changelog: https://github.com/temporalio/temporal/compare/v1.20.4...v1.20.5

All Changes

2023-11-13 - db23e8671 - Add action metrics for schedule and updates (#5101)
2023-11-03 - 37da275af - Allow list of keywords in dual visibility setting (#5065)
2023-11-01 - 26114d774 - Hopefully avoid panic in DescribeWorkflowExecution (#5057)
2023-10-31 - ddf979916 - Bump server version to 1.22.2 (#5056)
2023-10-31 - ef3b92b86 - Fix schedule workflow unit test (#5052)
2023-10-30 - 2320fc172 - Add force-continue-as-new signal to schedule workflow (#5034)
2023-10-30 - b8163ceb6 - Use ContinueAsNewSuggested in scheduler workflow (#4990)
2023-10-30 - 88f8f11da - Update Go SDK to v1.25.1 (#4955)
2023-10-30 - cba2ce7ec - Improve scheduler workflow unit test (#4962)
2023-10-30 - 53a4979f4 - Expose buffer size and # dropped actions in ScheduleInfo (#4839)
2023-10-30 - 0fc50f827 - Adjust action metrics (#5043)
2023-10-30 - 91425ba5d - Emit action metrics for batch of markers (#4905)
2023-10-30 - f442eb703 - Validate search attributes values in queries to ES (#5036)
2023-10-30 - 19ba0cec9 - Disable eager activities for incompatible versioned activities (#5030)
2023-10-30 - cd14e159f - Add refresh task for close workflow (#4999)
2023-10-30 - e022877a8 - Refresh upsert visibility task only for running workflows (#4909)
2023-10-30 - f99bd882d - Lock get current execution for API requests (#4970)
2023-10-30 - 8dc833c0a - Fix GenerateMigrationTasks behavior (#4987)
2023-10-30 - 1a3083326 - Return InvalidArgument error when we get HTTP 400 from elasticsearch (#4900)
2023-10-30 - b6c9868a6 - Reuse cached RemoteAdminClient in VerifyReplicationTasks (#4997) (#5050)

Important

There was a security issue in the Temporal Docker image that could expose your DB password in the shell logs. The issue is fixed in https://github.com/temporalio/docker-builds/pull/147, and new Docker image builds packing this version or later will contain the fix. Note that this issue is not related in any way to the Temporal Server itself. It only affects a script in the Docker image to launch your Temporal Server.

All Changes

2023-10-13 - 717352261 - Ensure latest Go version in GH workflows (#4974)
2023-10-12 - b154bcda8 - Bump base-ci-builder image (#4969)
2023-10-06 - 759b60896 - Refactor TaskPoller.PollAndProcessWorkflowTask (#4903)
2023-10-06 - 4a70d6293 - Don't track overlapping scheduled workflows (#4911)
2023-10-06 - 1f01c72a3 - Use RateBurst in RateLimitInterceptorProvider (#4933)
2023-10-06 - 9abaafd16 - Take history length into account when rate limiting generate replication task (#4929)
2023-10-06 - ba6391831 - Improve XDC logging (#4932)
2023-10-06 - 8cadb9400 - Handle XDC replication task retry error / not found error correctly (#4924)
2023-10-06 - 8d1e43aa9 - Make rate multi min and max dynamic (#4920)
2023-10-06 - b4df68298 - Add metric for dynamic rate limit multiplier (#4921)
2023-10-06 - 68604955a - Improve XDC Logging (#4917)
2023-10-06 - b5f73191d - Simplify DescribeSchedule (#4810)
2023-10-06 - 1af21c404 - Build id scavenger shouldn't look at recent build ids (#4846)
2023-10-06 - 9547e8249 - Exit build id scavenger on rate limiter error (#4841)
2023-10-06 - e9985714d - Fix QueryWorkflow forwarding on versioned queue (#4847)
2023-10-06 - 277e97d49 - Make versioning data limits configurable by namespace (#4843)

Release Highlights

Streaming for replication (WIP)

This version of server added gRPC streaming API for replications between clusters. This feature is still working in progress, is currently disabled by default. It will be enabled in future release.

Count workflow by ExecutionStatus

The Count API feature allows the user to count the number of workflow matching the given query. For example, the command temporal workflow count -q "WorkflowType='foo'" returns the number of workflows that matches WorkflowType='foo'.

Now, you can send queries to the Count API to group by a given search attribute. Eg: -q "WorkflowType='foo' GROUP BY ExecutionStatus returns the number of workflows that matches WorkflowType='foo' grouped by ExecutionStatus.

The GROUP BY clause is only supported in the Count API, and, at this moment, we only support grouping by ExecutionStatus. We plan to support group by any search attribute in a future release.

Default authorizer changes

The default authorizer now allows OperatorService.ListSearchAttributes as a namespace-level read-only API. Additionally, the implementation was changed: if you have custom authorizer code based on the default authorizer, you may want to examine the new implementation.

Schedules improvements

Schedules that are “idle”, meaning they are only configured to run for a fixed number of times (either with “limited actions” or with a spec that doesn’t match any times in the future), will now automatically be deleted after one week since their last action. This applies to existing schedules, however existing schedules that are already idle won’t be deleted unless they’re woken up by a signal.

Jitter is now calculated correctly so that jitter across separate schedules is not correlated. This automatically applies to existing schedules.

HTTP API

The frontend workflow service API can now exposed via HTTP in addition to the existing gRPC exposed form. The port is set on services.frontend.rpc.httpPort and is disabled if unset or 0. The default Docker template enables the HTTP API on port 7243. See the HTTP annotations on the workflow service API proto file for more information about which operations are available at which URL paths.

All changes

2023-09-05 - 9b877fae2 - increase max allowed shard linger limit (#4790)
2023-09-05 - c73430b8a - Mark QueryWorkflow as read-only in api metadata (#4834)
2023-09-05 - 86d883913 - Add method metadata and use for default authorizer (#4794)
2023-08-28 - bba21caee - Always persist guessed set id on unknown build id (#4800)
2023-08-28 - c13afdcef - Fix mutable state change not persisted bug (#4813)
2023-08-21 - df2167ef0 - Update API to v1.24.0 (#4785)
2023-08-21 - f6755a53c - Return NonRetryableApplicationError if Namespace is not found on remote cluster during verify (#4786)
2023-08-21 - f42d19eba - Split sync workflow state logic to dedicated file (#4784)
2023-08-18 - c7847968b - HTTP API support (#4543)
2023-08-17 - 8309152e1 - Update some task refresh logic (#4779)
2023-08-17 - 0080bd209 - Simplify XDC logic (#4546)
2023-08-14 - 84b1e8b5a - Generate txn id for each event (#4774)
2023-08-14 - aa5c3db6f - Add api to Eager retrieve the unknown namespace from remote cluster (#4769)
2023-08-14 - 9f9db3f06 - Increaes RecordHeartbeat frequency in VerifyReplicationTask (#4771)
2023-08-11 - 815c8a359 - Skip verifying workflow which has already passed retention time (#4770)
2023-08-11 - 0ce1234ee - Rename leftover master to main (#4768)
2023-08-11 - 7982f81e4 - Fix delete search attributes in mutable state (#4767)
2023-08-11 - 545c96378 - Count group by status capability (#4750)
2023-08-11 - 88f8c1c66 - Load task queue managers for demoted version set ids (#4762)
2023-08-11 - 2b3d52e15 - Improve otel gauge memory usage (#4722)
2023-08-10 - 673d5636b - Add corruption protection to ms (#4564)
2023-08-11 - e4ce970e2 - Turn ErrWorkflowClosing into a retryable error (#4765)
2023-08-10 - 3a9e34477 - Use raw history/matching client in task executor (#4710)
2023-08-10 - 97f15fcce - Fix SQLite FTS query expression with dashes (#4755)
2023-08-10 - 2da474317 - Remove debug logger from visibility manager (#4751)
2023-08-10 - 2eb96096d - Make client code generation safer (#4747)
2023-08-10 - 274de9870 - Make cassandra schema update idempotent (#4761)
2023-08-10 - 5ced7bf63 - Improve task range completion prerequisite check (#4754)
2023-08-10 - 7a11f91cf - Move getTaskQueueManager call into redirect methods (#4759)
2023-08-10 - 97dbe09b4 - Fix retry policy max attempts (#4752)
2023-08-10 - 31eada0cd - Reduce frontend handler retry max attempts (#4753)
2023-08-09 - 6d4a76b41 - Adjust API priorities (#4756)
2023-08-09 - cfad693d6 - Clean up some matching errors (#4758)
2023-08-09 - 0737a8be1 - Avoid potential double close on userDataChanged (#4757)
2023-08-09 - 63cd05122 - Do not load sticky queue if not loaded (#4694)
2023-08-09 - d87acbd41 - Add frontend.globalNamespaceCount (#4743)
2023-08-09 - 039de3d8d - Add descriptions for task processing metrics (#4746)
2023-08-08 - e7dd2ff5f - Clean task queue name from matching before writing to WorkflowTaskScheduledEvent (#4565)
2023-08-08 - b1804c219 - eager namespace refresher POC (#4725)
2023-08-08 - 881b92310 - APS limit error handling (#4739)
2023-08-08 - a60872188 - remove the event cache initial size and refactor putInternal (#4698)
2023-08-07 - a3a4ac541 - Add descriptions for schedule metrics (#4728)
2023-08-07 - 58e731a31 - Enable new schedule workflow logic (#4741)
2023-08-07 - 60f733a80 - Fix bug causing duplicates when listing s3-archived workflows (#4712)
2023-08-07 - 5889f1ccd - Continue replication verification by skipping workflow which should be or soon to be deleted (#4734)
2023-08-07 - c861c7ebe - Update main branch in Makefile (#4740)
2023-08-07 - c49bdc6b7 - Change schedule jitter to depend on schedule + namespace ids (#4685)
2023-08-07 - 3bece5f82 - Collapse duplicate visibility tasks (#4727)
2023-08-07 - 0541c8168 - Send versioned tasks to dlq when user data disabled (#4668)
2023-08-07 - bd909c967 - Return errors from task executors when processing actions (#4719)
2023-08-07 - df262d623 - Remove debug logging from Cassandra schema update tool (#4738)
2023-08-04 - 23aac05bf - Speed up passive task validation (#4700)
2023-08-04 - 10a6f6e74 - Return ResourceExhaustedError for consistent query buffer full (#4731)
2023-08-04 - ecb625a1e - Update timer task error handling (#4718)
2023-08-03 - dcccbf010 - Log actual error for get history data loss events (#4726)
2023-08-03 - 5285b80b4 - Update running server locally README (#4723)
2023-08-03 - ccd23cb6b - Bump UI version to 2.17.1 (#4720)
2023-08-03 - db491171e - Add worker versioning calls to read-only namespace APIs for authorizer (#4721)
2023-08-03 - 9de9ccf5c - Add ActivityType to ActivityInfo for mutable state (#4717)
2023-08-02 - 4928b73c4 - Reduce the heartbeat size of VerifyReplicationTasks activity (#4705)
2023-08-02 - 1eb3217a5 - Add replay tests for scheduler workflow (#4701)
2023-08-02 - 4efda3c9c - Add more errors to targeted fault injector (#4711)
2023-08-01 - f4ca57f0c - Special some handle history client generation (#4708)
2023-08-01 - fb1d507e7 - Provide a shard ownership-based rate limit quota scaler from the history/shard package (#4704)
2023-08-01 - ac1d1742f - Simplify newTaskQueueManager arguments (#4691)
2023-07-31 - 283291855 - Support injecting additional interceptors in history and matching (#4680)
2023-07-31 - 82fec8462 - Inject custom task executor wrapper (#4697)
2023-07-28 - c595eb89e - Finish idle schedule when a timer fires (#4602)
2023-07-28 - 4fb4dd9f8 - move IsValid to ControllableContext (#4699)
2023-07-27 - 926950793 - Add versioned tag to loaded task queue gauge (#4672)
2023-07-27 - 83a3a45a8 - Persist unknown build ids (#4667)
2023-07-27 - 0281243bb - enable shard linger with time limit config (#4696)
2023-07-28 - a13efeeb4 - Add quotas.DelayedRateLimiter (#4688)
2023-07-27 - 48b80c0b0 - Add shard.OwnershipScaledRateBurst (#4687)
2023-07-27 - 851e26b11 - Support count group by in visibility with SQL (#4666)
2023-07-27 - f9b2db525 - Support count group by in visibility with Elasticsearch (#4619)
2023-07-27 - a3b591147 - Matching service task GC (#4418)
2023-07-27 - 569010780 - Put user data replication tasks in DLQ (#4609)
2023-07-27 - 922cc2778 - Convert version history bad request to internal error (#4690)
2023-07-27 - fbd0de731 - Fix Go version string goreleaser.yml (#4689)
2023-07-27 - 541dee303 - Add back generated mocks (#4671)
2023-07-26 - 4cd4f7e31 - Refactor CalculateEffectiveResourceLimit and NamespaceCountLimitInterceptor (#4665)
2023-07-26 - 3be986cdb - support delaying shard close for membership change (#4659)
2023-07-25 - 8753b8eae - Adjust CLI page size (#4682)
2023-07-25 - 157bf673c - Fix tdbg unable to dump all workflow raw history issue (#4681)
2023-07-25 - c3fa08891 - Return nil instead of task discarded error if namespace is invalid locally (#4678)
2023-07-25 - e7bd02808 - add configurable ownership caching to history client (#4652)
2023-07-24 - fa1597b8e - Assign fx pass in raw task converter (#4679)
2023-07-24 - 5008255c1 - Add more replication error logs (#4677)
2023-07-24 - 386a651fe - Minor fixes in force replication verification (#4675)
2023-07-24 - 48d3107a3 - Restore defaultListWorkflowsPageSize back to 1000 (from 100) (#4674)
2023-07-22 - ce3f42e70 - Add clock.TimeSource.AfterFunc (#4611)
2023-07-21 - c57f58779 - Add XDC replication cache (#4379)
2023-07-21 - e10f21389 - Fix flaky TestForceReplicationWorkflow_VerifyReplicationTaskNonRetryableError (#4669)
2023-07-21 - f708c81fe - Move redirect logic into taskQueueManager (#4657)
2023-07-21 - baa964eb7 - Fix flaky versioning integration test (#4651)
2023-07-21 - bd18f4940 - defer shard.Unlock() (#4664)
2023-07-20 - 00587f605 - Execute VerifyReplicationTasks as an individual activity (#4656)
2023-07-21 - a834ea49e - Validate workflow task start time when complete (#4663)
2023-07-20 - a4a2ec027 - Fix future TestSetReadyGet_Parallel (#4661)
2023-07-20 - fc414a85d - Custom search attributes validation per store (#4655)
2023-07-20 - 61ff86b64 - update linter version in update-linters (#4653)
2023-07-19 - fdb94a577 - Do not use Unvailable as error type for expected error cases (#4650)
2023-07-19 - 724f23f25 - Cleanup new update related functional tests (#4563)
2023-07-19 - 2605c0b72 - Add support for ES Scroll for Scan API (#4614)
2023-07-19 - 02862114f - restructure history client for alternate ownership handling (#4636)
2023-07-19 - d7b4da84e - Extract common code from service/frontend/{fx,fx_test}.go (#4643)
2023-07-19 - 9a0745c48 - change the history cache key name to *Bytes (#4649)
2023-07-19 - a903b404c - Change nil metrics handle to no op handler (#4646)
2023-07-19 - 35ab8bdea - Add a test for NamespaceCountLimitInterceptor (#4639)
2023-07-19 - 0646c0a1d - Log when health rate limiter value changes (#4648)
2023-07-18 - e8ae1650e - Fail force-replication fast if any GenerateReplicationTasks/GenerateAndVerifyReplicationTasks activity returns error (#4642)
2023-07-19 - bf23997a1 - Better handle admin handler stream replication API lifecycle (#4647)
2023-07-18 - 8a75aed2b - Allow configurating history cache non-user context lock timeout (#4645)
2023-07-18 - 964a60068 - do not cache large blob history event (#4621)
2023-07-18 - 1caf08df5 - Make replication task convertor injectable by fx (#4635)
2023-07-18 - ba5b23d31 - Prioritize UI/tctl when system is throttled (#4623)
2023-07-17 - a4929062e - Ignore global rate limit if there are no hosts (#4627)
2023-07-17 - 72fe26c36 - Exclude history APIs from persistence health signals (#4637)
2023-07-14 - 7efba9fb0 - Add verification of replication tasks in force replication (#4630)
2023-07-14 - 2d86390f4 - Do not generate visibility task for unchanged BuildIds (#4632)
2023-07-14 - a01a309af - Support cluster metadata customized tags (#4622)
2023-07-14 - 2fab90b0f - Drop standby tasks in standby taks executor (#4626)
2023-07-14 - 7e2de6902 - Fix some minor warnings in the matching package (#4608)
2023-07-13 - cc38a4133 - Add activity / workflow task version check (#4580)
2023-07-13 - b326a4a0f - Fix metrics tagging in stream replication task processing (#4625)
2023-07-13 - 6e811dd84 - Handle replicate workflow state with exist workflow data (#4617)
2023-07-12 - 1b82336cd - Send default tasks to unversioned queue when user data disabled (#4610)
2023-07-12 - 9193a90f9 - Disable user data when fetch fails (#4604)
2023-07-12 - 2fd612343 - Deprecate QueueAckLevel (#4616)
2023-07-11 - 83faadf3c - Clean up cluster metadata initialization logic (#4531)
2023-07-11 - 87b0f6034 - Fix metrics double emission in admin and operator handler (#4613)
2023-07-11 - 7dddf7202 - Support cluster metadata customized tags (#4544)
2023-07-10 - d0c2a887d - Add frontend global RPS limit (#4583)
2023-07-10 - 3918d9342 - emit history service panic metric for all requests (#4497)
2023-07-10 - 687125a01 - move controller shard membership handling (#4607)
2023-07-10 - ec1a0106e - use a factory to create shard contexts (#4586)
2023-07-10 - 2374cc4bb - Add a Clock option to our LRU cache (#4593)
2023-07-10 - 3847f909d - Remove durable archival flag (#4594)
2023-07-10 - 9547c8328 - Disable confusing-naming linter and tune unhandled-error (#4606)
2023-07-10 - 0b0c398e4 - add stop reason for context (#4601)
2023-07-10 - 8f798a9d6 - Use time.NewTimer instead of time.After (#4596)
2023-07-07 - ec89a820d - properly fix migration in mem ack aggregation (#4571)
2023-07-07 - 5c9600d67 - Remove verbose logging (#4603)
2023-07-07 - 52997fdc3 - Remove time.Sleep from util_test (#4592)
2023-07-07 - 3860610f6 - Remove flusher package (#4595)
2023-07-07 - a3f530de4 - Update go version to 1.20 (#4597)
2023-07-07 - 619e2f968 - Update trigger-publish GHA (#4599)
2023-07-07 - 731cc3c8b - Fix MySQL 8 upgrade script to schema v1.3 (#4598)
2023-07-06 - 061dbb262 - wait for drain duration before stopping shard controller (#4587)
2023-07-06 - 92c841874 - Refactor fx service Start/Stop (#4584)
2023-07-06 - bd241c014 - Log VisibilityStoreConfig for debugging (#4588)
2023-07-06 - a84c2e0ad - support delay before history joins membership (#4582)
2023-07-06 - 15f201745 - Allow context disable frontend redirection (#4547)
2023-07-06 - 6d175c219 - move ringpop start call to service start (#4510)
2023-07-05 - 1ca697991 - Add help text to a few more metrics (#4529)
2023-07-05 - 17259dbc8 - Use DefaultServices for server_test (#4566)
2023-07-03 - db8b67a84 - Test speculative workflow task creation while completing previous workflow task (#4517)
2023-07-03 - 05e6ea0ce - Disable goerr113 in test files (#4574)
2023-07-03 - 4b4dbf397 - Make all Go source file names snake case (#4558)
2023-07-03 - 2a22821b4 - Remove common.Daemon interface (#4573)
2023-06-30 - 76b5f338e - Improve build id scavenger (#4568)
2023-06-30 - 3bb5c5802 - Fix migration in mem ack aggregation (#4569)
2023-06-29 - 6a92ddd4b - Versioning cleanup (#4556)
2023-06-29 - 2a2734e6e - Skip final task queue update if lost ownership (#4554)
2023-06-29 - 911d7c359 - Measure backlog after completing current task (#4562)
2023-06-29 - 598aa479c - Add corruption protection to ms (#4560)
2023-06-29 - f97237fa1 - Handle get namespace by id error (#4559)
2023-06-29 - 5e03077e0 - Use lowest persistence priority for standby tasks (#4551)
2023-06-29 - 435cffe9a - Add a test to our gRPC resolver (#4548)
2023-06-29 - c0fb121ac - Consider non default sets reachable by new workflows for a while after they stop being queue default (#4545)
2023-06-28 - 87cc6f421 - Get execution start/close time when using cassandra visibility store (#4549)
2023-06-27 - f51b96d92 - Add an error log detection test to the server (#4555)
2023-06-27 - a7b0afddc - Handle new fx event type (#4553)
2023-06-27 - 11a30b23d - Per shard per namespace RPS warning log (#4525)
2023-06-27 - 39bf6a8ff - Add Semgrep scanning (#4550)
2023-06-26 - 1e18621cb - Add more metrics for stream replication (#4539)
2023-06-26 - 7198aa74a - Remove unused metrics def (#4540)
2023-06-23 - b5979f96d - Update go dependencies (#4534)
2023-06-23 - 326020ac9 - Update versionChecker.go for the next release (#4533)
2023-06-23 - 7314ea42e - Update updateRegistry store on mutable state reload (#4532)
2023-06-22 - c52957bb5 - Make replication tasks units 'dimensionless' (#4515)
2023-06-22 - cca978f31 - Remove ReplicationTasksReturned (#4514)
2023-06-22 - 9405cf848 - Perform task range completion on shard rangeID update (#4528)
2023-06-21 - 61b7d6443 - Add an option for metrics descriptions (#4486)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.22.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Important

Temporal Server v1.21.0 introduced a security bug in the Cassandra tool (aka temporal-cassandra-tool) that can expose the credentials in debug level logs. The issue is fixed with this release in #4738.

All Changes

2023-08-14 - 1eaaaf8da - Increaes RecordHeartbeat frequency in VerifyReplicationTask (#4771)
2023-08-14 - 194a8938d - Skip verifying workflow which has already passed retention time (#4770)
2023-08-14 - 8e56612d5 - Turn ErrWorkflowClosing into a retryable error (#4765)
2023-08-14 - cadd9337b - Remove debug logging from Cassandra schema update tool (#4738)
2023-08-07 - 7e216c3bb - Bump server version to 1.21.5
2023-08-07 - 524bda390 - Fix bug causing duplicates when listing s3-archived workflows (#4712)
2023-08-07 - 5722c2332 - Continue replication verification by skipping workflow which should be or soon to be deleted (#4734)
2023-08-07 - a8ff17f49 - Change schedule jitter to depend on schedule + namespace ids (#4685)
2023-08-07 - 5f1455fac - Collapse duplicate visibility tasks (#4727)
2023-08-07 - 8bda654e6 - Return ResourceExhaustedError for consistent query buffer full (#4731)
2023-08-07 - 95a76efbd - Add worker versioning calls to read-only namespace APIs for authorizer (#4721)
2023-08-07 - 9970ea504 - Reduce the heartbeat size of VerifyReplicationTasks activity (#4705)
2023-08-07 - 9d59be327 - Finish idle schedule when a timer fires (#4602)
2023-08-07 - 120b8be8d - enable shard linger with time limit config (#4696)
2023-08-07 - 4448efacd - Put user data replication tasks in DLQ (#4609)
2023-08-07 - b8e7b2837 - support delaying shard close for membership change (#4659)
2023-08-07 - 12c95ddf0 - Return nil instead of task discarded error if namespace is invalid locally (#4678)
2023-08-07 - a7832c76b - add configurable ownership caching to history client (#4652)
2023-08-07 - 37a70494f - Minor fixes in force replication verification (#4675)
2023-08-07 - ed5aded36 - Fix flaky TestForceReplicationWorkflow_VerifyReplicationTaskNonRetryableError (#4669)
2023-08-07 - eb0ee5ed1 - restructure history client for alternate ownership handling (#4636)
2023-08-07 - b105e5b78 - move controller shard membership handling (#4607)
2023-08-07 - 23135a350 - use a factory to create shard contexts (#4586)
2023-08-07 - aeb47aab0 - add stop reason for context (#4601)
2023-08-07 - 6c80ff6e0 - wait for drain duration before stopping shard controller (#4587)
2023-08-07 - 35f6670c2 - support delay before history joins membership (#4582)
2023-08-07 - a6dfbc6ea - move ringpop start call to service start (#4510)

Schema Upgrades

See the 1.21.0 release notes for schema upgrade information for upgrading to the 1.21 series. There are no new schema changes in 1.21.4.

Release Highlights

The release removes recent changes to cache handling to fix regressions that could cause excessive memory usage and block workflows under high load (5dbbc4522ff1d43c4002b9850b5b678054ac5aaf, 6c48ed2d942a95e37ee8b1484546ebcbc8aed552).

Details about the v1.21.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.21.4)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Important

We identified a bug in this release relating to the mutable state cache, that could cause workflows to be blocked under high load. We recommend not deploying this release.

If downgrading from this release to the 1.20 series, downgrade to 1.20.4 instead of 1.20.3. This addresses an issue where the new BuildIds search attribute was not understood and would cause errors in visibility processing.

Schema Upgrades

See the 1.21.0 release notes for schema upgrade information for upgrading to the 1.21 series.

Release Highlights

This release fixes a number of bugs:

• Workflows that use the new Workflow Update feature can be blocked with "non deterministic" errors under very rare circumstances.
• Introduce an option in the force-replication workflow to verify workflows were successfully replicated and applied on a target cluster.
• Fixed standby task discarding logic so that only task that does verification will be dropped.
• Changed the Elasticsearch Scan API back to use scroll if point in time is not available.
• Few other minor bugs.

And improvements:

• Change history event cache size from count-based to byte size-based to avoid caching large blob history events. The dynamic config keys history.eventsCacheInitialSize and history.eventsCacheMaxSize are changed to history.eventsCacheInitialSizeByte and history.eventsCacheMaxSizeByte, respectively.

All Changes

2023-07-24 - 55c4d36f8 - Restore defaultListWorkflowsPageSize back to 1000 (from 100) (#4674)
2023-07-21 - d0974cdaa - Execute VerifyReplicationTasks as an individual activity (#4656)
2023-07-21 - 9f9b172f6 - Fail force-replication fast if any GenerateReplicationTasks/GenerateAndVerifyReplicationTasks activity returns error (#4642)
2023-07-21 - 6247d8814 - Add verification of replication tasks in force replication (#4630)
2023-07-21 - 1d5601a4b - Validate workflow task start time when complete (#4663)
2023-07-21 - 655c51aad - Exclude history APIs from persistence health signals (#4637)
2023-07-21 - 4edce52ec - Per shard per namespace RPS warning log (#4525)
2023-07-20 - 53517fd40 - Custom search attributes validation per store (#4655)
2023-07-20 - 2830f3816 - Do not use Unvailable as error type for expected error cases (#4650)
2023-07-20 - a9a9acbbb - Add support for ES Scroll for Scan API (#4614)
2023-07-20 - 67de6dfef - change the history cache key name to *Bytes (#4649)
2023-07-20 - 268e27408 - Log when health rate limiter value changes (#4648)
2023-07-20 - 32b4219ca - Better handle admin handler stream replication API lifecycle (#4647)
2023-07-20 - 7a125d94e - Allow configurating history cache non-user context lock timeout (#4645)
2023-07-20 - f97b681b3 - do not cache large blob history event (#4621)
2023-07-20 - 49b99fe6a - Drop standby tasks in standby taks executor (#4626)
2023-07-20 - a466c4da0 - Fix metrics tagging in stream replication task processing (#4625)
2023-07-20 - bcbb3b926 - Handle replicate workflow state with exist workflow data (#4617)
2023-07-20 - 561495045 - properly fix migration in mem ack aggregation (#4571)
2023-07-20 - 1ca8443ed - Allow context disable frontend redirection (#4547)
2023-07-20 - 674657bd2 - Skip final task queue update if lost ownership (#4554)

Details about the v1.21.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.21.3)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Important

• If downgrading from this release to the 1.20 series, downgrade to 1.20.4 instead of 1.20.3. This addresses an issue where the new BuildIds search attribute was not understood and would cause errors in visibility processing.

Schema Upgrades

See the 1.21.0 release notes for schema upgrade information for upgrading to the 1.21 series.

Release Highlights

This release fixes a bug that could cause delays in task processing during upgrades and excessive load on visibility. Also, it fixes a few minor other bugs.

All Changes

2023-07-14 - 9219be243 - Do not generate visibility task for unchanged BuildIds (#4632)
2023-07-12 - 74b51d84d - Send default tasks to unversioned queue when user data disabled (#4610)
2023-07-12 - d4cd7372b - Disable user data when fetch fails (#4604)
2023-07-12 - 71ae36c75 - Fix MySQL 8 upgrade script to schema v1.3 (#4598)

Details about the v1.21.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.21.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release addresses an issue when downgrading a cluster running the 1.21 series back to 1.20: New workflows would receive a BuildIds search attribute that would not be understood by 1.20 and would cause errors in visibility task processing. There's no need to upgrade to this release from 1.20.3, but if downgrading from 1.21, the downgrade should be done to this version.

All Changes

2023-07-13 - f12479619 - Add BuildIds predefined search attribute (#4624)

Details about the v1.20.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.4)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

IMPORTANT

• Do not upgrade to this version if you using advanced visibility with MySQL as storage. The release of v1.21.0 included an script to upgrade the schema, but it contains a bug. Please upgrade to v1.21.2 which fixes the bug.
• If downgrading from this release to the 1.20 series, downgrade to 1.20.4 instead of 1.20.3. This addresses an issue where the new BuildIds search attribute was not understood and would cause errors in visibility processing.

Schema Upgrades

See the 1.21.0 release notes for schema upgrade information for upgrading to the 1.21 series.

Release Highlights

This release fixes these bugs:

• Skip reading close event for retention task if not using Cassandra visibility store.
• Use lower priority for standby tasks.
• Added sanity check to protect against mutable state corruption.
• Fix replication ack aggregation logic.

All Changes

2023-06-30 - 873df3467 - Improve build id scavenger (#4568)
2023-06-30 - ed7f94cbe - Fix migration in mem ack aggregation (#4569)
2023-06-29 - 0fcff5159 - Measure backlog after completing a current task (#4562)
2023-06-29 - 275567976 - Add corruption protection to ms (#4560)
2023-06-29 - 8b3426bf1 - Handle get namespace by id error (#4559)
2023-06-29 - 72fb120c7 - Use lowest persistence priority for standby tasks (#4551)
2023-06-29 - ca41bb034 - Consider non default sets reachable by new workflows for a while after they stop being queue default (#4545)
2023-06-29 - 0d7350d6e - Get execution start/close time when using Cassandra visibility store (#4549)
2023-06-29 - 0c441a796 - Add more metrics for stream replication (#4539)

Details about the v1.21.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use the tag 1.21.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

IMPORTANT

• Do not upgrade to this version if you using advanced visibility with MySQL as storage. The release of v1.21.0 included an script to upgrade the schema, but it contains a bug. Please upgrade to v1.21.2 which fixes the bug.
• If downgrading from this release to the 1.20 series, downgrade to 1.20.4 instead of 1.20.3. This addresses an issue where the new BuildIds search attribute was not understood and would cause errors in visibility processing.

Schema Upgrades

This release requires schema upgrades for both persistence and visibility. See the schema upgrade documentation for the required commands for performing the upgrade.

Release Highlights

Worker Versioning (preview)

Links to docs:

• https://docs.temporal.io/workers#worker-versioning
• https://docs.temporal.io/dev-guide/java/versioning#worker-versioning
• https://docs.temporal.io/dev-guide/golang/versioning#worker-versioning

Description: Worker Versioning simplifies the process of deploying changes to Workflow Definitions.

⚠️ This feature requires Advanced Visibility

In the SDK, a user can identify a Worker with a Build ID, which should be associated at build time, and configure that Worker to use Versioning.

An operator can use the UpdateWorkerBuildIdCompatibility gRPC API (and the high-level APIs in the SDKs) to define which builds are compatible with each other and control task routing for existing and new Workflows based on the compatibility constraints.

Additionally, operators and automated scripts can use the GetWorkerTaskReachability API to query for build IDs that are no longer reachable and whose associated workers can be retired.

⚠️ To enable worker versioning, turn on the frontend.workerVersioningDataAPIs, frontend.workerVersioningWorkflowAPIs, and worker.buildIdScavengerEnabled dynamic config flags (disabled by default in this release).

This feature introduces a couple of schema changes:

1. A new BuildIds Search Attribute that is used to track reachability. This new Search Attribute requires a Visibility schema update. See the schema upgrade documentation for the required commands for performing the upgrade.
2. A new task_queue_user_data table, which stores the Versioning data for the various Task Queues and is replicated using the Namespace replication queue. See the schema upgrade documentation for the required commands for performing the upgrade.

The Matching Service will now load Task Queue user data from persistence for the Task Queue root partition, and non-root partitions will try to obtain and maintain a fresh view of the user data by long-polling other Task Queue partitions for the same Task Queue.

If user data loading is suspected to be causing processing overhead, it can be completely disabled by using the matching.loadUserData dynamic config flag.

Note that disabling the loading of user data will essentially disable the Versioning feature and will cause tasks for versioned Workflows to be dropped by the Matching Service, causing versioned Workflows to be stuck. This approach was chosen over dispatching versioned Workflows to arbitrary Workers and breaking the user-defined compatibility semantics. Manual intervention will be required to regenerate Tasks for stuck versioned Workflows.

As part of this work, the UpdateWorkerBuildIdCompatiblity, UpdateNamespace, and RegisterNamespace APIs now have stricter rate limiting because they can produce replication messages for Global Namespaces. The Namespace replication queue is a critical resource that is not partitioned and serves as a mechanism for delivering failover messages between Clusters.

A few more dynamic configuration options were added for this feature:

• limit.workerBuildIdSize is the byte length limit for a Worker Build ID as used in the RPC methods for updating the version sets for a Task Queue. Do not set this option to a value greater than 255 for Clusters that use SQL-based persistence due to predefined VARCHAR column width.
• limit.versionCompatibleSetLimitPerQueue is the maximum number of compatible sets allowed in the Versioning data for a Task Queue. Update requests that would cause the Versioning data to exceed this number will fail with a FailedPrecondition error.
• limit.versionBuildIdLimitPerQueue is the maximum number of Build IDs allowed to be defined in the Versioning data for a Task Queue. Update requests that would cause the Versioning data to exceed this number will fail with a FailedPrecondition error.
• limit.taskQueuesPerBuildId limits the number of Task Queue names that can be mapped to a single Build ID.
• limit.reachabilityTaskQueueScan limits the number of Task Queues to scan when responding to a GetWorkerTaskReachability query.
• limit.reachabilityQueryBuildIds limits the number of Build IDs that can be requested in a single call to the GetWorkerTaskReachability API.
• worker.removableBuildIdDurationSinceDefault is the minimum duration since a Build ID was last set as default in its containing set for it to be considered for removal. The value is used by the Build ID scavenger.
• frontend.rps.namespaceReplicationInducingAPIs limits the requests-per-second (RPS) rate for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.namespaceRPS.namespaceReplicationInducingAPIs is a per-host and per-Namespace RPS limit for Namespace replication-including APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.namespaceBurst.namespaceReplicationInducingAPIs is a per-host and per-Namespace burst limit for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.globalNamespaceRPS.namespaceReplicationInducingAPIs is a Cluster global, per-Namespace RPS limit for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). The limit is evenly distributed among available Frontend Service instances. If this option is set, it overwrites the per-instance limit configured with frontend.namespaceRPS.namespaceReplicationInducingAPIs. This config is EXPERIMENTAL and might be changed or removed in a later release.

Synchronous Update (preview)

Synchronous Workflow Update enables a gRPC client of a Workflow Execution to issue requests to that Workflow Execution and receive a response. These requests are delivered to and processed by a client-specified Workflow Execution. Updates are differentiated from Queries in that the processing of an Update is allowed to modify the state of a Workflow Execution. Updates are different from Signals in that an Update returns a response.

Updates can be handled by Workflows written using the latest Go and Java SDKs, with other languages to follow.

Any gRPC client can invoke Updates via the WorkflowService.UpdateWorkflowExecution API, which returns the Update outcome inline. Additionally, past Update outcomes can be observed via the WorkflowService.PollWorkflowExecutionUpdate API.

Update functionality is disabled by default but can be enabled per Namespace by setting the frontend.enableUpdateWorkflowExecution flag to true for that Namespace in dynamic config.

In this release, Update outcomes are stored for the same period as the Workflow history itself and can be observed for as long as that history exists. The execution and retention of Updates is configured via two optional dynamic configuration values:

• history.maxTotalUpdates controls the total number of Updates that a single Workflow Execution can support. The default is 2000.
• history.maxInFlightUpdates controls the number of Updates that can be “in-flight” (that is, concurrently executing, not having completed) for a given Workflow Execution. The default is 10.

Warning: when resolving history event conflict in global namespace, history event for update may be lost. This will be addressed in future release.

Batch Reset via BatchOperation API

Description: The server now supports resetting a batch of Workflow Executions. StartBatchOperationRequest now supports the Operation type of ResetOperation.

This operation depends on two additional parameters:

• ResetReapplyType: Determines whether to reapply Signals after resetting the Workflow.
• ResetType: Determines which Event ID to choose when resetting the batch of Workflows.
  • RESET_TYPE_FIRST_WORKFLOW_TASK: Resets to the Event of the first Workflow Task completed. If the Task doesn’t exist, resets to the Event after the Task was scheduled.
  • RESET_TYPE_LAST_WORKFLOW_TASK : Resets to the Event of the last Workflow Task completed. If the Task doesn’t exist, resets to the Event after the Task was scheduled.

Dynamic rate limiter

The dynamic rate limiter is an advanced system-protection mechanism. By default, it’s disabled. When enabled, the per-host persistence rate limiter automatically adjust based on the average number of errors returned from persistence and the average latency of persistence requests.

The rate limit is adjusted using a multiplier between 0.1 and 1.0 (default 1.0) on the host-level persistence rate limiter. If either the ratio of persistence calls that result in an error or the average latency of persistence requests exceeds their respective thresholds, the multiplier is reduced. If the multiplier is less than its maximum (1.0) and neither threshold is exceeded, the multiplier is increased.

Several new dynamic config options are available to tune these adjustments. They are grouped in a map for each service, with the key <service>.persistenceDynamicRateLimitingParams, but currently only apply to the History Service.

• enabled: Whether dynamic rate limiting is enabled for this service (default false).
• refreshInterval: How often the rate limit multiplier and dynamic config options are updated (default 10s).
• latencyThreshold: The limit for average latency that should trigger a rate limit reduction when exceeded (default 0.0; not used).
• errorThreshold: The ratio of errors to total requests that should trigger a rate limit reduction when exceeded (should be between 0 and 1; default 0.0; not used).
• rateBackoffStepSize: The amount the rate limiter multiplier is reduced when either limit is exceeded (should be between 0 and 1; default 0.3).
• rateIncreaseStepSize: The amount the rate limiter multiplier is increased when neither threshold is exceeded and the multiplier is less than its maximum (should be between 0 and 1; default 0.1).

Updated defaultAuthorizer and defaultJWTClaimMapper

Important: If you use defaultAuthorizer or defaultJWTClaimMapper, this is a breaking change. You need to update your JWT generator to produce permission for the system-level role as “temporal-system:role” instead of “system:role”, where role is read, write, admin, or worker. For safe rollout, you can add “temporal-system:role” to the list of permissions before rollout and remove “system:role” after rollout is done.

Related PRs: https://github.com/temporalio/temporal/pull/4227 https://github.com/temporalio/temporal/pull/4244

Note that defaultAuthorizer and defaultJWTClaimMapper are not used by default. You must explicitly configure it to use them.

• Remove access to AdminService and OperatorService APIs from System RoleWriter.
• Remove the Namespace lowercase change. Admins of Namespace “Foo” will not have access to namespace “foo”.
• Previously, the defaultJWTClaimMapper treated “system” as a special Namespace, and granted system-level access to the admin of the Namespace named “system”. This is now changed to use “temporal-system”, which is a real system Namespace. Roles on the “temporal-system” Namespace will have system-level access.

Visibility

Deprecation heads up

• Standard Visibility is entering a deprecation process. We are marking it deprecated with this version (1.21), and we expect to remove it in v1.23.0. If you are using Cassandra as your Visibility store, check the deprecation warning below. If you use Standard Visibility with a SQL database, see the v1.20.0 release highlights for details about upgrading to Advanced Visibility. If you want to migrate to a different Visibility store, see Visibility store.
• Cassandra, as a Visibility store, will be deprecated together with Standard Visibility. Please migrate away from using Cassandra as your Visibility store. For migration information, see Visibility store.
• Docs links:
  • Concept - Visibility
  • Cluster Deployment - Visibility store configuration

Configs

Some Visibility configs have changed. The following lists all the changes, and all the old keys will be removed in the next release.

For the Server config

• Configure Elasticsearch as a Visibility store
  • New key: visibilityStore
  • Old key: advancedVisibilityStore

For dynamic configs

• Enable reading from secondary Visibility
  • New key: system.enableReadFromSecondaryVisibility
  • Old keys: system.enableReadVisibilityFromES / system.enableReadFromSecondaryAdvancedVisibility
• Secondary Visibility writing mode
  • New key: system.secondaryVisibilityWritingMode
  • Old keys: system.enableWriteToSecondaryAdvancedVisibility / system.advancedVisibilityWritingMode
• Visibility maximum read QPS
  • New key: system.visibilityPersistenceMaxReadQPS
  • Old keys: system.standardVisibilityPersistenceMaxReadQPS / system.advancedVisibilityPersistenceMaxReadQPS
• Visibility maximum write QPS
  • New key: system.visibilityPersistenceMaxWriteQPS
  • Old keys: system.standardVisibilityPersistenceMaxWriteQPS / system.advancedVisibilityPersistenceMaxWriteQPS

List and Scan APIs

The List and Scan APIs have the following changes.

• For List and Scan API, the default maximum page size is reduced from 10000 to 1000. If you call with a larger number, it will cap at 1000, and it will not throw an error. As for pagination, as long as you get a non-null token, there might be a next page: a page with less than 1000 does not necessarily mean that it’s the last page. The maximum page size can be changed using the dynamic config frontend.visibilityMaxPageSize; the dynamic config frontend.esIndexMaxResultWindow has been removed.
• The Scan API, even though its documentation says that the result set is not guaranteed to be sorted, is used to support the ORDER BY clause, and it had a default order. With this release, the ORDER BY clause is no longer supported. For Elasticsearch, it has been optimized for faster scanning without guaranteeing order.
• When using Elasticsearch, the Scan API is used to do a scroll search or open a point in time search if available. Now, it supports only point in time searches, as supported by Elasticsearch 7.10 and later. Currently, we do not support OpenSearch, and the Scan API won’t work with OpenSearch. [Update 7/24/2023: this was reverted in v1.21.3]
• Support for the ORDER BY clause is now disabled by default. You can enable it by setting the dynamic config system.visibilityDisableOrderByClause to false, and it can be set per Namespace. Although you can still enable it, this feature will likely be removed in a future release.

Frontend Service shutdown behavior

Description: The shutdown procedure for the Frontend Service has been changed slightly to work better with load balancers.

• First, the Frontend Service returns NOT_SERVING to gRPC health checks for the duration of the dynamic config frontend.shutdownFailHealthCheckDuration (default 0 seconds).
• Second, the Frontend Service tells gRPC to stop accepting new connections and RPCs. Currently running RPCs (including long polls) will not be interrupted yet.
• After the value of the dynamic config frontend.shutdownDrainDuration (default 1 second), currently running RPCs will be canceled.
• The sum of the two durations is limited to 300s. After that time, the process exits.

The default maximum long-poll duration is 60s. To avoid any client errors, we recommend setting frontend.shutdownDrainDuration to at least the 60s and setting frontend.shutdownFailHealthCheckDuration to at least as long as it takes your health checking system to notice the failed health check and stop sending traffic to that Frontend Service.

Temporal SDKs expect errors and retry intelligently, so these settings are not required for the correct execution of Workflow.

(The previous behavior had incoming RPCs during the drain interval return an application-level “Shutting down” error instead of a gRPC-level error. Also, the sum of the two durations was limited to 120s.)

Dynamic config validation

Description: For dynamic config values that are interpreted as durations, previously, an integer value (that is, with no unit specified) would be treated as an error, and the default value would be used instead. Now integers are treated as seconds. (Floats are still not allowed.)

It’s highly recommended to use values with units for clarity. If using the default YAML-based dynamic config, use strings like "30s" or "5m" instead of 30 and 300. If using a custom dynamic config implementation, ensure that values are returned with the Go type time.Duration.

Additionally, dynamic config values that fail to parse are logged at warning level. (After some amount of them, they’re logged only sporadically).

Per workflow limit

Related PR: https://github.com/temporalio/temporal/pull/4326

Description: The default values for various per-Workflow limits have been reduced or tightened. More specifically:

• limit.numPendingChildExecutions.error, limit.numPendingActivities.error, limit.numPendingSignals.error, and limit.numPendingCancelRequests.error have been reduced to 2000. If exceeded, Workflow Tasks that generate new Child Workflows or Activities, signal external Workflows, or request cancellation of external Workflow Commands will fail. Be aware that there’s no partial acceptance of Workflow Commands. As a result, if Commands in a single Workflow Task response exceed those limits, the Workflow Task will keep failing.
• history.maximumSignalsPerExecution is now 10,000. If exceeded, new Signal requests sent to the Workflow are rejected and return an InvalidArgument error.

All changes

2023-06-23 - 521641f62 - Update updateRegistry store on mutable state reload (#4532)
2023-06-22 - b8feb950a - Perform task range completion on shard rangeID update (#4528)
2023-06-21 - 869991655 - Prevent scavenger from removing build ids that were recently default for a set (#4527)
2023-06-21 - 5b4e2e317 - Add per build id timestamp for when it was last made set default (#4526)
2023-06-21 - cdf158e4d - Add per version set timestamp for when it was last made default in queue (#4524)
2023-06-21 - 2cd157975 - Simplify XDC test logic (#4522)
2023-06-21 - f2fb09850 - Add string dynamic config filter by namespace id (#4523)
2023-06-21 - 198f0e498 - remove monitor WhoAmI (#4506)
2023-06-20 - a62450acd - Bring back branch token for branch deletion (#4521)
2023-06-20 - 7f064eac4 - Wrap setting futures in taskQueueManager (#4494)
2023-06-20 - 35ad0a599 - Validate startedEventId when close WorkflowTask (#4519)
2023-06-20 - 36c60c369 - Update duplicate WorkflowExecutionInfo CLI proto (#4520)
2023-06-16 - bdd58017f - Use search attribute size limit for BuildIds truncation (#4516)
2023-06-16 - 20baa97cf - Spool tasks with "default" versioning directive along with unversioned tasks (#4512)
2023-06-16 - 7f3bb1a0e - Apply stricter rate limiting for the RegisterNamespace API (#4513)
2023-06-16 - 3392a7ab5 - Fix wrong format specifier in client_gen (#4511)
2023-06-16 - bddbc7439 - Cluster not found (#4501)
2023-06-16 - 9330f5253 - Record ES bulk took time after err check (#4508)
2023-06-15 - 187f699a6 - Add InvalidStateTransitionWorkflowExecutionUpdateCounter metric (#4503)
2023-06-15 - 0a109d579 - Add missing reset type params (#4505)
2023-06-15 - 6a9d9d8a4 - Fix typo in batch reset (#4504)
2023-06-15 - 1644435ae - Check if WorkflowTaskFailed event is nil when workflow completes (#4500)
2023-06-15 - ae2d232f8 - Refactor matching operations when user data disabled (#4493)
2023-06-14 - 73189b7bd - Return NonRetryableApplicationError if input is invalid in ForceReplicationWorkflow and NamespaceHandoverWorkflow (#4498)
2023-06-14 - c712b57e2 - Fix error checking (#4496)
2023-06-14 - 021a84e21 - Functional tests: Set ReturnNewWorkflowTask for task poller always to true (#4476)
2023-06-14 - c5efe932b - Remove usages of AcceptedRequest field (#4489)
2023-06-14 - 2db855d12 - Schedule a new workflow task after force flush buffer events (#4490)
2023-06-14 - 8795c6c12 - Add namespace migration support for replication stream (#4488)
2023-06-14 - 741edb50e - Test update not found due to shard reload case (#4495)
2023-06-13 - 363e92c98 - Disable revive.unexported-return linter (#4466)
2023-06-13 - a24557b4e - Log errors instead of crashing in OTel metrics handler (#4485)
2023-06-13 - 598b0c387 - Fix some typos and comments in metrics package (#4478)
2023-06-13 - 03ca8e8f2 - Refactor task queue user data loading (#4487)
2023-06-13 - 71748a0cf - Only replicate user data for multi cluster setups (#4484)
2023-06-12 - ea9ad7854 - Reuse unloadTaskQueue helper in versioning_test (#4482)
2023-06-12 - a6ae65114 - Fix some invalid metrics definitions (#4465)
2023-06-12 - fc281193f - Increase test timeout to 30 minutes (#4480)
2023-06-12 - 75ffab3a5 - Add TestDescribeTaskQueue to versioning tests (#4474)
2023-06-12 - 6e432225b - Add TestDescribeWorkflowExecution to versioning tests (#4473)
2023-06-12 - 30c9be548 - Add worker identity to TestUpdateWorkflow tests (#4475)
2023-06-10 - eeec6e01e - Get rid of MetricName type (#4479)
2023-06-09 - eeee3a069 - Update poller info on base tqm (#4472)
2023-06-09 - 06d2ce841 - Add dynamic config to disable loading task queue user data (#4477)
2023-06-09 - d44aa1e7e - Add special rate limiter for namespace replication inducing APIs (#4455)
2023-06-09 - cee5ba319 - Move poller map to matchingEngine (#4471)
2023-06-09 - 65300d506 - Remove duplicate keys from dynamic config (#4470)
2023-06-09 - e100485c4 - Block versioned cross-tq activities, child workflows, CAN (#4459)
2023-06-09 - bef83ae71 - Create normal workflow task if unseen events exist (#4469)
2023-06-09 - 54abd995d - Merge history tree insert with history node append (#4464)
2023-06-09 - 30be627e5 - Implement build id scavenger (#4430)
2023-06-08 - d8f70e15a - Remove unused metrics.ServiceIdx (#4454)
2023-06-08 - 912b8890b - Remove unused MetricType concept (#4452)
2023-06-08 - 137886a7b - Implement merge sets operation for Build IDs (#4447)
2023-06-08 - dd4c7de94 - Enable workflow update is server development configs (#4461)
2023-06-08 - ac1944fde - Enforce uniqueness of replication stream (#4448)
2023-06-08 - a4bc99111 - Add more asserts on history events attributes (#4456)
2023-06-08 - 27d0a1f48 - Use task queue base name for dynamic config (#4445)
2023-06-08 - e5d58dc7e - More functional tests for worker versioning (#4446)
2023-06-08 - f8453ceb7 - Hide BuildIds search attribute for schedules (#4458)
2023-06-07 - 7608a0c8b - Add retryable client for persistence task manager (#4457)
2023-06-07 - a51abcfff - Comments and event attributes in expected history in tests (#4426)
2023-06-07 - b61c3f351 - Drop task when namespace is not on the cluster (#4444)
2023-06-06 - 79e212d85 - Add functional test to deduplicate updates by ID (#4438)
2023-06-06 - da83b0a5e - Add the event batch ID to update CompletionInfo (#4441)
2023-06-06 - c107bfd3b - Take acceptedEventID write out of commit callback (#4443)
2023-06-06 - 25c734287 - use existing acquire shard test setup (#4442)
2023-06-05 - 81c619766 - Remove unneeded FlushAndCreateNewBatch call from AddWorkflowTaskScheduleToStartTimeoutEvent (#4439)
2023-06-05 - d2ec9692c - Add functional test for shard reload during update (#4437)
2023-06-02 - 6691620a2 - Write accepted event ID in update completion event (#4436)
2023-06-02 - 426f6d17f - Improve flexibility of UpdateInfo proto message (#4433)
2023-06-02 - de7f6790d - Dynamic rate limiter (#4390)
2023-06-02 - 3bee7f782 - Upgrade OTEL module to the latest version (#4423)
2023-06-02 - c4c6d6d2d - Limit max number of update for workflow execution (#4413)
2023-06-02 - 2c90feb99 - Add test for heartbeat from speculative workflow task (#4425)
2023-06-02 - ce4734012 - Fix task inMemoryNoUserLatency 2.0 (#4431)
2023-06-02 - b34d21a99 - Fix handling of batch signal input payloads (#4374)
2023-06-01 - d619d79fc - Limit build IDs per queue to 100 (#4415)
2023-06-01 - d909a9673 - Fix task inMemoryNoUserLatency (#4429)
2023-06-01 - 5bc1a7248 - Add update counts to mutable state statistics (#4422)
2023-05-31 - e8c2a19f4 - Fix TestDefaultPublishRetryPolicy (#4421)
2023-05-31 - 35e7696cc - Remove speculative workflow task restoration code (#4424)
2023-05-31 - 10a14e319 - Inline UpdateInfo Storage (#4420)
2023-05-31 - 5e0459e17 - Add functional test for update with existing scheduled workflow task (#4414)
2023-05-30 - 2a4aec073 - Make server fx init more extendable (#4419)
2023-05-30 - 8aab8ab9c - Improve task latency calculation and retry behavior (#4408)
2023-05-30 - 35708d94e - Fix delete visibility task notification (#4416)
2023-05-26 - d2f251638 - Refactor ES query converter (#4382)
2023-05-26 - 81eaa7759 - Allow workflow to complete if there are updates in the registry (#4412)
2023-05-26 - 7e64d1d41 - Final update of api/sdk for worker versioning
2023-05-26 - a9505282f - Allow GetTaskQueueUserData and matching long polls to use a 5m timeout (#4407)
2023-05-26 - 2de41e587 - Handle worker versioning for sticky queues (#4397)
2023-05-26 - b717268be - Address comments from #4346 review (#4405)
2023-05-26 - 188a7ec84 - Update API and SDK to latest master (#4402)
2023-05-26 - 209e2501f - Worker versioning leftovers (#4400)
2023-05-26 - 8a325f86e - Implement GetWorkerTaskReachability API (#4346)
2023-05-26 - afe0304a1 - Add dynamic config to disable worker versioning progress APIs (#4398)
2023-05-26 - 453f54a18 - Server updates for use_compatible_version (#4394)
2023-05-26 - 773aefd8c - Add a few functional tests for worker versioning (#4375)
2023-05-26 - 8ffe902bc - Redirect spooled tasks using versioning directive (#4357)
2023-05-26 - 3805cfdf5 - Implement use_latest_build_id for activities, child workflows, continue-as-new (#4347)
2023-05-26 - 42e766e9d - Propagate task queue user data with long-poll requests (#4334)
2023-05-26 - bc58ba146 - Fix some versioning redirection logic (#4286)
2023-05-26 - 53a4a3aab - Worker versioning - add BuildIDs search attribute (#4284)
2023-05-26 - 6b4e3cc1f - Redirect to versioned queues in matching (#4241)
2023-05-26 - ff6e6af7c - Minor fixes
2023-05-26 - 2d2f94ba4 - Implement task queue user data seed (forced replication) (#4259)
2023-05-26 - 9e9f9cae8 - Rename imports to match convention (#4261)
2023-05-26 - 6e7917bec - Implement task queue user data replication (#4237)
2023-05-26 - 1a3e27766 - Copy build id to binary checksum when present (#4230)
2023-05-26 - 931adff08 - Thread version stamps through from history to matching (#4220)
2023-05-26 - 35dfeede1 - Misc versioning tasks (#4221)
2023-05-26 - 219f67479 - Pass worker version capabilities through poller info (#4219)
2023-05-26 - 283931a5b - Refactor to use OwnsUserData (#4218)
2023-05-26 - 84437cf01 - Versioning data merge algo and hybrid logical clock utils (#4205)
2023-05-26 - 0a9785a15 - Refactor to use new protos and persistence (#4194)
2023-05-26 - c2fdc9f2b - Proto and persistence definitions for replicated versioning (#4172)
2023-05-26 - 47d3fe5f9 - Hot shard metrics (#4365)
2023-05-26 - 759aa2cfe - Reject signal if workflow attempted to close (#4395)
2023-05-26 - 01034e4b4 - Add ES point in time to Scan API (#4380)
2023-05-26 - d1e1243bd - Rewrite some workflow history size calculation (#4258)
2023-05-26 - 15bc10d5c - Fix mutable state get event error handling (#4396)
2023-05-26 - 6132e5831 - Improve retry util (#4403)
2023-05-26 - df9162b7b - Bump CI builder base version (#4406)
2023-05-26 - eb2845e10 - Fix prometheus metrics (#4393)
2023-05-25 - 0059b83ea - Use WTFailed.EventId as completion_event_batch_id when workflow is terminated or timed out (#4392)
2023-05-25 - 9aa4ade25 - Fix merge issues from PR 4317 (#4404)
2023-05-25 - 378e1e60f - Add function to get custom search attributes mapper in namespace registry (#4317)
2023-05-24 - fea13adee - Do not generate scheduled task with timestamp in the past (#4367)
2023-05-24 - c2aee406d - Refactor UpdateWorkflowExecution functional tests (#4388)
2023-05-24 - 2fce7a56c - Enforce mutable state size limit (#4350)
2023-05-24 - df69684c7 - Bugfix: backfill replication queue state read ID calculation (#4391)
2023-05-24 - ca28eb077 - Fix partial namespace creation (2.0) (#4330)
2023-05-23 - 68856fef7 - Do not let API call timeout if workflow can't be locked (#4341)
2023-05-23 - b191203e4 - Avoid making a subscope in tallyMetricsHandler if possible (#4363)
2023-05-23 - e720b5533 - fix: create btree_gin extension only if not exists (#4371)
2023-05-23 - 6be18a183 - Add functional test with OpenSearch 2 (#4124)
2023-05-23 - 6b600dcf7 - Add metric to track ES bulk response took field (#4355)
2023-05-19 - 981d22bbe - Remove unused toDC config (#4370)
2023-05-19 - 09fc4c644 - Fail workflow execution update fast if workflow task constantly fails (#4376)
2023-05-19 - 944b893e5 - Add functional test for speculative workflow task in sticky task queue timeout (#4369)
2023-05-19 - d73481ae4 - Add functional test to cover conversion of scheduled speculative workflow task to normal (#4364)
2023-05-19 - c719eac18 - Capture UpdateInfos in MutableState mutation (#4372)
2023-05-19 - ed4960423 - Return # of state transition when generating last replication tasks (#4352)
2023-05-19 - f72345035 - Bugfix: do not leak gRPC stream (#4368)
2023-05-18 - e61de47cd - Add functional test to terminate workflow with pending speculative workflow task (#4366)
2023-05-18 - 680e4e170 - Fail query fast if workflow task attempt greater or equals 3 (#4359)
2023-05-18 - 9a6edae08 - Fix: failWorkflowTask returns wrong nextEventBatchID when failing speculative workflow tasks (#4354)
2023-05-18 - 3bfbb814b - Don't return workflow task if it is speculative and doesn't have any messages (empty) (#4356)
2023-05-17 - 007606d66 - Fix: use normal task queue if sticky worker unavailable during update workflow API call (#4353)
2023-05-16 - ab2f6b9b3 - Ensure workflow key is fully specified (#4351)
2023-05-16 - b9a9029e7 - Use the correct event version for task refresh (#4349)
2023-05-16 - f74966f15 - Increase default number of Elasticsearch bulk processor workers to 2 (#3738)
2023-05-16 - c42e7dd64 - Per-shard per-namespace rate limiter (#4302)
2023-05-16 - 5e9f623d5 - Add a config flag for enabling async updates (#4342)
2023-05-15 - 1240df2ee - Fix mutable state access after workflow lock is released (#4333)
2023-05-15 - d5f026449 - Tag metric tracking custom order by with namespace (#4344)
2023-05-15 - d2014911c - Bugfix: ExecutableTaskTracker LowWatermark funcion (#4343)
2023-05-15 - 9f9b828f8 - Fix misc. context value propagation issues (#4279)
2023-05-15 - 9712711c1 - Handle ProtocolMessageCommands in WFT completion (#4328)
2023-05-15 - 1a479ad80 - Use sequential task processor for replication (#4339)
2023-05-15 - 2607d6159 - Add some documentation to the maximumInterval field of ExponentialRet… (#4316)
2023-05-15 - d4af86087 - Add some documentation to the maximumInterval field of ExponentialRet… (#4318)
2023-05-15 - 467ea4c7b - Update Go SDK to v1.22.2 (#4327)
2023-05-15 - 11f4e6852 - Add linters goerr113, paralleltest, and tparallel (#4239)
2023-05-15 - 0d140a3c3 - Fix caller info for migration activities (#4340)
2023-05-15 - 9d7545b09 - Truncate activity failure in mutable state (#4338)
2023-05-14 - a05d21cdc - Fix: set nonRetryable field of server failure to "true" when failure size exceeds limit (#4335)
2023-05-14 - d6ab8c531 - Add additional metrics for stream replication (#4323)
2023-05-12 - a136de7ae - Make sure shard info always stays in a cleanup state (#4336)
2023-05-12 - aca0d911a - Change schedules default catchup window to 1 year (#4332)
2023-05-12 - f84a3c370 - Handful of minor changes to the update package (#4331)
2023-05-12 - f07921b38 - Update api-go (#4320)
2023-05-12 - 62611e2e5 - Refactor task queue related methods of mutable state (#4325)
2023-05-11 - a343beaba - Revert "Fix partial namespace creation (#4267)" (#4322)
2023-05-11 - cd8821c85 - Enforce stricter per workflow limit (#4326)
2023-05-11 - 40e76275a - Remove unneeded duration/seconds conversion (#4321)
2023-05-11 - 6e7c669ae - Create dynamic config to enable manual pagination for Elasticsearch (#4314)
2023-05-11 - a3ae60298 - Add replication dlq metrics (#3876)
2023-05-11 - 3cbab42bc - Fix partial namespace creation (#4267)
2023-05-11 - 58e247bf8 - Prevent workflow close when updates are undelivered (#4313)
2023-05-11 - a6861bfb0 - Async Updates and the Poll API (#4309)
2023-05-11 - a1fe0e6a5 - Manually build ES pagination query for default sorter (#4271)
2023-05-11 - d040cdfb3 - Improve Update.WaitAccepted - followup (#4312)
2023-05-11 - 3bfac8a19 - Improve Update.WaitAccepted (#4311)
2023-05-10 - f34d5b778 - Default disable order by clause support (#4305)
2023-05-10 - 8ed296c12 - Use executable task tracker to dedup tasks when stream is re-established (#4310)
2023-05-10 - 25b7cbc73 - Add a per-workflow limit to in-flight updates (#4307)
2023-05-10 - 233953103 - Fix adding multiple search attributes of same type at once (#4273)
2023-05-10 - 1c7e8955f - Refactor the Update type to a state machine (#4297)
2023-05-10 - dc7f28df4 - Skip over entire time range if paused and batch and cache time queries (#4215)
2023-05-10 - 807b791d2 - Add automaxprocs. (#4301)
2023-05-09 - 46caef3df - Refactor matching handlerContext (#4303)
2023-05-09 - 15974517a - Simplify task queue liveness (#4290)
2023-05-09 - 9f6651042 - Use search attribute type map in visibility archival (#4304)
2023-05-09 - d0007b065 - In memory timeout timer queue for speculative workflow task (#4254)
2023-05-09 - 8bc74cb43 - Add server stream interceptor (#4300)
2023-05-08 - 5af117ea3 - Bring PollWorkflowExecutionUpdate to HistoryEngine (#4298)
2023-05-08 - 8e6203a45 - Event buffer size limit (#4296)
2023-05-08 - 8fa4d3f71 - Correct spelling: updat_id -> update_id (#4299)
2023-05-08 - 89f707b1b - Small refactor to matching signalFatalProblem (#4292)
2023-05-08 - a43d451b8 - Enable Elasticsearch body compression with gzip (#4289)
2023-05-05 - 392e769fb - Bump gocql to v1.4 (#4294)
2023-05-05 - 9fd0e2607 - Handle namespace not found in history scan (#4287)
2023-05-05 - 08dcc7055 - Expose better logs for dynamic config conversion errors (#4282)
2023-05-05 - c243c084c - Verify WFTs with large event buffers fail (#4272)
2023-05-05 - 9fdd08c79 - Make shard ownership more resilient when shard is busy (#4283)
2023-05-04 - 0bdbbfc73 - Allow plain int seconds for duration in dynamic config (#4280)
2023-05-04 - bb7d4afe5 - Introduce update.Storage (#4277)
2023-05-04 - ea4946c9e - Use GracefulStop in frontend (#4212)
2023-05-04 - e6e2372b0 - Add Reset to Batch API (#4157)
2023-05-03 - 1d1ef27d2 - Remove force task refresh in passive processing logic (#4278)
2023-05-03 - 6e43d5432 - Fix task attempt reset (#4276)
2023-05-03 - b757d8b20 - Fix metric tag for delete execution task (#4275)
2023-05-03 - 5783e7815 - Adds update state indexing to MutableState (#4224)
2023-05-01 - d4cd05e04 - Remove support for ES PIT and scroll in ES client (#4223)
2023-05-01 - b6525f179 - Remove unused matching DB layer impl (#4141)
2023-05-01 - 2763a0f3b - Remove usages of ES PIT and Scroll (#4249)
2023-05-01 - 661d6d431 - Refactor fx Start/Stop usage (#4248)
2023-05-01 - 80687d797 - Remove unused functions in shard interface (#4255)
2023-05-01 - 2aa548199 - Remove unused dynamicconfig.Collection in top level fx container (#4253)
2023-04-28 - d6d7a0b0b - Add more logging (#4252)
2023-04-28 - 625f6d2c5 - Increase shutdown timeout to 5 minutes (#4195)
2023-04-28 - d94444d06 - Change dynamic config for worker versioning to be specific to data APIs (#4238)
2023-04-28 - 10c20a6f0 - Update dynamic config related to visibility (#4250)
2023-04-28 - 8740a205f - Support signals that do not immediately generate a workflow task (#4091)
2023-04-27 - 7a24d8efb - defaultJWTClaimMapper treat namespace as case sensitive (#4244)
2023-04-27 - 62c35aa97 - Remove noisy queue action log (#4245)
2023-04-27 - 1ea69e026 - Turn autofix for golangci-lint and add goimports make target (#3958)
2023-04-27 - 4d274876d - Run no-op claim mapper even without auth info (#4197)
2023-04-27 - 23147c36c - Attempt to unify queue state (#4190)
2023-04-26 - ccbfa7d47 - Add histogram support in metricstest (#4177)
2023-04-26 - b1caaa004 - Code style simplification and improved readability for returning errors (#4236)
2023-04-26 - cb0a9c792 - Code style fix (#4235)
2023-04-26 - c0029aa54 - [WIP] Fix error return bug and some minor code style (#4234)
2023-04-25 - d8d9289a9 - Fix test race condition (#4232)
2023-04-25 - ec3ff8bbd - Block non admin role to access system services (#4227)
2023-04-25 - 5797648dd - Fix broken tests (#4231)
2023-04-25 - 8eb42e314 - Bugfix shard ID mapping util & stream receiver monitor (#4229)
2023-04-25 - 7c4f3fd2a - Rename some var names in replication stream logic (#4211)
2023-04-25 - 2c2fb71fc - Support ES secondary_visibility index in docker config template (#4213)
2023-04-25 - aceb87878 - Use sqlite by default for functional test (#4210)
2023-04-25 - 683064c1a - Add sequential task scheduler (#4130)
2023-04-24 - e5372ec2b - Add dynamic config flag to disable worker versioning (#4222)
2023-04-24 - 0953f9b62 - Call List API from Scan API for SQL visibility (#4217)
2023-04-24 - 37cebd0b0 - Update vis store dev configs (#4208)
2023-04-24 - 6715e3819 - Revert "Enable Scan API in SQL visibility store" (#4214)
2023-04-24 - 12655c469 - Refactor configs for visibility (#4095)
2023-04-24 - b97be7814 - Wire replication stream caller side (#4178)
2023-04-21 - a52bb7331 - Test sync update on sticky task queue (#4200)
2023-04-21 - ba09219d4 - Fix cron for schedules that do not match any time (#4206)
2023-04-21 - d6c298b35 - Use int64 as type for queue readerID (#4176)
2023-04-21 - feb938eae - Add flush method to flush buffer (#3521)
2023-04-20 - 6c39401e5 - Validate messages against pending updates (#4188)
2023-04-20 - 0ca9e83a1 - Convert speculative workflow task to normal when it is persisted (#4069)
2023-04-19 - 501fdae54 - Implement batch history replication API (#4167)
2023-04-19 - bdbec131c - Rename UpdateQueueState to SetQueueState (#4187)
2023-04-19 - becc2f3d5 - Remove DefaultVisibilityIndexName from history config (#4189)
2023-04-19 - 8d3836a50 - Don't log error for ownership lost in matching task reader (#4183)
2023-04-19 - 2ff8d379b - Use correct API for transfer queue (#4186)
2023-04-19 - 8b7e3f7d9 - Add visibility manager dependency to delete manager (#4138)
2023-04-18 - af066c3f8 - Enable Scan API in SQL visibility store (#4161)
2023-04-17 - e792a6808 - Add util to map shard IDs (#4175)
2023-04-17 - 0cf036505 - Handle errors when registering new queue readers (#4055)
2023-04-17 - 74d3f15ab - Enforce visibility max page size (#4164)
2023-04-17 - 3cf1eb811 - Abort a task if schedule shuts down (#4125)
2023-04-17 - 2d0272c6e - Remove unused exporter from openTelemetryProviderImpl (#4169)
2023-04-14 - 52b036574 - Remove version_set_id from CompatibleVersionSet (#4170)
2023-04-14 - c1ec2bb4c - Switch from opaque branch token to branch info (#4168)
2023-04-13 - cdd1a218f - Split history tree insert from history node append (#4147)
2023-04-13 - 87dbab0ad - Update Go SDK to v1.22.1 (#4166)
2023-04-12 - b8aab1034 - Update Go SDK to v1.22.0 (#4163)
2023-04-12 - 1e2ecc45a - Fix schedule catchup window metric calculation while paused (#4152)
2023-04-12 - 04729dc27 - Add a hermetic PipeListener for testing net code (#4098)
2023-04-12 - cb70d5816 - Use fx parameter object pattern for ringpop factory (#4115)
2023-04-11 - 0e18e19d2 - Validation API should use low priority lock (#4140)
2023-04-07 - 3ce9a171e - Fix error handling add search attributes (#4148)
2023-04-05 - 0d6e92f8b - Addressing comments in PR 4134 (#4144)
2023-04-01 - 6f1f02280 - Change Scan API to have no Order(ES) (#4134)
2023-03-31 - ca1d906a5 - Do not generate workflow timeout task if workflow is closed (#4135)
2023-03-31 - 2ef3afd55 - Properly handle replication task lifecycle (#4126)
2023-03-31 - 7244681f3 - Reconnect on a few more gocql errors (#4132)
2023-03-31 - c97c973ec - Use correct time for initial schedule memo (#4128)
2023-03-31 - fec04d12d - Reuse timer in schedule workflow (#4129)
2023-03-31 - 8955ebfc3 - Cleanup redundant input args (#4096)
2023-03-29 - 0ca5360a1 - Auto clear stickiness (#4104)
2023-03-29 - d4b64e854 - Reduce the # of resend events if workflow is a reset workflow (#4123)
2023-03-29 - 9bc7eddb6 - Remove now as required var for workflow mutation (#4122)
2023-03-29 - 3a5665264 - Enforce concurrent count limit to all long poll calls (#4120)
2023-03-29 - d4e325114 - Fix data race in test (#4121)
2023-03-29 - 7ce28f8ff - Log warnings for retryable ringpop errors (#4102)
2023-03-29 - 88585475f - Inject service name (#4119)
2023-03-29 - 8bd067a9f - Add poison pill support for replication stream (#4116)
2023-03-29 - 7228219f5 - Don't include stack traces for warnings during tests (#4101)
2023-03-29 - 8f030dbb8 - Wire replication stream E2E (#4097)
2023-03-29 - fca366247 - Add history size bytes to SQL visibility (#4090)
2023-03-29 - f4c188813 - Addressing PR-4099 comments (#4113)
2023-03-29 - f1b88e8d5 - Bump UI version (#4047)
2023-03-28 - bccdaaac4 - Add base workflow info to workflow replication task (#4107)
2023-03-28 - e4a49b264 - Add flag to enable execution scanner event Id validator (#4114)
2023-03-28 - 47424c0ba - Better error message in ES query converter (#4112)
2023-03-28 - 939eefae8 - Fix missing import bug (#4111)
2023-03-28 - a20faeb88 - Use dedicated proto definition for replication stream (#4106)
2023-03-28 - afc5ac7e4 - Remove common.Daemon from Monitor interface (#4084)
2023-03-28 - 443144646 - Add replication stream receiver logic (#4093)
2023-03-26 - aef7c0b93 - Fix some minor linter warnings in the membership package (#4100)
2023-03-24 - f1ef4db67 - Change update namespace to upsert custom search attributes (#4080)
2023-03-24 - 8bdfccc51 - Refresh per-namespace worker periodically (#4103)
2023-03-24 - 3d6ac72f5 - Fail QueryWorkflow if last workflow task failed (#4099)
2023-03-24 - d05b15fce - Support empty queries when listing archived workflows (#4027)
2023-03-24 - 63b22ca41 - Add support for workflow start delay (#3984)
2023-03-23 - 11cef80fe - Flush buffer event when failing workflow task (#4010)
2023-03-23 - 19a30b926 - Fix build id compatibility spelling (#4086)
2023-03-23 - cd157354e - Hide ringpop config validation in ringpop package (#4085)
2023-03-23 - aee7c5f9a - Stub out new PollWorkflowExecutionUpdate API (#4081)
2023-03-23 - 5210ad4c2 - Add gRPC replication stream (#4057)
2023-03-23 - e67a1ce17 - Account for local activity run time in schedule sleep (#4079)
2023-03-22 - 832f4e6da - Break replication ack manager into smaller component (#4088)
2023-03-22 - cc2eb80b2 - Fix test failure from merge conflict (#4087)
2023-03-22 - 22522bf95 - Use more reliable workflow mutation check (#4076)
2023-03-22 - 6974ed355 - Fix namespace handover replication queue notification (#4082)
2023-03-22 - 66352edef - Track and emit metrics for entities in a single workflow (#4065)
2023-03-22 - 1439b87bd - Handle max signal count limit for signal transfer task (#4051)
2023-03-22 - b3bc60203 - Remove Lookup method from Monitor interface (#4072)
2023-03-21 - 19b465be2 - Remove unused methods from Monitor interface (#4071)
2023-03-21 - e8be006e2 - Lock current execution on start workflow (#4066)
2023-03-21 - d53f1b6f4 - Use preemptable caller type for low priority history tasks (#3993)
2023-03-21 - 78fdba2c5 - Add deadlines to schedule local activities (#4078)
2023-03-21 - 2af5bf4b2 - Fix SQL visibility page token deserializer (#4074)
2023-03-21 - fc56eeac0 - Rename visibility writing mode consts (#4068)
2023-03-21 - dc018cd7d - Fixing saving custom search attribute after upgrading SQL DB (#4062)
2023-03-21 - e7b42766f - Add counter to track custom order by usage in Elasticsearch (#4064)
2023-03-21 - 4033f2105 - Rename variables in timer package (#4067)
2023-03-21 - 411c65bf2 - Set default values in returned calendar specs (#3969)
2023-03-20 - 0d0cad4a1 - Change disable order by dynamic config to per namespace (#4063)
2023-03-16 - 129f2f841 - Pass in namespace ID to initialize new history branch (#4058)
2023-03-15 - 07ffab53e - Use new API's simpler representation of versioning data (#3432)
2023-03-14 - 086f5ecfb - Rename inflight WT to started WT (#4043)
2023-03-14 - ddd0911c4 - Add gRPC stream error interceptor (#4019)
2023-03-13 - ed47a167a - Fix UnhandledCommand error for workflow task completion in CancelTimer (#4050)
2023-03-13 - b520a5b11 - Improve message sequence validator (#4037)
2023-03-10 - 122bb36ed - Add more test for XDC (#4025)
2023-03-09 - 7b6ce2545 - Unique owner for each shard context object (#4008)
2023-03-09 - 4c47aecb4 - Convert query converter error to invalid argument (#4036)
2023-03-09 - 7f848d1d6 - Move update registry from mutable state to workflow context (#4032)
2023-03-09 - 85aa550f9 - Extract cluster config logic into ConfigureCassandraCluster() (#4034)
2023-03-09 - 75facb16b - Remove unused HostsUpdated field from ChangedEvent (#4015)
2023-03-09 - 818cbb173 - Refactor gocql.NewSession to allow passing in cluster config factory method (#4033)
2023-03-08 - 010917c76 - Flush buffer event with single cluster namespace (#4031)
2023-03-08 - d1159dc01 - Add support for namespace readthrough to persistence (#3908)
2023-03-06 - 6ea38d79f - Update history task reader progress (#4004)
2023-03-06 - e7cdd7140 - Register history queue readers (#3995)
2023-03-06 - dc692785b - Restructure replication task implementation & UT (#3992)
2023-03-06 - e4162e80d - Remove unused persistence GetHistoryTask API (#4003)
2023-03-06 - 94f58249b - Add more test cases for XDC (#4009)
2023-03-06 - c8ad74a61 - Return raw replication task info from read dlq api (#4007)
2023-03-05 - c172708e3 - Hide remaining exported symbols in ringpop package (#4012)
2023-03-05 - 9740e7f31 - Move RingPop config validation to membership package (#4011)
2023-03-05 - 8c7e79a30 - Fix comment above NewServerFxImpl (#4013)
2023-03-05 - 061593b0f - Move ringpop code from membership to ringpop package (#3999)
2023-03-05 - abd02977d - Extract interface from membership.HostInfo (#3996)
2023-03-02 - 701b77528 - Specify readerID on loading history tasks (#3994)
2023-03-02 - 715fe3fe5 - History task reader persistence hint API (#3986)
2023-03-02 - 2977c7243 - Upgrade sqlite to v1.21.0 to address security vulnerability (#4001)
2023-03-01 - 7086b94a4 - Make types not used outside of membership unexported (#3998)
2023-03-01 - 13e317075 - Use snake case for file names in common/membership (#3997)
2023-02-28 - eea242d50 - Refactor history branch manipulation logic into its own utility (#3946)
2023-02-27 - 35dcba861 - Improve scheduled queue lookahead logic (#3982)
2023-02-24 - edd92bbf6 - Streamline localstack s3 development (#3976)
2023-02-22 - 3b982585b - Add SQL query converter tests (#3972)
2023-02-21 - bc2d02555 - Refactor SQL query converter (#3971)
2023-02-21 - f522ff3eb - Upgrade dependency for next release (#3974)
2023-02-21 - 3b7139358 - Fix passing pointer of slice element in SQL query converter (#3970)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.21.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes few schedule and search attributes issues as well as minor bugs in various parts of the system.

All Changes

2023-05-12 - 3ce630832 - Fix cron for schedules that do not match any time (#4206)
2023-05-12 - d8a6ca7e6 - Run no-op claim mapper even without auth info (#4197)
2023-05-12 - 9448a6f3a - Fix metric tag for delete execution task (#4275)
2023-05-12 - 5225f13f7 - Fix task attempt reset (#4276)
2023-05-12 - f163b880c - Make shard ownership more resilient when shard is busy (#4283)
2023-05-12 - d6f7c9172 - Event buffer size limit (#4296)
2023-05-12 - f64bb0eb0 - Use search attribute type map in visibility archival (#4304)
2023-05-12 - 4ef165a03 - Skip over entire time range if paused and batch and cache time queries (#4215)
2023-05-12 - caac9c20c - Fix adding multiple search attributes of same type at once (#4273)
2023-05-12 - c54890d91 - Change schedules default catchup window to 1 year (#4332)
2023-05-15 - a97cdef0e - Truncate activity failure in mutable state (#4338)
2023-05-15 - 6eba60dc3 - Fix caller info for migration activities (#4340)
2023-05-15 - fcebf6956 - Tag metric tracking custom order by with namespace (#4344)
2023-05-15 - a4c0b81c4 - Bump server version for 1.20.3 patch release

Details about the v1.20.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.3)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes few schedule and search attributes issues as well as minor bugs in various parts of the system.

All Changes

2023-04-17 - 9729d0368 - Reuse timer in schedule workflow (#4129)
2023-04-17 - 8f178a3c7 - Use correct time for initial schedule memo (#4128)
2023-04-17 - 16d53cb9b - Reconnect on a few more gocql errors (#4132)
2023-04-17 - 50b7daf66 - Fix error handling add search attributes (#4148)
2023-04-17 - d017c66ef - Validation API should use low priority lock (#4140)
2023-04-17 - eec0dbe2e - Fix schedule catchup window metric calculation while paused (#4152)
2023-04-17 - f638c97fb - Update Go SDK to v1.21.2
2023-04-17 - 99c10ccf1 - Bump server version for 1.20.2 patch release

Details about the v1.20.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes few SQL-based advanced visibility bugs as well as minor bugs in various parts of the system.

All Changes

2023-03-24 - f38e041e7 - Fix passing pointer of slice element in SQL query converter (#3970)
2023-03-24 - 6d28fe257 - Refactor SQL query converter (#3971)
2023-03-24 - f714a91dc - Add SQL query converter tests (#3972)
2023-03-24 - 5c06a2bdb - Flush buffer event with single cluster namespace (#4031)
2023-03-24 - 1855bb4ef - Refactor gocql.NewSession to allow passing in cluster config factory method (#4033)
2023-03-24 - d8aaa5f25 - Extract cluster config logic into ConfigureCassandraCluster() (#4034)
2023-03-24 - f17467100 - Convert query converter error to invalid argument (#4036)
2023-03-24 - 07cf95172 - Fix UnhandledCommand error for workflow task completion in CancelTimer (#4050)
2023-03-24 - 305b770c7 - Change disable order by dynamic config to per namespace (#4063)
2023-03-24 - f8c06b537 - Set default values in returned calendar specs (#3969)
2023-03-24 - 8c1a0fb12 - Add counter to track custom order by usage in Elasticsearch (#4064)
2023-03-24 - 89f2095bf - Fixing saving custom search attribute after upgrading SQL DB (#4062)
2023-03-24 - b0ef992c0 - Fix SQL visibility page token deserializer (#4074)
2023-03-24 - 5a83f71b2 - Add deadlines to schedule local activities (#4078)
2023-03-24 - cb7d42c85 - Use preemptable caller type for low priority history tasks (#3993)
2023-03-24 - 633a05657 - Lock current execution on start workflow (#4066)
2023-03-24 - 145c99b76 - Handle max signal count limit for signal transfer task (#4051)
2023-03-24 - a8b591d15 - Track and emit metrics for entities in a single workflow (#4065)
2023-03-24 - df2d8eb67 - Fix namespace handover replication queue notification (#4082)
2023-03-24 - de840920c - Use more reliable workflow mutation check (#4076)
2023-03-24 - c0484f9dc - Fix test failure from merge conflict (#4087)
2023-03-24 - dcaa3a378 - Account for local activity run time in schedule sleep (#4079)
2023-03-24 - 57347cb91 - Flush buffer event when failing workflow task (#4010)
2023-03-24 - cbb9db75f - Support empty queries when listing archived workflows (#4027)
2023-03-24 - aabbae0d7 - Fail QueryWorkflow if last workflow task failed (#4099)
2023-03-24 - 6b747d3b0 - Refresh per-namespace worker periodically (#4103)
2023-03-24 - 7c0a0e036 - Change update namespace to upsert custom search attributes (#4080)
2023-03-28 - 212332b6a - Add flag to enable execution scanner event Id validator (#4114)
2023-03-29 - cf61e9eed - Enforce concurrent count limit to all long poll calls (#4120)
2023-03-28 - 292f4600d - Bump server version for 1.20.1 patch release

Details about the v1.20.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

Upgrade action

• SERVICE_FLAGS environment variable for docker images is removed, use SERVICES instead.
• The dynamic config setting matching.useOldRouting was removed and old routing is not available anymore. If you’re upgrading and did not set it earlier, set it to false across your cluster before the upgrade to minimize task dispatch disruption during deployment.

Deprecation heads-up

The following softwares will not be supported in future versions:

• MySQL 5.7: planned to be removed by v1.21; please upgrade to MySQL 8.0.17+.
• PostgreSQL 9.6 to 11: planned to be removed by v1.21; please upgrade to PostgreSQL 12+.

Advanced visibility

We’re launching advanced visibility features for SQL databases (see more below). However, this required a significant change in the existing functionality when used with Elasticsearch. Up until v1.19, when you create a custom search attribute of a given type, say Keyword, you could actually store a list of Keyword values. This will no longer be supported. If you have a search attribute of type Keyword, you will only be able to store a single Keyword value. If you wish to store a list of values, you can create a search attribute of type KeywordList. This is the only type that supports list of values, ie., there’s no equivalent for any other type (Int, Datetime, etc.).

$ temporal operator search-attribute create --name CustomField --type KeywordList

If you are using Elasticsearch, you can change the type of a Keyword search attribute to KeywordList type by re-creating the search attribute (warning: this will only work as expected if the value stored is already a list; otherwise, it will lead to unexpected errors):

$ temporal operator search-attribute remove --name CustomField
$ temporal operator search-attribute create --name CustomField --type KeywordList

For backwards compatibility, if you’re using Elasticsearch, list of values is still supported for all search attribute types. However, we discourage such usage. Furthermore, future releases of SDK will deprecate API accepting list of values, and change it to be strong typed so it won't accept list of values.

Advanced visibility on SQL DB (beta testing)

Advanced visibility features are now supported using one of the supported SQL databases (MySQL 8.0.17+, PostgreSQL 12+, and SQLite):

• Per namespace, you can create up to 3 custom search attributes for each of the types Bool, Datetime, Double, Int, Text and KeywordList, and up to 10 custom search attributes of type Keyword.
• You’ll be able to call the List API using SQL-like queries (ORDER BY and LIKE operators are not supported, check the documentation for more details).

Here are the steps to enable this feature [warning: for all commands below, please be aware of replacing the database parameters (user, password, host, port) accordingly to your use case]:

1. Upgrade Temporal Server to v1.20. If you don't want to enable advanced visibility features, then stop here. Everything should work as usual.

2. Upgrade your database to MySQL 8.0.17+ or PostgreSQL 12+ if you’re using an older version. Please check with official documentation on how to upgrade MySQL and PostgreSQL. We also recommend making a backup of your data before taking any actions.

   • [Optional step for MySQL users] If you upgraded from MySQL 5.7, convert the database character set to utf8mb4(learn more about character sets in the MySQL documentation). The following commands are an example of how you can convert both the temporal and temporal_visibility databases (check the MySQL documentation for more details on changing the character set for databases and tables):

     $ DB=temporal; ( echo 'SET foreign_key_checks=0; ALTER DATABASE `'$DB'` CHARACTER SET utf8mb4;'; mysql $DB -u root -proot -e "SHOW TABLES" --batch --skip-column-names | xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8mb4;' ) | mysql $DB -u root -proot
     $ DB=temporal_visibility; ( echo 'SET foreign_key_checks=0; ALTER DATABASE `'$DB'` CHARACTER SET utf8mb4;'; mysql $DB -u root -proot -e "SHOW TABLES" --batch --skip-column-names | xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8mb4;' ) | mysql $DB -u root -proot
     

3. Update the schema (if you need to set TLS cert, check our documentation for more details on the temporal-sql-tool command):

   • MySQL 8.0.17+:

     $ ./temporal-sql-tool --ep localhost -p 3306 -u root -pw root --pl mysql8 --db temporal_visibility update-schema -d ./schema/mysql/v8/visibility/versioned
     

   • PostgreSQL 12+:

     $ ./temporal-sql-tool --ep localhost -p 5432 -u temporal -pw temporal --pl postgres12 --db temporal_visibility update-schema -d ./schema/postgresql/v12/visibility/versioned
     

4. Change the sql.pluginName config to mysql8 or postgres12 accordingly.

5. Restart Temporal Server.

If you have previously created custom search attributes while using standard visibility, you’ll need to create them again. Do not delete the existing ones, just execute the command to create a new custom search attribute. For example, if you had created a custom search attribute named CustomField of type Keyword, then execute the following command (make sure you update to the latest version of Temporal CLI tool):

$ temporal operator search-attribute create --namespace default --name CustomField --type Keyword

For SQLite users, we do not support schema updates. Therefore, if you're using SQLite file-based storage, you'll need to reset your database from scratch.

Archival feature

We reimplemented the Archival feature using a separate task queue, providing higher efficiency, better resource isolation, and strong durability guarantees. Most importantly, Archival will now infinitely retry with an exponential backoff function before deleting any data. Before this change, Archival would only retry for a maximum of five minutes.

Workflow Update (alpha)

Workflow Updates enable gRPC clients to issue requests to workflow executions that mutate internal workflow state and return a response.

• This is an alpha quality release that must not be used for production workloads. Only workers using the Go SDK are supported. Workflow histories that include update events from this release may not be replayable by future releases.
• Workflow Updates are disabled by default. To enable UpdateWorkflowExecution API change frontend.enableUpdateWorkflowExecution dynamic config value to true.
• There is a new example in the samples-go repository demonstrating both how to invoke and how to handle updates.

Replication between clusters with different shard counts

Temporal Server 1.20 added capability to connect and replicate between 2 clusters with different shard count. The requirement is one cluster’s shard count is another one’s multiple (Y=NX, Y and X are shard count, and N is a natural number). Shard count is a scale unit, cluster could run into scale limit if the choose shard count is too small. This new feature make it possible to move live workload to a new cluster with larger shard count. It works the other way as well if you want to reduce your shard count.

Note: Temporal server 1.19 and earlier versions require 2 clusters to have exact same shard count if they want to be connected.

BatchOperation to support DeleteWorkflow

Add delete workflow as another supported action to BatchOperation API.

Otel 0.34

Open telemetry upgraded to 0.34. 0.34 Otel release contains a breaking change. All counters will be append a ‘_total’ suffix.

Default authorizer

If you're using the default authorizer (note that the default authorizer is not used by default, you have to explicitly configure it), you should be aware of a few changes:

• The special case for temporal-system namespace was removed. This namespace is used by various workflows that run on the system worker service.

  Separately, two recent features, Schedules and Batch v2, require the system worker to connect to namespaces other than temporal-system. If your claim mapper was not properly authenticating the system worker service and giving it a System: RoleAdmin claim, then this special case meant that system workflows would work but Schedules and Batch v2 would not.

  Now that this case is removed, your system workers might have trouble connecting to the frontend unless you make some other changes. Either 1) your claim mapper should properly authenticate the system worker and give it a System: RoleAdmin claim, or 2) use Internal Frontend to bypass the claim mapper for internode connections to frontend. See the Internal Frontend section below.

• The special case for requests with no namespace was removed, and replaced with a check for two specific requests: gRPC health checks, and GetSystemInfo. All other requests with no namespace in the request, e.g. ListNamespaces now require corresponding claims to be allowed. If you were relying on that special case, you may have to adjust your claim mapper to add a System: RoleReader claim for authorized clients.

  Because frontend.enableTokenNamespaceEnforcement defaulted to false, the no-namespace special case could also have been a security hole, depending on your claim mapper: if a client sent a request with a namespace in the token but an empty namespace field at the top level of the request, it would pass the authorizer without a proper check for a namespace claim. That dynamic config now defaults to true.

Internal frontend

There's a new deployment option that can simplify worker→frontend communication, mTLS, claim mappers, and authorizers: the internal-frontend. You should consider using internal-frontend if:

• You're using a claim mapper and authorizer (either default or custom) and there's no easy way to differentiate between the system worker service and external clients.
• You were relying on the special case for temporal-system (see above) in the default authorizer or your custom authorizer.
• You're using mTLS and you want to use a different CA for internode connections and external client connections to frontend.

Basically, internal-frontend makes the system worker service and frontend work like the other "internode" services (history and matching) so you can secure them in the same way.

To use it:

• Create an internal-frontend section in your static config under services by copying frontend and adjusting the ports (we suggest 7236/6936 but any works).
• Make sure the publicClient section is removed from static config.
• Add a new deployment that uses -service internal-frontend on its command line. If using mTLS, provide it with the same "internode" certs and keys as history and matching.

If you're using pre-built docker images, set environment variable USE_INTERNAL_FRONTEND to 1 instead of the static config changes above.

Note that metrics will appear with service_name=internal-frontend so you might have to make adjustments to your obsrevability as well. Logs will appear tagged with {"service":"frontend","internal-frontend":true}.

You can use dynamic configs internal-frontend.globalNamespaceRPS and internal-frontend.globalNamespaceRPS.visibility to adjust the incoming rate limits for internal-frontend separately from [external] frontend.

Security-related changes

• Multiple changes to the default authorizer (see Default authorizer section above) https://github.com/temporalio/temporal/pull/3706
• frontend.enableTokenNamespaceEnforcement dynamic configuration setting now defaults to true https://github.com/temporalio/temporal/pull/3798
  • The previous setting allowed cross-namespace access when presenting a task token where the namespace ID and the namespace fields have different values.
  • Task tokens are generally handled entirely by clients, but can be created by end-users
  • Thanks to Jacob LeGrone at DataDog for finding and reporting this insecure default.
  • We strongly urge all users to upgrade to 1.20 or, if unable to upgrade, enable this configuration option

Scheduled Workflows GA

There are a few new dynamic config options for tuning the system worker service and schedules:

• worker.perNamespaceWorkerOptions: map option to set options for workers for schedules and batch v2. Allows namespace constraints. Set values like this:

  worker.perNamespaceWorkerOptions:
  - value:
      MaxConcurrentWorkflowTaskPollers: 10
  

• worker.stickyCacheSize: integer option to control sticky cache size for workers. Global to the process. Defaults to the SDK default (10,000).

• worker.schedulerNamespaceStartWorkflowRPS: float value to set a maximum
  number of workflows started by schedules at the namespace level. Allows
  namespace constraints. Defaults to 30.

You may need to adjust these if you're making heavy use of the schedules feature.

There are a few new metrics you can monitor also: schedule_missed_catchup_window, schedule_rate_limited, schedule_buffer_overruns, schedule_action_success, schedule_action_errors, schedule_cancel_workflow_errors schedule_terminate_workflow_errors.

With these changes and the authorizer/internal-frontend changes to make it easy for the system worker service to have the necessary access in all configurations, we’re calling the scheduled workflows feature generally available.

All changes

2022-11-30 - 681ce454f - Fix service_latency_nouserlatency (#3677)
2022-12-01 - 79a572de6 - Remove empty visibility tag (#3681)
2022-12-01 - 21ed2aa5a - Update next release version (#3683)
2022-12-05 - 4907ff999 - Use empty namespace metrics tag if namespace not found (#3686)
2022-12-05 - c94d2bfa5 - Refactor history cache to its own package (#3601)
2022-12-05 - 34e256a95 - Test that ShardOwnershipLostErrors are never retried (#3625)
2022-12-07 - d112ea818 - Update dashboards (#3694)
2022-12-07 - bece27de3 - Add metrics scopes for the archival queue (#3566)
2022-12-08 - 72aa9bd03 - Add an integration test for acquireShard (#3678)
2022-12-08 - eb88af41e - Add random delay to ArchiveExecutionTask (#3565)
2022-12-08 - 8a71b9455 - Add a utility for reliably fetching search attributes and memo (#3687)
2022-12-08 - f0c2e5871 - Disable the Test_DeleteExecutionsWorkflow_ManyExecutions_NoContinueAsNew test because it is flaky (#3699)
2022-12-08 - 1af8a94da - Fix flaky CreateIndex method (#3698)
2022-12-08 - b1943f740 - docs: add specific make step to README on the tools/sql path (#3697)
2022-12-08 - b47046f5b - Move ServiceName to primitives and use everywhere (#3695)
2022-12-09 - e12743079 - Update Go SDK to v1.19.0 (#3701)
2022-12-09 - 55cdd690d - Adds DPanic and Panic level logging (writes log, then panics) (#3689)
2022-12-09 - ad1176c51 - Adds DPanic and Panic level logging (writes log, then panics) (#3689)
2022-12-09 - 4c4ef467e - Adds DPanic and Panic level logging (writes log, then panics) (#3689)
2022-12-09 - ae223cf1e - Filter replication task on source cluster (#3641)
2022-12-09 - 912dc51b9 - Fix flaky shard controller tests (#3693)
2022-12-12 - 0f27ed22a - Add sdk-features trigger to PRs (#3627)
2022-12-12 - 8ca4b2cc8 - Handle data loss error in replication read history (#3704)
2022-12-13 - 2abbea974 - Separate archival integration tests into their own suite (#3707)
2022-12-13 - 033154933 - Add an archival task executor (#3663)
2022-12-13 - 532c422c2 - Add an archival queue factory (#3562)
2022-12-13 - 7a147f6e3 - Use namespace from schedule activity worker (#3680)
2022-12-15 - ea698fa1b - Remove old queue processing logic (#3716)
2022-12-15 - 0b4df6a67 - Rename metrics.Handler to avoid name stuttering (#3713)
2022-12-16 - 728f6ce68 - Improve archival_test (#3719)
2022-12-16 - 5a4e0a97f - Fix ArchiveExecutionTask key type (#3720)
2022-12-16 - 70b59693d - Remove unused cluster parameter from task notification (#3718)
2022-12-16 - edd3f0a4b - Fix bug not setting CanSkipVisibilityArchival (#3721)
2022-12-19 - 23a2c4e67 - Fix errcheck warnings in cmd/tools (#3729)
2022-12-20 - a7240b4de - Fix ES bulk processor commit timeout (#3696)
2022-12-20 - 6fbc4f4af - Rename mutableStateImpl receiver (#3737)
2022-12-20 - 0ad773b8c - Remove scheduled and started events from transient workflow task info (#3736)
2022-12-20 - b57241955 - Fix bug where we return nil when there's an error in the archival queue (#3723)
2022-12-20 - e57edab44 - Fix range variable in archiver_test not being captured (#3724)
2022-12-20 - fb3f2f969 - Fix errcheck in service/frontend (#3733)
2022-12-20 - 12e416293 - Document that errors in goro.Group.Go functions are ignored (#3734)
2022-12-20 - c6a89c095 - Workflow task state machine: minor renames (#3735)
2022-12-20 - 2b761b495 - Remove internal bulk processor retries (#3739)
2022-12-21 - 661880f37 - Refactor transient workflow task state machine (#3740)
2022-12-21 - b2c5357ec - Verify cluster name in remove cluster call (#3715)
2022-12-21 - 11b808ad6 - Fix errcheck in ./host/ (#3741)
2022-12-21 - d8961577b - Fix errcheck in ./common/persistence/ (#3743)
2022-12-21 - 1454f84c0 - Fix errcheck in service/worker/ (#3731)
2022-12-21 - 957cfa4f2 - Fix errcheck in ./common/archiver/ (#3744)
2022-12-21 - 676b74459 - Fix errcheck in ./common/membership/ (#3742)
2022-12-21 - 3cb92321a - Clean namespace handover (#3692)
2022-12-21 - 2171a17e3 - Fix errcheck in ./common/rpc/ (#3752)
2022-12-22 - 3a322bb07 - Use latest replication task time as sync shard time (#3722)
2022-12-22 - 86e9d324b - Consolidate linters with golangci-lint (#3758)
2022-12-22 - ba2d711e7 - Remove unused Temporal interface (#3761)
2022-12-22 - 825d39c6a - Fix errcheck in service/matching (#3756)
2022-12-22 - 329290776 - Fix errcheck in temporal/ (#3757)
2022-12-22 - f09ea5cbf - Fix errcheck in service/history (#3754)
2022-12-23 - ff5805726 - Handle unhandled errors in tests/ directory (#3730)
2022-12-23 - 930a5e8a8 - Handle unhandled errors in tools/ (#3748)
2022-12-23 - 5278b2e97 - Fix some unhandled deferred errors in ./common/ (#3750)
2022-12-23 - 7d6608749 - Fix gofmt and add Go report card (#3751)
2022-12-28 - 4e8601a05 - Create an FX Module for Archival (#3726)
2022-12-28 - 5640a989a - Fix some lostcancel errors (#3764)
2022-12-28 - c9124547f - Fix errcheck in service/history/shard (#3755)
2022-12-28 - 92b875889 - Add an integration test for durable archival (#3732)
2022-12-28 - e57f97f9f - Fix token count in archival rate limiter (#3749)
2022-12-28 - d970aa493 - Fine-tune the Revive linter (#3763)
2022-12-29 - 98545f9b3 - Remove unused parameters from task executable (#3747)
2022-12-29 - adf7c5471 - Fix NPE in task channelWeightFn (#3766)
2023-01-03 - a22db81db - Remove deprecated code from queue implementation (#3770)
2023-01-03 - 32b5d9133 - Make persistence tests consistent in handling ShardID (#3772)
2023-01-04 - 7215a3253 - Validate shard id in shard controller (#3776)
2023-01-04 - a026002b5 - Remove task max retry count config (#3771)
2023-01-04 - ac8717cc8 - Separate integration tests and unit tests (#3760)
2023-01-06 - 9c2528620 - Trigger golangci-lint on pull requests (#3781)
2023-01-06 - e91760f88 - Add speculative workflow task (#3768)
2023-01-06 - 3882bb635 - Capture task processing panic (#3779)
2023-01-06 - 04ce8e382 - Deprecate old namespace change callback (#3774)
2023-01-09 - aa7671974 - Expose replicateWorkflowState API in history (#3783)
2023-01-09 - 3e31880b0 - Add a component to assign polling shards (#3775)
2023-01-09 - ea893d18a - Disable eager activities for Python 0.1a1, 0.1a2, and0.1b1 too (#3793)
2023-01-10 - bb4ec7f01 - Prioritize worker pool shutdown (#3795)
2023-01-10 - c5254f396 - Address a few linter errors (#3780)
2023-01-10 - 1a9695e10 - Add internal-frontend role (#3706)
2023-01-11 - 851c634c6 - Rename host tests to functional tests (#3782)
2023-01-11 - e5883b930 - Minor test target fixes (#3801)
2023-01-11 - bd13fc6b3 - TW 219 - update links across repos (#3785)
2023-01-11 - ef982348f - Turn on frontend.enableTokenNamespaceEnforcement by default (#3798)
2023-01-11 - 6319a2848 - Add NewInternalErrorWithDPanic for common dpanic pattern (#3797)
2023-01-11 - 5b182a72d - Capture panic in replication task processing (#3799)
2023-01-11 - d527a1992 - Rename sdk-features to features (#3796)
2023-01-11 - daff5f1bf - Add separate dynamic config knobs for internal-frontend rate limiting (#3800)
2023-01-11 - dc932a16d - Fix recover call (#3804)
2023-01-11 - bd068266e - Read cluster shard count from DB (#3788)
2023-01-12 - 26503cfbb - Drop task on serialization error (#3803)
2023-01-12 - b132fc2d4 - Redirect replication requests when shard count is different (#3789)
2023-01-13 - 0bf1137f0 - Add test definition in readme (#3807)
2023-01-13 - 51fd1dd7c - Allow start many replication pollers in one shard (#3790)
2023-01-13 - 394885498 - Allow connect clusters with different shards (#3777)
2023-01-17 - 0a90b770e - Fix datarace in forwarder_test (#3767)
2023-01-17 - efb3bc100 - Fix redirect replicate workflow state (#3809)
2023-01-17 - c24c83f85 - Use generics in jitter (#3717)
2023-01-18 - a54483d91 - Fix flaky jitter test (#3814)
2023-01-19 - 4576a1046 - Add debug.TimeoutMultiplier to all timeouts (#3815)
2023-01-19 - 5413d1866 - Add mock archival queue processors to history engine tests (#3817)
2023-01-19 - eb4b574bc - Fix some bugs in the archival queue task executor (#3813)
2023-01-19 - e0081fde1 - Fix a rare deadlock in scanner.Stop (#3818)
2023-01-20 - 82c958d33 - Partial revert #3731, fix errcheck, add comments (#3787)
2023-01-20 - 5175ab603 - Single scheduled queue max read level (#3778)
2023-01-24 - 9389673ba - Support batch operation with list of workflow executions (#3812)
2023-01-24 - 3abd50d31 - Better SQL query splitter (#3791)
2023-01-24 - b9bba94dd - Fix GetWorkflowExecution in PostgreSQL (#3816)
2023-01-24 - 554b32d33 - Produce archival tasks conditionally (#3823)
2023-01-24 - 285bf333f - Start the archival queue iff history or visibility archival is enabled in the static config (#3827)
2023-01-24 - a06da897c - Remove matching.useOldRouting dynamic config (#3824)
2023-01-24 - 278f545c2 - Add dynamic config for sdk worker options (#3806)
2023-01-24 - 4c7189903 - Update Go SDK to 1.20.0 (#3810)
2023-01-25 - 7e54876e1 - Rename UpdateWorkflow to UpdateWorkflowExecution (#3832)
2023-01-25 - 1c638ac56 - Enable durable archival (#3828)
2023-01-25 - e6113da91 - Fix SQL NPE after connection is closed (#3829)
2023-01-25 - 430d3a90a - Messages protocol implementation (#3843)
2023-01-25 - 1543bd4ad - Update UI to v2.10.0 (#3841)
2023-01-25 - 5652dc4f1 - Fix time fields of rows from GetClusterMembers (#3840)
2023-01-25 - 4f53af912 - Tune linters (#3848)
2023-01-25 - 5338f96fb - Add env vars to control log format and level for tests (#3836)
2023-01-25 - b4947362d - Add RefreshPerNSWorkerManager and RemoveOverride for tests (#3837)
2023-01-25 - 1c84aadc2 - Fix continue as new validation (#3845)
2023-01-25 - 5c23836c0 - Fix handover callback (#3847)
2023-01-25 - 7e89af482 - Add explicit per-namespace rate limit to schedules (#3838)
2023-01-25 - 909ca9688 - Add initial memo to scheduler workflows (#3839)
2023-01-26 - c52848e8d - Revert #3755 (#3842)
2023-01-26 - 9d3fedeec - Add index name to constructor of search attribute validator (#3834)
2023-01-26 - 25015d470 - Emit lag metrics for all registered history queues (#3851)
2023-01-26 - de6f2dfde - Remove internal-frontend from default services (#3849)
2023-01-26 - 391ed6109 - Factor out task queue name mangling into new package (#3549)
2023-01-26 - d1c5d108f - Change deadline too short to failedPrecondition error (#3856)
2023-01-26 - fc6efef1f - Remove failover levels (#3854)
2023-01-27 - dc257ce87 - Add GetIndexName method to visibility manager (#3820)
2023-01-27 - 3b3498a38 - Move namespace handler to service/frontend (#3855)
2023-01-27 - 0e4506be2 - Remove internal-frontend from development configs (#3859)
2023-01-27 - 6e732e79d - Return last error instead of ctx error (#3857)
2023-01-30 - 9a4f19e15 - Mark state transition & completion metrics with namespace activeness (#3831)
2023-01-30 - 3501f6700 - Conditionally register archival category (#3867)
2023-01-30 - b40a4ac97 - Fix activity eager execution on workflow close (#3868)
2023-01-30 - 473b449d4 - update development version of temporalio/ui to 2.10.1 (#3869)
2023-01-30 - 7e83ca2fc - Add custom search attribute aliases map to namespace config (#3866)
2023-01-30 - 416abfbda - Update type of keyword list search attributes (#3865)
2023-01-30 - fa429a5ab - [Advanced Visibility with SQL] Adding MySQL 8 schema and interface (#3552)
2023-01-31 - 7e3daf1fd - Add lower priority for replication tasks (#3870)
2023-01-31 - 962985432 - Update search attributes operator commands to handle aliasing (#3871)
2023-01-31 - f3cf50429 - [Advanced Visibility with SQL] Changing visibility schema for SQLite (#3819)
2023-01-31 - cb22aab7c - [Advanced Visibility with SQL] Adding PostgreSQL 12 schema and interface (#3844)
2023-01-31 - 750fc54c3 - Implement searchattribute.MapperProvider (#3873)
2023-01-31 - 29afc2f1e - Handle updating cluster metadata search attributes info when using SQL DB (#3861)
2023-01-31 - c257d84c1 - Implement default mapper for SQL visibility (#3875)
2023-02-01 - de0b04906 - Fix visibility processor panic on add after stop (#3830)
2023-02-01 - 2bc48d21f - Upgrade otel to 0.34 (#3850)
2023-02-01 - af0071934 - Merge custom search attributes from index name with empty string key (#3877)
2023-02-01 - d9bac92f2 - Return error if cluster metadata is invalid (#3879)
2023-02-01 - cbd73e01e - Synchronous workflow update (#3822)
2023-02-01 - dbeab313e - clean replication task processor after disconnect (#3882)
2023-02-01 - 6ef774953 - Eager workflow dispatch (#3835)
2023-02-01 - 8b6b2fbab - Keep queue factory group tag (#3880)
2023-02-02 - b5f64f2f6 - Protect UpdateWorkflowExecution API with dynamic config feature flag (#3884)
2023-02-02 - 5721cf897 - Add MySQL 8 admin interface (#3886)
2023-02-02 - 19d36fb49 - Fix docker config template to set only one visibility (#3889)
2023-02-02 - cf9e84adf - Overwrite operation for admin and operator APIs (#3890)
2023-02-02 - e43e5b500 - Advanced visibility for MySQL (#3878)
2023-02-02 - f94e57545 - Update admin handler to handle search attributes operations with SQL DB (#3894)
2023-02-02 - 52d1ba08f - Advanced visibility for PostgreSQL (#3896)
2023-02-02 - 32a468689 - Pass workflowTask object to completion methods of mutable state (#3897)
2023-02-03 - bd2d3b3cb - Advanced visibility for SQLite (#3895)
2023-02-03 - 82f911ea5 - Disable unused-parameter linter (#3891)
2023-02-03 - 75cf92b4d - Ignore unhandled errors from fmt.Printf (#3892)
2023-02-03 - c7d7a896e - Listen for loopback instead of all incoming traffic in RP test (#3893)
2023-02-03 - a4943bbd2 - Add metrics + max buffer size to schedule workflow (#3872)
2023-02-03 - 30264ca61 - Update SDK dependency to v1.21.0 (#3902)
2023-02-03 - d43e112d9 - Fix fail of speculative workflow task (#3898)
2023-02-03 - e95e082e7 - Fix namespace interceptor to allow search attributes operations (#3899)
2023-02-03 - 20a5ce339 - Remove search attributes with nil values (#3903)
2023-02-03 - 3ade2631d - Allow setting cluster Id explicitly (#3883)
2023-02-03 - 008f056d6 - Improve task scheduler rate limiter (#3860)
2023-02-03 - ca5be2995 - Reimplement DC redirection handler (#3887)
2023-02-03 - a070fb511 - Fix clientFactory build issue (#3904)
2023-02-04 - 157cd1142 - Bugfix: conditional variable broadcast channel size (#3906)
2023-02-06 - 9972e7738 - Fix backward compatibility with std visibility when doing search attribute operations (#3907)
2023-02-06 - a65c7d47a - Fix MySQL visibility select query (#3909)
2023-02-06 - 52c3a9eef - Fix silly bug in matching client (#3910)
2023-02-07 - a840d7529 - Strict validation when using SQL DB for visibility (#3905)
2023-02-07 - 793dd7e71 - Use max join time as start timeout (#3911)
2023-02-08 - 9186ca763 - Fix visibility full text search query in MySQL (#3913)
2023-02-08 - 9e37fece5 - Refactor failWorkflowTask method (#3915)
2023-02-08 - 908a672d3 - Fail workflow task with BadSearchAttributes cause if search attributes mapper returned an error (#3919)
2023-02-08 - 0da3d990e - Default authorizer always allow health checks (#3920)
2023-02-08 - 158a70eb7 - Reduce unnecessary log noise (#3918)
2023-02-09 - 304bc9861 - Fix MySQL visibility index for full text search (#3921)
2023-02-09 - 2d17cb83f - Replicate custom search attributes aliases with namespace replication (#3922)
2023-02-09 - d4498d1a8 - Expose history size to workflows (#3055)
2023-02-09 - 75c05a2ba - Call RedirectionInterceptor only for WorkflowHandler (#3924)
2023-02-09 - 77a4487b9 - Fix MySQL visibility indexes with close_time column (#3927)
2023-02-09 - 0da29e6cd - Fix typo in visibility schema for SQLite (#3932)
2023-02-09 - de6ca9c90 - Tokenize and build FTS query string for postgres and sqlite (#3933)
2023-02-09 - 7faf6f89d - Handle namespace division filter in SQL queries (#3931)
2023-02-10 - f3327486a - Schedules start workflows through frontend (#3935)
2023-02-10 - 1fb069706 - Make frontend drain traffic time configurable (#3934)
2023-02-10 - 2117c81dc - Add integration tests for eager workflow start (#3928)
2023-02-10 - 74bfaffa8 - Enable Elasticsearch functional tests for SQL advanced visibility (#3923)
2023-02-10 - 2d9bb1608 - Frontend health check waits until membership initialized (#3936)
2023-02-10 - 8501b046d - Remove foreign key from MySQL visibility schema (#3937)
2023-02-10 - 4f5b1b06d - Remove unnecessary aliasing/unaliasing of scheduled workflow search attributes (#3943)
2023-02-10 - 3b27372d0 - Support mysql8 and postgres12 dbs in docker config template (#3941)
2023-02-10 - df9ec3423 - Fix TestUpdateWeight (#3942)
2023-02-10 - 85c438143 - Address #3923 feedback (#3938)
2023-02-10 - e96b529a7 - Update failure detect default time (#3940)
2023-02-13 - 75a4c0f99 - Fix recreate custom search attribute after upgrade to advanced visibility with SQL (#3945)
2023-02-13 - 677890bc9 - Separate metrics on read history and read raw history (#3925)
2023-02-13 - 654bc3ffd - Fix create custom search attribute with SQL DB (#3951)
2023-02-13 - 4b78d28d3 - Fix ExecutionTime rounding when using PostgreSQL (#3952)
2023-02-14 - 0a22e1c8e - Fix Elasticsearch index creation in functional tests (#3954)
2023-02-14 - d353dcc43 - Emit inordered buffered events metric (#3949)
2023-02-14 - 0b698c1a0 - Wrap user's visibility query string in parenthesis (#3956)
2023-02-14 - f4e4ea2a8 - Remove isRecordValid check from ES visibility store (#3948)
2023-02-14 - 099cef3e2 - Support CountWorkflowExecutions for SQL DB (#3955)
2023-02-14 - 5d6d93c1c - Refactor SQL query converter (#3950)
2023-02-15 - c3578127d - Propagate SDK Metadata & add capability (#3944)
2023-02-15 - 8917ecc95 - Update Go SDK to v1.21.1 (#3960)
2023-02-15 - a23489b96 - Set namespace on API if not present (#3953)
2023-02-15 - 98a52d197 - Quick fix for generated mock file name (#3959)
2023-02-15 - 868b2d16c - Update IsAdvancedVisibilityActivity to include SQL DB (#3957)
2023-02-15 - 0bb688437 - Handle namespace not found case in redirection interceptor (#3947)
2023-02-15 - 2db41752a - Fix IWRR scheduler UpdateWeight test (#3961)
2023-02-16 - 7ab7e9c5d - Metering metadata propagation (#3965)
2023-02-16 - cbed0fe41 - Fix wrap user query string in parenthesis (#3967)
2023-02-16 - f9bd5b083 - DeleteExecutions workflow: pass nextPageToken with ContinueAsNewError (#3966)
2023-02-16 - 93fb0ec66 - Populate history size in describe workflow and list workflow response (#3964)
2023-02-16 - 06188f3bc - Suport dual visibility in visibility persistence checks (#3968)
2023-02-16 - b313b7f58 - Delete workflow execution from visibility store even if it was deleted from main store (#3962)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes a few minor bugs related to task processing and ES bulk processor. It also enables a security check feature by default to enforce the task token check to prohibit caller from modifiying namespace info in the task token.

All Changes

2022-12-29 - 6b2e44863 - Fix NPE in task channelWeightFn (#3766)
2023-01-12 - 23d8cf9a4 - Fix ES bulk processor commit timeout (#3696)
2023-01-12 - 7b64aa0a0 - Remove internal bulk processor retries (#3739)
2023-01-12 - c46fccbad - Capture task processing panic (#3779)
2023-01-12 - ea8af5056 - Disable eager activities for Python 0.1a1, 0.1a2, and0.1b1 too (#3793)
2023-01-12 - 5237e7f4a - Prioritize worker pool shutdown (#3795)
2023-01-12 - c22128835 - Capture panic in replication task processing (#3799)
2023-01-12 - a8da3587b - Fix recover call (#3804)
2023-01-12 - 5e784b48b - Drop task on serialization error (#3803)
2023-01-13 - 96d6b0064 - Turn on frontend.enableTokenNamespaceEnforcement by default
2023-01-13 - 95a40941b - Upgrade version to 1.19.1

Details about the v1.19.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.19.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

Schema changes

Before upgrading to your Temporal Cluster to release v1.19, you must upgrade your storage schema version to the following:

• MySQL schema version 1.9
• PostgreSQL schema version 1.9
• Cassandra schema version 1.7

Use the appropriate schema upgrade tool to upgrade your schema version.
For details, see https://docs.temporal.io/cluster-deployment-guide#upgrade-server.

Note that schema version is not the same as the storage type (database) version.

Golang version

• Upgraded golang to 1.19.

tctl

• tctl release with v1.17.2 (https://github.com/temporalio/tctl/releases/tag/v1.17.2).

Metric

A major refactoring has been done in the metrics code. Both metrics.Client and metrics.Scope are replaced by metrics.metricsHandler.

Customized metricsHandler is also supported by using temporal.WithCustomMetricsHandler .

Deprecated metrics

All metrics that have the suffix _per_tl are deprecated.

Improvements and fixes

Clean up mutable state when Workflow is closed

When the Workflow is closed, you can trigger a cleanup to remove Memos and Search Attributes from the mutable state. This helps reducing the size of the mutable state for closed Workflows. However, because the cleanup deletes data, it’s best to pair Advanced Visibility with Elasticsearch to ensure that Search Attributes can be retrieved. This feature is controlled by the dynamic config history.visibilityProcessorEnableCloseWorkflowCleanup.

Count limits for pending Child Workflows, Activities, Signals, and cancellation requests

This change adds a new system protection limit for the maximum number of pending Child Workflows, pending Activities, pending Signals to external Workflows, and pending requests to cancel external Workflows.
Operators can set different values for those new limits via the dynamic config. The default is currently 50,000 for each value.

limit.numPendingChildExecutions.error
limit.numPendingActivities.error
limit.numPendingSignals.error
limit.numPendingCancelRequests.error

Maximum number of concurrent pollers

The dynamic config frontend.namespaceCount controls how many concurrent pollers are allowed to connect to Temporal Server. Currently, this limit applies to the total number of pollers; that is, Activity plus Workflow pollers. In v1.19.0, this limit now applies to the number of pollers per type; that is, Activity and Workflow pollers won’t compete with each other for a connection to Temporal Server.

Batch deletion

The batch delete operation is now support in the batch operation API. Use BatchOperationDeletion option in the StartBatchOperationRequest for a batch delete operation.

History task processing

Default implementation

• Host-level Task scheduler is now enabled by default, meaning the default value of history.[timer|transfer|visibility]ProcessorEnablePriorityTaskScheduler is true.
• Multi-cursor queue implementation is now enabled by default, meaning the default value of history.[timer|transfer|visibility]ProcessorEnableMultiCursor is true.

Rate limiting

• Added priority rate limiter for Task processing. By default this rate limiter is disabled but can be enabled by setting history.taskSchedulerEnableRateLimiter to true. The rate limiter should be enabled only when multi-cursor queue implementation is enabled.
• The rate is controlled by history.taskSchedulerMaxQPS for per-host limit and history.taskSchedulerNamespaceMaxQPS for per-Namespace limit. All queues (timer, transfer, visibility) share the same limit. Please note that deletion Tasks have lower priority when consuming rate limit tokens and do not count against the per-Namespace limit.
• The default value for the preceding two configurations is 0, which means they will fall back to and use the value specified by history.persistenceMaxQPS and history.persistenceNamespaceMaxQPS.

Metrics

• task_latency_processing_nouserlatency, task_latency_userlatency, task_latency_nouserlatency, and task_latency_queue_nouserlatency are removed.
• New task_latency_user metric, which captures the latency for acquiring Workflow lock in a single Task Execution attempt.

History Scavenger

A retention validation on workflow has been added in the history scavenger. This feature is enabled by fault with worker.historyScannerVerifyRetention. A default grace period on this retention validation is 90 days + namespace retention time. The grace period is configurable by worker.executionDataDurationBuffer.

All changes

2022-09-19 - 9703d3363 - Post-release: bump version and upgrade dependencies (#3408)
2022-09-19 - 91c31f1b6 - Fix merge map of payload (#3412)
2022-09-19 - 1088382bd - Fix task reschedule policy (#3413)
2022-09-20 - c672c37c0 - Adds a retryable error for when we try to delete open executions (#3405)
2022-09-20 - d890a2f17 - Propagate CloseVisibilityTaskId to DeleteExecutionVisibilityTask (#3404)
2022-09-20 - be727b37a - Retry attempts to delete open visibility records (#3402)
2022-09-21 - 8f24d1fcb - Revert "Retry attempts to delete open visibility records" (#3420)
2022-09-21 - 9dfdf757a - Rename parameters of MergeMapOfPayload (#3418)
2022-09-22 - ff40b89e1 - Retry attempts to delete open visibility records (#3402) (#3421)
2022-09-23 - c6fb6b833 - Fixes an issue where last-heartbeat time was set to the first event's timestamp (#3361)
2022-09-26 - f4af2d5ff - Fix list batch operation to include division (#3431)
2022-09-27 - b4b61ff39 - Reorder grpc interceptors (#3423)
2022-09-27 - 85f400e57 - Add postgres es development script (#3429)
2022-09-27 - e3e1ccea6 - Retry attempts to delete open workflow executions (#3411)
2022-09-27 - 769b8658f - Add cluster ID into ringpop app (#3415)
2022-09-27 - 6530847a2 - Use original branch token instead of deserializing as branch info and then re-serializing (#3384)
2022-09-27 - d27ea8936 - Fix action metrics (#3434)
2022-09-28 - 8929def00 - Namespace replication for failover history (#3439)
2022-09-28 - c15047428 - Validate structured calendar specs and improve error messages (#3425)
2022-09-28 - e0bbabe21 - Per-namespace workers should only run on active cluster (#3426)
2022-09-29 - b497033e8 - Ensure urfave/cli accepts flag values with comma (#3440)
2022-09-29 - 363d4c0de - Log warning only when there is an error in SA size validation (#3443)
2022-09-29 - 97d0ec352 - Supply optionally configured workflow settings as hints (#3442)
2022-09-29 - 47bb15510 - Update dependencies and pin otel (#3444)
2022-09-29 - 38159152f - Implement GetAllHistoryTreeBranches for SQL persistence backends (#3438)
2022-09-30 - 3099274b4 - Fix reset workflow in replication reapply (#3449)
2022-10-03 - 175b916a4 - Use safer default TIMESTAMP for MySQL. (#3424)
2022-10-04 - a86d45534 - Index history size when workflow closes (#3451)
2022-10-05 - fc37cd4ba - Update ns version history in handover state (#3456)
2022-10-06 - 0ae4cc520 - Add config filter by task type (#3455)
2022-10-06 - 0df719d2d - Update replication use branch token (#3447)
2022-10-10 - fa51f5d24 - Move record child workflow completed to api package (#3350)
2022-10-10 - 62cd1439d - Move verify child workflow completion recorded to api package (#3351)
2022-10-10 - 71a1de37d - Use separate metric for resource exhausted error in task processing (#3463)
2022-10-10 - bc451dbac - Compare task type enum (#3464)
2022-10-10 - c6b472ef6 - Fix exclude tags with withTags method (#3466)
2022-10-10 - 158737abd - Remove old logic for checking workflow deletion task dependencies from delete_manager (#3427)
2022-10-10 - d33559a54 - Properly handle min task ID > max task ID case during shard re-balancing (#3470)
2022-10-11 - 0e70cf77f - Move get / poll mutable state to api package (#3467)
2022-10-11 - 7278168ad - Move describe workflow to api package (#3469)
2022-10-11 - 0dddd3cd6 - Fix timer task visibility timestamp for workflow refresh (#3460)
2022-10-11 - 429c0afc6 - Move replication APIs to api package (#3472)
2022-10-11 - d90f3abe0 - Move NDC logic to ndc package, move workflow reset to api package (#3465)
2022-10-11 - 740c7a3b1 - Move reapply events to api package (#3476)
2022-10-11 - d25265617 - Move remove signal from mutable state to api package (#3475)
2022-10-11 - b292c222d - Move delete workflow to api package (#3473)
2022-10-11 - 32ea0bf01 - Move refresh workflow to api package (#3477)
2022-10-12 - 16184da74 - Fix scheduled queue max read level update for single processor (#3474)
2022-10-12 - ceae24181 - Handle sync workflow state task in replication dlq handler (#3482)
2022-10-13 - bf8e1f170 - Bump UI to v2.7.0 (#3480)
2022-10-13 - 46dcb4fdb - Turns on the history scavenger for SQL backends (#3462)
2022-10-13 - 86fde8eff - Move query workflow to api package (#3486)
2022-10-13 - ef2140507 - Fix sanitize mutable state after replication (#3479)
2022-10-13 - 457eb05e6 - Clean up duplicate empty task id (#3490)
2022-10-13 - a0ea1772e - Add execution scavenger for retention (#3457)
2022-10-15 - a4fe3a7cf - Remove UI v1 from development environment (#3485)
2022-10-17 - dc84e87c5 - Update replication timestamp with no task (#3487)
2022-10-17 - 8e5ccb5a0 - schama -> schema (#3501)
2022-10-17 - b8d46a578 - Do not add version 0 to failover history (#3483)
2022-10-17 - cebbde9e5 - Sync proto API 1.12.0 (#3500)
2022-10-17 - 968c5765f - Create visibility GetWorkflowExecution API (#3488)
2022-10-18 - ecde5435b - Remove now parameter from task generator interface (#3478)
2022-10-18 - 9cb646be3 - Rewrite mysql PaginateBranchesFromHistoryTree query (#3509)
2022-10-18 - 8a7538ad1 - Warning log on new event during set workflow (#3508)
2022-10-19 - 8d645e85c - Increase visibility RPS and burst dynamic configs for integration tests (#3221)
2022-10-19 - 4b4493ba9 - Bump Golang version to 1.19 (#3507)
2022-10-19 - 3aff85ed6 - Fix scanner start dep (#3513)
2022-10-19 - c7713b14c - Support arbitrary history task category in SQL persistence (#3489)
2022-10-19 - d5edb0e61 - Plumb in timeouts instead of referencing the uninitialized execution info during workflow creation (#3512)
2022-10-20 - 6ee5ac524 - Check shard ownership by assertion (#3416)
2022-10-20 - c21ffad95 - initialize otelMetricsHandler logger (#3518)
2022-10-20 - af187569f - Simplify interface & remove unused vars (#3516)
2022-10-22 - 39ee37211 - Fix otel gauge metric type (#3525)
2022-10-24 - 827a59f82 - Support batch operation for delete workflow (#3497)
2022-10-25 - 42fca45ca - add metricstest package (#3528)
2022-10-26 - 5fac4d4eb - Add ArchiveExecutionTask to our set of tasks (#3524)
2022-10-26 - b42bc8593 - Add task version mismatch metrics (#3491)
2022-10-26 - 119478a59 - Change execution scavenger to call admin delete (#3526)
2022-10-26 - 275b62b64 - Fix replication task nil case (#3531)
2022-10-27 - 3a0d879ab - Remove SA and memo after close visibility task is completed (#3504)
2022-10-27 - ddc33bc3d - Skip archival if it was already done before deleting history data (#3523)
2022-10-28 - d4d7684e7 - Add service name tag to metrics emitted using metrics handler (#3537)
2022-10-31 - 4eec379f4 - Add retention validation in history scavenger (#3541)
2022-10-31 - 6d9c59528 - Increase timeouts in debug mode (#3542)
2022-10-31 - 88d34110d - Take rpc address from config for local cluster (#3546)
2022-10-31 - 1f697219e - Add deadlock detector (#3492)
2022-11-01 - 176cd42a9 - Add a new archiver client which doesn't retry (#3544)
2022-11-03 - 4a9d6ab38 - Queue processor handle shard ownership lost (#3553)
2022-11-03 - e2ef735f6 - Add a CanSkipVisibilityArchival flag to the CloseExecutionTask (#3529)
2022-11-03 - 7860ec059 - Fix shard info serialization (#3555)
2022-11-04 - 53000fc1d - Fix namespace handover log (#3559)
2022-11-04 - c2d5496e9 - Generate ArchiveExecutionTask instead of DeleteHistoryEventTask whenever an execution is closed (#3533)
2022-11-04 - 6721acec6 - Improve integration test framework (#3547)
2022-11-06 - b300010dd - Add some config values for the archival queue (#3554)
2022-11-07 - 02b8cca7b - Make shard more resilient to append history timeout (#3564)
2022-11-07 - 9fa1a9043 - Replace metrics client/scope with MetricsHandler (#3561)
2022-11-08 - bccf128f9 - Handle namespace not found in replication (#3557)
2022-11-08 - bbd386e03 - Add jitter for retention timer (#3569)
2022-11-09 - 86833988a - Fix task serialization in persistence tests (#3576)
2022-11-10 - 44154d324 - bump ui to v2.8.3 (#3573)
2022-11-11 - 5e466c30e - Fix metrics tag after metrics refactoring (#3579)
2022-11-14 - 574aee6b4 - Move queue metrics to metrics package (#3587)
2022-11-14 - 50936e1d5 - Add a tool to limit the number of pending child workflows (#3575)
2022-11-14 - ef05ac8bd - Fix scheduled queue lookahead (#3563)
2022-11-14 - 3a08eed68 - Limit Cassandra memory appetite to 1Gb in development configuration (#3581)
2022-11-14 - 76b58a74e - Fix resilient shard (#3584)
2022-11-14 - b74b0e3aa - Upgrade dev grafana version used in docker (#3593)
2022-11-14 - dde2afb3c - Update api and sdk references (#3595)
2022-11-14 - 4f2b98e89 - Wrap gocql.Iter with custom wrapper (#3577)
2022-11-14 - e7f99d074 - Fix history scavenger bug on delete mutable state (#3588)
2022-11-15 - c82d43c6d - Return an error whenever we exceed the number of pending child executions (#3586)
2022-11-15 - c14f64346 - Fix visibility queue log message and level (#3600)
2022-11-15 - 25e4a8fb3 - Don't log a warning when the number of child workflows is high (#3598)
2022-11-15 - 02b913fd4 - Clean up task latency metrics (#3510)
2022-11-16 - 3d32ebc82 - Add tdbg to goreleaser (#3609)
2022-11-16 - e96c17ae4 - Initialize new branch token based on replication target (#3604)
2022-11-17 - be30bab1a - Disable eager activity dispatch for buggy clients (#3610)
2022-11-17 - ee1abe1c7 - Fix scheduled task precision (#3591)
2022-11-18 - d34998708 - Add request validation on batch APIs (#3621)
2022-11-18 - be7daec26 - Add limits on other workflow fields (#3599)
2022-11-18 - e26ea3893 - Retry CreateIndex in integration tests request upon failure (#3618)
2022-11-18 - e4d1b98ad - Add methods to check workflow size violations for several other fields (#3602)
2022-11-18 - 81b79a73d - Change max namespace count limit to per poller type (#3608)
2022-11-18 - 66db3aebe - Remember retention timer task state in memory for retry (#3620)
2022-11-18 - 21118ed8b - Enforce the per-workflow pending activity constraint (#3611)
2022-11-18 - b13ed161f - Add flag to enable workflow closed clean up (#3603)
2022-11-18 - 4483f1023 - Handle unknown cluster during replication (#3619)
2022-11-18 - 0c735e496 - Fix function name typos (#3623)
2022-11-18 - df160db9f - Enforce the pending cancel request limit (#3615)
2022-11-18 - 32212eabd - Task scheduler rate limiter (#3606)
2022-11-18 - 91a67f41e - Enforce the per-workflow pending signal limit (#3617)
2022-11-18 - 954137483 - Keep multi cursor default reader always running (#3626)
2022-11-18 - b43ca639a - Fix immediate task fire time equality check (#3628)
2022-11-21 - 86631e9ce - Fix typo in dynamic config for VisibilityProcessorEnsureCloseBeforeDelete (#3632)
2022-11-21 - ded2f6761 - Improve test log format (#3634)
2022-11-22 - 3520803e2 - Extract duplicated getTemporalPackageDir into common function (#3637)
2022-11-22 - b0647ae55 - Randomization and configuration for scheduler throttle duration (#3636)
2022-11-22 - 13fa5f7bc - Handle unknown cluster in replication (#3639)
2022-11-22 - 473d48772 - Revert change 'De-duplicate reapply event' (#3640)
2022-11-22 - e40a1df2e - Enable history scavenger to scan expired workflow (#3643)
2022-11-22 - 13047df9b - Fix persistence GetHistoryTask (#3645)
2022-11-22 - 17b121a00 - Fix fail workflow batch id (#3646)
2022-11-22 - a4002ab04 - Fix pending tasks metric (#3648)
2022-11-22 - cd1872545 - Fix bug in GetRepoRootDirectory code (#3644)
2022-11-23 - d11074d0b - Make replication task reading logic more resilient (#3596)
2022-11-23 - d54cd1df9 - Simplify ServerOptions applyFunc (#3649)
2022-11-23 - 788b2b76a - Use fx group for deadlock detector (#3650)
2022-11-23 - 5658d5e79 - Disable visibility verify close before delete (#3651)
2022-11-23 - 9bbaa2731 - Handle unknown cluster with a unique string (#3652)
2022-11-23 - 5dd8cddb1 - Log when force replicate failed (#3653)
2022-11-23 - f46f2a466 - Fix one metrics name (#3654)
2022-11-23 - 306797b3b - Update persistence error metrics (#3655)
2022-11-24 - b717bd530 - Enforce min interval for continue as new runs (#3659)
2022-11-25 - 7a2fab580 - Fix a flaky persistence integration test (#3664)
2022-11-25 - 53f461577 - Enable host level pool and multi-cursor by default (#3660)
2022-11-28 - 001ec0e6a - Consolidate service name constants (#3656)
2022-11-28 - c2f21fe58 - Fix grpc listener initialization (#3657)
2022-11-28 - 0547245e7 - Set default catchup window eagerly (#3658)
2022-11-28 - 82cbe647a - make some s3 utility functions public (#3668)
2022-11-28 - 997434cc5 - Fix replication task execution latency (#3671)
2022-11-28 - b3812938f - Do not panic if passive processor is missing (#3672)
2022-11-28 - 362987f9b - Return an error when we fail to instantiate the server (#3642)
2022-11-28 - 937c38f8e - Update workflow state validator (#3673)
2022-11-28 - ee4648e43 - Set limit on DescribeSchedule query/signal loop (#3669)
2022-11-28 - 318749b75 - Don't wake up scheduler workflow when paused (#3670)
2022-11-29 - f38961142 - Fix defer not able to capture return error issue (#3675)
2022-11-29 - f6a340d23 - Fix slice pending task count (#3676)
2022-11-30 - 0101924db - Improve execution scavenger (#3674)
2022-12-01 - ff3f95cd3 - Remove empty visibility tag (#3681)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.19.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes minor bugs in history scavenger, gocql client recovery logic, and internal queue processing.

All Changes

2022-11-15 - 00bb51389 - Update version to 1.18.5
2022-11-15 - f103b1920 - Fix history scavenger bug on delete mutable state (#3588)
2022-11-15 - ec332ef75 - Wrap gocql.Iter with custom wrapper (#3577)
2022-11-15 - 077c5e0d7 - Fix resilient shard (#3584)
2022-11-15 - 0f68d2687 - Queue processor handle shard ownership lost (#3553)

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.5)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes a few minor bugs and upgrades the gocql dependency. It also upgrades the expat package in the admin-tools Docker image for CVE-2022-40674 (this doesn't affect the Temporal server itself).

All Changes

2022-10-27 - 0616c5c89 - Port util.CloneMapNonNil to release branch (#3530)
2022-10-31 - 327d46e45 - Upgrade gocql dependency to v1.2.1
2022-10-31 - 69e2bb949 - Change execution scavenger to call admin delete (#3526)
2022-10-31 - c6cfe6d52 - Fix replication task nil case (#3531)
2022-10-31 - 26782e9f5 - Add retention validation in history scavenger (#3541)
2022-10-31 - fe6a873b1 - Take rpc address from config for local cluster (#3546)
2022-10-31 - 03d6c0336 - Update version to 1.18.4

In docker-builds:

2022-10-31 - 85d405a - Use larger GitHub-hosted runners for Docker builds (# 74)
2022-10-31 - e85d0cc - Upgrade expat in admin-tools (# 75)
2022-10-31 - 07e4b80 - Update temporal submodule for branch release/v1.18.x

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.4)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools