Release Highlights

This release fixes few SQL-based advanced visibility bugs as well as minor bugs in various parts of the system.

All Changes

2023-03-24 - f38e041e7 - Fix passing pointer of slice element in SQL query converter (#3970)
2023-03-24 - 6d28fe257 - Refactor SQL query converter (#3971)
2023-03-24 - f714a91dc - Add SQL query converter tests (#3972)
2023-03-24 - 5c06a2bdb - Flush buffer event with single cluster namespace (#4031)
2023-03-24 - 1855bb4ef - Refactor gocql.NewSession to allow passing in cluster config factory method (#4033)
2023-03-24 - d8aaa5f25 - Extract cluster config logic into ConfigureCassandraCluster() (#4034)
2023-03-24 - f17467100 - Convert query converter error to invalid argument (#4036)
2023-03-24 - 07cf95172 - Fix UnhandledCommand error for workflow task completion in CancelTimer (#4050)
2023-03-24 - 305b770c7 - Change disable order by dynamic config to per namespace (#4063)
2023-03-24 - f8c06b537 - Set default values in returned calendar specs (#3969)
2023-03-24 - 8c1a0fb12 - Add counter to track custom order by usage in Elasticsearch (#4064)
2023-03-24 - 89f2095bf - Fixing saving custom search attribute after upgrading SQL DB (#4062)
2023-03-24 - b0ef992c0 - Fix SQL visibility page token deserializer (#4074)
2023-03-24 - 5a83f71b2 - Add deadlines to schedule local activities (#4078)
2023-03-24 - cb7d42c85 - Use preemptable caller type for low priority history tasks (#3993)
2023-03-24 - 633a05657 - Lock current execution on start workflow (#4066)
2023-03-24 - 145c99b76 - Handle max signal count limit for signal transfer task (#4051)
2023-03-24 - a8b591d15 - Track and emit metrics for entities in a single workflow (#4065)
2023-03-24 - df2d8eb67 - Fix namespace handover replication queue notification (#4082)
2023-03-24 - de840920c - Use more reliable workflow mutation check (#4076)
2023-03-24 - c0484f9dc - Fix test failure from merge conflict (#4087)
2023-03-24 - dcaa3a378 - Account for local activity run time in schedule sleep (#4079)
2023-03-24 - 57347cb91 - Flush buffer event when failing workflow task (#4010)
2023-03-24 - cbb9db75f - Support empty queries when listing archived workflows (#4027)
2023-03-24 - aabbae0d7 - Fail QueryWorkflow if last workflow task failed (#4099)
2023-03-24 - 6b747d3b0 - Refresh per-namespace worker periodically (#4103)
2023-03-24 - 7c0a0e036 - Change update namespace to upsert custom search attributes (#4080)
2023-03-28 - 212332b6a - Add flag to enable execution scanner event Id validator (#4114)
2023-03-29 - cf61e9eed - Enforce concurrent count limit to all long poll calls (#4120)
2023-03-28 - 292f4600d - Bump server version for 1.20.1 patch release

Details about the v1.20.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

Upgrade action

• SERVICE_FLAGS environment variable for docker images is removed, use SERVICES instead.
• The dynamic config setting matching.useOldRouting was removed and old routing is not available anymore. If you’re upgrading and did not set it earlier, set it to false across your cluster before the upgrade to minimize task dispatch disruption during deployment.

Deprecation heads-up

The following softwares will not be supported in future versions:

• MySQL 5.7: planned to be removed by v1.21; please upgrade to MySQL 8.0.17+.
• PostgreSQL 9.6 to 11: planned to be removed by v1.21; please upgrade to PostgreSQL 12+.

Advanced visibility

We’re launching advanced visibility features for SQL databases (see more below). However, this required a significant change in the existing functionality when used with Elasticsearch. Up until v1.19, when you create a custom search attribute of a given type, say Keyword, you could actually store a list of Keyword values. This will no longer be supported. If you have a search attribute of type Keyword, you will only be able to store a single Keyword value. If you wish to store a list of values, you can create a search attribute of type KeywordList. This is the only type that supports list of values, ie., there’s no equivalent for any other type (Int, Datetime, etc.).

$ temporal operator search-attribute create --name CustomField --type KeywordList

If you are using Elasticsearch, you can change the type of a Keyword search attribute to KeywordList type by re-creating the search attribute (warning: this will only work as expected if the value stored is already a list; otherwise, it will lead to unexpected errors):

$ temporal operator search-attribute remove --name CustomField
$ temporal operator search-attribute create --name CustomField --type KeywordList

For backwards compatibility, if you’re using Elasticsearch, list of values is still supported for all search attribute types. However, we discourage such usage. Furthermore, future releases of SDK will deprecate API accepting list of values, and change it to be strong typed so it won't accept list of values.

Advanced visibility on SQL DB (beta testing)

Advanced visibility features are now supported using one of the supported SQL databases (MySQL 8.0.17+, PostgreSQL 12+, and SQLite):

• Per namespace, you can create up to 3 custom search attributes for each of the types Bool, Datetime, Double, Int, Text and KeywordList, and up to 10 custom search attributes of type Keyword.
• You’ll be able to call the List API using SQL-like queries (ORDER BY and LIKE operators are not supported, check the documentation for more details).

Here are the steps to enable this feature [warning: for all commands below, please be aware of replacing the database parameters (user, password, host, port) accordingly to your use case]:

1. Upgrade Temporal Server to v1.20. If you don't want to enable advanced visibility features, then stop here. Everything should work as usual.

2. Upgrade your database to MySQL 8.0.17+ or PostgreSQL 12+ if you’re using an older version. Please check with official documentation on how to upgrade MySQL and PostgreSQL. We also recommend making a backup of your data before taking any actions.

   • [Optional step for MySQL users] If you upgraded from MySQL 5.7, convert the database character set to utf8mb4(learn more about character sets in the MySQL documentation). The following commands are an example of how you can convert both the temporal and temporal_visibility databases (check the MySQL documentation for more details on changing the character set for databases and tables):

     $ DB=temporal; ( echo 'SET foreign_key_checks=0; ALTER DATABASE `'$DB'` CHARACTER SET utf8mb4;'; mysql $DB -u root -proot -e "SHOW TABLES" --batch --skip-column-names | xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8mb4;' ) | mysql $DB -u root -proot
     $ DB=temporal_visibility; ( echo 'SET foreign_key_checks=0; ALTER DATABASE `'$DB'` CHARACTER SET utf8mb4;'; mysql $DB -u root -proot -e "SHOW TABLES" --batch --skip-column-names | xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8mb4;' ) | mysql $DB -u root -proot
     

3. Update the schema (if you need to set TLS cert, check our documentation for more details on the temporal-sql-tool command):

   • MySQL 8.0.17+:

     $ ./temporal-sql-tool --ep localhost -p 3306 -u root -pw root --pl mysql8 --db temporal_visibility update-schema -d ./schema/mysql/v8/visibility/versioned
     

   • PostgreSQL 12+:

     $ ./temporal-sql-tool --ep localhost -p 5432 -u temporal -pw temporal --pl postgres12 --db temporal_visibility update-schema -d ./schema/postgresql/v12/visibility/versioned
     

4. Change the sql.pluginName config to mysql8 or postgres12 accordingly.

5. Restart Temporal Server.

If you have previously created custom search attributes while using standard visibility, you’ll need to create them again. Do not delete the existing ones, just execute the command to create a new custom search attribute. For example, if you had created a custom search attribute named CustomField of type Keyword, then execute the following command (make sure you update to the latest version of Temporal CLI tool):

$ temporal operator search-attribute create --namespace default --name CustomField --type Keyword

For SQLite users, we do not support schema updates. Therefore, if you're using SQLite file-based storage, you'll need to reset your database from scratch.

Archival feature

We reimplemented the Archival feature using a separate task queue, providing higher efficiency, better resource isolation, and strong durability guarantees. Most importantly, Archival will now infinitely retry with an exponential backoff function before deleting any data. Before this change, Archival would only retry for a maximum of five minutes.

Workflow Update (alpha)

Workflow Updates enable gRPC clients to issue requests to workflow executions that mutate internal workflow state and return a response.

• This is an alpha quality release that must not be used for production workloads. Only workers using the Go SDK are supported. Workflow histories that include update events from this release may not be replayable by future releases.
• Workflow Updates are disabled by default. To enable UpdateWorkflowExecution API change frontend.enableUpdateWorkflowExecution dynamic config value to true.
• There is a new example in the samples-go repository demonstrating both how to invoke and how to handle updates.

Replication between clusters with different shard counts

Temporal Server 1.20 added capability to connect and replicate between 2 clusters with different shard count. The requirement is one cluster’s shard count is another one’s multiple (Y=NX, Y and X are shard count, and N is a natural number). Shard count is a scale unit, cluster could run into scale limit if the choose shard count is too small. This new feature make it possible to move live workload to a new cluster with larger shard count. It works the other way as well if you want to reduce your shard count.

Note: Temporal server 1.19 and earlier versions require 2 clusters to have exact same shard count if they want to be connected.

BatchOperation to support DeleteWorkflow

Add delete workflow as another supported action to BatchOperation API.

Otel 0.34

Open telemetry upgraded to 0.34. 0.34 Otel release contains a breaking change. All counters will be append a ‘_total’ suffix.

Default authorizer

If you're using the default authorizer (note that the default authorizer is not used by default, you have to explicitly configure it), you should be aware of a few changes:

• The special case for temporal-system namespace was removed. This namespace is used by various workflows that run on the system worker service.

  Separately, two recent features, Schedules and Batch v2, require the system worker to connect to namespaces other than temporal-system. If your claim mapper was not properly authenticating the system worker service and giving it a System: RoleAdmin claim, then this special case meant that system workflows would work but Schedules and Batch v2 would not.

  Now that this case is removed, your system workers might have trouble connecting to the frontend unless you make some other changes. Either 1) your claim mapper should properly authenticate the system worker and give it a System: RoleAdmin claim, or 2) use Internal Frontend to bypass the claim mapper for internode connections to frontend. See the Internal Frontend section below.

• The special case for requests with no namespace was removed, and replaced with a check for two specific requests: gRPC health checks, and GetSystemInfo. All other requests with no namespace in the request, e.g. ListNamespaces now require corresponding claims to be allowed. If you were relying on that special case, you may have to adjust your claim mapper to add a System: RoleReader claim for authorized clients.

  Because frontend.enableTokenNamespaceEnforcement defaulted to false, the no-namespace special case could also have been a security hole, depending on your claim mapper: if a client sent a request with a namespace in the token but an empty namespace field at the top level of the request, it would pass the authorizer without a proper check for a namespace claim. That dynamic config now defaults to true.

Internal frontend

There's a new deployment option that can simplify worker→frontend communication, mTLS, claim mappers, and authorizers: the internal-frontend. You should consider using internal-frontend if:

• You're using a claim mapper and authorizer (either default or custom) and there's no easy way to differentiate between the system worker service and external clients.
• You were relying on the special case for temporal-system (see above) in the default authorizer or your custom authorizer.
• You're using mTLS and you want to use a different CA for internode connections and external client connections to frontend.

Basically, internal-frontend makes the system worker service and frontend work like the other "internode" services (history and matching) so you can secure them in the same way.

To use it:

• Create an internal-frontend section in your static config under services by copying frontend and adjusting the ports (we suggest 7236/6936 but any works).
• Make sure the publicClient section is removed from static config.
• Add a new deployment that uses -service internal-frontend on its command line. If using mTLS, provide it with the same "internode" certs and keys as history and matching.

If you're using pre-built docker images, set environment variable USE_INTERNAL_FRONTEND to 1 instead of the static config changes above.

Note that metrics will appear with service_name=internal-frontend so you might have to make adjustments to your obsrevability as well. Logs will appear tagged with {"service":"frontend","internal-frontend":true}.

You can use dynamic configs internal-frontend.globalNamespaceRPS and internal-frontend.globalNamespaceRPS.visibility to adjust the incoming rate limits for internal-frontend separately from [external] frontend.

Security-related changes

• Multiple changes to the default authorizer (see Default authorizer section above) https://github.com/temporalio/temporal/pull/3706
• frontend.enableTokenNamespaceEnforcement dynamic configuration setting now defaults to true https://github.com/temporalio/temporal/pull/3798
  • The previous setting allowed cross-namespace access when presenting a task token where the namespace ID and the namespace fields have different values.
  • Task tokens are generally handled entirely by clients, but can be created by end-users
  • Thanks to Jacob LeGrone at DataDog for finding and reporting this insecure default.
  • We strongly urge all users to upgrade to 1.20 or, if unable to upgrade, enable this configuration option

Scheduled Workflows GA

There are a few new dynamic config options for tuning the system worker service and schedules:

• worker.perNamespaceWorkerOptions: map option to set options for workers for schedules and batch v2. Allows namespace constraints. Set values like this:

  worker.perNamespaceWorkerOptions:
  - value:
      MaxConcurrentWorkflowTaskPollers: 10
  

• worker.stickyCacheSize: integer option to control sticky cache size for workers. Global to the process. Defaults to the SDK default (10,000).

• worker.schedulerNamespaceStartWorkflowRPS: float value to set a maximum
  number of workflows started by schedules at the namespace level. Allows
  namespace constraints. Defaults to 30.

You may need to adjust these if you're making heavy use of the schedules feature.

There are a few new metrics you can monitor also: schedule_missed_catchup_window, schedule_rate_limited, schedule_buffer_overruns, schedule_action_success, schedule_action_errors, schedule_cancel_workflow_errors schedule_terminate_workflow_errors.

With these changes and the authorizer/internal-frontend changes to make it easy for the system worker service to have the necessary access in all configurations, we’re calling the scheduled workflows feature generally available.

All changes

2022-11-30 - 681ce454f - Fix service_latency_nouserlatency (#3677)
2022-12-01 - 79a572de6 - Remove empty visibility tag (#3681)
2022-12-01 - 21ed2aa5a - Update next release version (#3683)
2022-12-05 - 4907ff999 - Use empty namespace metrics tag if namespace not found (#3686)
2022-12-05 - c94d2bfa5 - Refactor history cache to its own package (#3601)
2022-12-05 - 34e256a95 - Test that ShardOwnershipLostErrors are never retried (#3625)
2022-12-07 - d112ea818 - Update dashboards (#3694)
2022-12-07 - bece27de3 - Add metrics scopes for the archival queue (#3566)
2022-12-08 - 72aa9bd03 - Add an integration test for acquireShard (#3678)
2022-12-08 - eb88af41e - Add random delay to ArchiveExecutionTask (#3565)
2022-12-08 - 8a71b9455 - Add a utility for reliably fetching search attributes and memo (#3687)
2022-12-08 - f0c2e5871 - Disable the Test_DeleteExecutionsWorkflow_ManyExecutions_NoContinueAsNew test because it is flaky (#3699)
2022-12-08 - 1af8a94da - Fix flaky CreateIndex method (#3698)
2022-12-08 - b1943f740 - docs: add specific make step to README on the tools/sql path (#3697)
2022-12-08 - b47046f5b - Move ServiceName to primitives and use everywhere (#3695)
2022-12-09 - e12743079 - Update Go SDK to v1.19.0 (#3701)
2022-12-09 - 55cdd690d - Adds DPanic and Panic level logging (writes log, then panics) (#3689)
2022-12-09 - ad1176c51 - Adds DPanic and Panic level logging (writes log, then panics) (#3689)
2022-12-09 - 4c4ef467e - Adds DPanic and Panic level logging (writes log, then panics) (#3689)
2022-12-09 - ae223cf1e - Filter replication task on source cluster (#3641)
2022-12-09 - 912dc51b9 - Fix flaky shard controller tests (#3693)
2022-12-12 - 0f27ed22a - Add sdk-features trigger to PRs (#3627)
2022-12-12 - 8ca4b2cc8 - Handle data loss error in replication read history (#3704)
2022-12-13 - 2abbea974 - Separate archival integration tests into their own suite (#3707)
2022-12-13 - 033154933 - Add an archival task executor (#3663)
2022-12-13 - 532c422c2 - Add an archival queue factory (#3562)
2022-12-13 - 7a147f6e3 - Use namespace from schedule activity worker (#3680)
2022-12-15 - ea698fa1b - Remove old queue processing logic (#3716)
2022-12-15 - 0b4df6a67 - Rename metrics.Handler to avoid name stuttering (#3713)
2022-12-16 - 728f6ce68 - Improve archival_test (#3719)
2022-12-16 - 5a4e0a97f - Fix ArchiveExecutionTask key type (#3720)
2022-12-16 - 70b59693d - Remove unused cluster parameter from task notification (#3718)
2022-12-16 - edd3f0a4b - Fix bug not setting CanSkipVisibilityArchival (#3721)
2022-12-19 - 23a2c4e67 - Fix errcheck warnings in cmd/tools (#3729)
2022-12-20 - a7240b4de - Fix ES bulk processor commit timeout (#3696)
2022-12-20 - 6fbc4f4af - Rename mutableStateImpl receiver (#3737)
2022-12-20 - 0ad773b8c - Remove scheduled and started events from transient workflow task info (#3736)
2022-12-20 - b57241955 - Fix bug where we return nil when there's an error in the archival queue (#3723)
2022-12-20 - e57edab44 - Fix range variable in archiver_test not being captured (#3724)
2022-12-20 - fb3f2f969 - Fix errcheck in service/frontend (#3733)
2022-12-20 - 12e416293 - Document that errors in goro.Group.Go functions are ignored (#3734)
2022-12-20 - c6a89c095 - Workflow task state machine: minor renames (#3735)
2022-12-20 - 2b761b495 - Remove internal bulk processor retries (#3739)
2022-12-21 - 661880f37 - Refactor transient workflow task state machine (#3740)
2022-12-21 - b2c5357ec - Verify cluster name in remove cluster call (#3715)
2022-12-21 - 11b808ad6 - Fix errcheck in ./host/ (#3741)
2022-12-21 - d8961577b - Fix errcheck in ./common/persistence/ (#3743)
2022-12-21 - 1454f84c0 - Fix errcheck in service/worker/ (#3731)
2022-12-21 - 957cfa4f2 - Fix errcheck in ./common/archiver/ (#3744)
2022-12-21 - 676b74459 - Fix errcheck in ./common/membership/ (#3742)
2022-12-21 - 3cb92321a - Clean namespace handover (#3692)
2022-12-21 - 2171a17e3 - Fix errcheck in ./common/rpc/ (#3752)
2022-12-22 - 3a322bb07 - Use latest replication task time as sync shard time (#3722)
2022-12-22 - 86e9d324b - Consolidate linters with golangci-lint (#3758)
2022-12-22 - ba2d711e7 - Remove unused Temporal interface (#3761)
2022-12-22 - 825d39c6a - Fix errcheck in service/matching (#3756)
2022-12-22 - 329290776 - Fix errcheck in temporal/ (#3757)
2022-12-22 - f09ea5cbf - Fix errcheck in service/history (#3754)
2022-12-23 - ff5805726 - Handle unhandled errors in tests/ directory (#3730)
2022-12-23 - 930a5e8a8 - Handle unhandled errors in tools/ (#3748)
2022-12-23 - 5278b2e97 - Fix some unhandled deferred errors in ./common/ (#3750)
2022-12-23 - 7d6608749 - Fix gofmt and add Go report card (#3751)
2022-12-28 - 4e8601a05 - Create an FX Module for Archival (#3726)
2022-12-28 - 5640a989a - Fix some lostcancel errors (#3764)
2022-12-28 - c9124547f - Fix errcheck in service/history/shard (#3755)
2022-12-28 - 92b875889 - Add an integration test for durable archival (#3732)
2022-12-28 - e57f97f9f - Fix token count in archival rate limiter (#3749)
2022-12-28 - d970aa493 - Fine-tune the Revive linter (#3763)
2022-12-29 - 98545f9b3 - Remove unused parameters from task executable (#3747)
2022-12-29 - adf7c5471 - Fix NPE in task channelWeightFn (#3766)
2023-01-03 - a22db81db - Remove deprecated code from queue implementation (#3770)
2023-01-03 - 32b5d9133 - Make persistence tests consistent in handling ShardID (#3772)
2023-01-04 - 7215a3253 - Validate shard id in shard controller (#3776)
2023-01-04 - a026002b5 - Remove task max retry count config (#3771)
2023-01-04 - ac8717cc8 - Separate integration tests and unit tests (#3760)
2023-01-06 - 9c2528620 - Trigger golangci-lint on pull requests (#3781)
2023-01-06 - e91760f88 - Add speculative workflow task (#3768)
2023-01-06 - 3882bb635 - Capture task processing panic (#3779)
2023-01-06 - 04ce8e382 - Deprecate old namespace change callback (#3774)
2023-01-09 - aa7671974 - Expose replicateWorkflowState API in history (#3783)
2023-01-09 - 3e31880b0 - Add a component to assign polling shards (#3775)
2023-01-09 - ea893d18a - Disable eager activities for Python 0.1a1, 0.1a2, and0.1b1 too (#3793)
2023-01-10 - bb4ec7f01 - Prioritize worker pool shutdown (#3795)
2023-01-10 - c5254f396 - Address a few linter errors (#3780)
2023-01-10 - 1a9695e10 - Add internal-frontend role (#3706)
2023-01-11 - 851c634c6 - Rename host tests to functional tests (#3782)
2023-01-11 - e5883b930 - Minor test target fixes (#3801)
2023-01-11 - bd13fc6b3 - TW 219 - update links across repos (#3785)
2023-01-11 - ef982348f - Turn on frontend.enableTokenNamespaceEnforcement by default (#3798)
2023-01-11 - 6319a2848 - Add NewInternalErrorWithDPanic for common dpanic pattern (#3797)
2023-01-11 - 5b182a72d - Capture panic in replication task processing (#3799)
2023-01-11 - d527a1992 - Rename sdk-features to features (#3796)
2023-01-11 - daff5f1bf - Add separate dynamic config knobs for internal-frontend rate limiting (#3800)
2023-01-11 - dc932a16d - Fix recover call (#3804)
2023-01-11 - bd068266e - Read cluster shard count from DB (#3788)
2023-01-12 - 26503cfbb - Drop task on serialization error (#3803)
2023-01-12 - b132fc2d4 - Redirect replication requests when shard count is different (#3789)
2023-01-13 - 0bf1137f0 - Add test definition in readme (#3807)
2023-01-13 - 51fd1dd7c - Allow start many replication pollers in one shard (#3790)
2023-01-13 - 394885498 - Allow connect clusters with different shards (#3777)
2023-01-17 - 0a90b770e - Fix datarace in forwarder_test (#3767)
2023-01-17 - efb3bc100 - Fix redirect replicate workflow state (#3809)
2023-01-17 - c24c83f85 - Use generics in jitter (#3717)
2023-01-18 - a54483d91 - Fix flaky jitter test (#3814)
2023-01-19 - 4576a1046 - Add debug.TimeoutMultiplier to all timeouts (#3815)
2023-01-19 - 5413d1866 - Add mock archival queue processors to history engine tests (#3817)
2023-01-19 - eb4b574bc - Fix some bugs in the archival queue task executor (#3813)
2023-01-19 - e0081fde1 - Fix a rare deadlock in scanner.Stop (#3818)
2023-01-20 - 82c958d33 - Partial revert #3731, fix errcheck, add comments (#3787)
2023-01-20 - 5175ab603 - Single scheduled queue max read level (#3778)
2023-01-24 - 9389673ba - Support batch operation with list of workflow executions (#3812)
2023-01-24 - 3abd50d31 - Better SQL query splitter (#3791)
2023-01-24 - b9bba94dd - Fix GetWorkflowExecution in PostgreSQL (#3816)
2023-01-24 - 554b32d33 - Produce archival tasks conditionally (#3823)
2023-01-24 - 285bf333f - Start the archival queue iff history or visibility archival is enabled in the static config (#3827)
2023-01-24 - a06da897c - Remove matching.useOldRouting dynamic config (#3824)
2023-01-24 - 278f545c2 - Add dynamic config for sdk worker options (#3806)
2023-01-24 - 4c7189903 - Update Go SDK to 1.20.0 (#3810)
2023-01-25 - 7e54876e1 - Rename UpdateWorkflow to UpdateWorkflowExecution (#3832)
2023-01-25 - 1c638ac56 - Enable durable archival (#3828)
2023-01-25 - e6113da91 - Fix SQL NPE after connection is closed (#3829)
2023-01-25 - 430d3a90a - Messages protocol implementation (#3843)
2023-01-25 - 1543bd4ad - Update UI to v2.10.0 (#3841)
2023-01-25 - 5652dc4f1 - Fix time fields of rows from GetClusterMembers (#3840)
2023-01-25 - 4f53af912 - Tune linters (#3848)
2023-01-25 - 5338f96fb - Add env vars to control log format and level for tests (#3836)
2023-01-25 - b4947362d - Add RefreshPerNSWorkerManager and RemoveOverride for tests (#3837)
2023-01-25 - 1c84aadc2 - Fix continue as new validation (#3845)
2023-01-25 - 5c23836c0 - Fix handover callback (#3847)
2023-01-25 - 7e89af482 - Add explicit per-namespace rate limit to schedules (#3838)
2023-01-25 - 909ca9688 - Add initial memo to scheduler workflows (#3839)
2023-01-26 - c52848e8d - Revert #3755 (#3842)
2023-01-26 - 9d3fedeec - Add index name to constructor of search attribute validator (#3834)
2023-01-26 - 25015d470 - Emit lag metrics for all registered history queues (#3851)
2023-01-26 - de6f2dfde - Remove internal-frontend from default services (#3849)
2023-01-26 - 391ed6109 - Factor out task queue name mangling into new package (#3549)
2023-01-26 - d1c5d108f - Change deadline too short to failedPrecondition error (#3856)
2023-01-26 - fc6efef1f - Remove failover levels (#3854)
2023-01-27 - dc257ce87 - Add GetIndexName method to visibility manager (#3820)
2023-01-27 - 3b3498a38 - Move namespace handler to service/frontend (#3855)
2023-01-27 - 0e4506be2 - Remove internal-frontend from development configs (#3859)
2023-01-27 - 6e732e79d - Return last error instead of ctx error (#3857)
2023-01-30 - 9a4f19e15 - Mark state transition & completion metrics with namespace activeness (#3831)
2023-01-30 - 3501f6700 - Conditionally register archival category (#3867)
2023-01-30 - b40a4ac97 - Fix activity eager execution on workflow close (#3868)
2023-01-30 - 473b449d4 - update development version of temporalio/ui to 2.10.1 (#3869)
2023-01-30 - 7e83ca2fc - Add custom search attribute aliases map to namespace config (#3866)
2023-01-30 - 416abfbda - Update type of keyword list search attributes (#3865)
2023-01-30 - fa429a5ab - [Advanced Visibility with SQL] Adding MySQL 8 schema and interface (#3552)
2023-01-31 - 7e3daf1fd - Add lower priority for replication tasks (#3870)
2023-01-31 - 962985432 - Update search attributes operator commands to handle aliasing (#3871)
2023-01-31 - f3cf50429 - [Advanced Visibility with SQL] Changing visibility schema for SQLite (#3819)
2023-01-31 - cb22aab7c - [Advanced Visibility with SQL] Adding PostgreSQL 12 schema and interface (#3844)
2023-01-31 - 750fc54c3 - Implement searchattribute.MapperProvider (#3873)
2023-01-31 - 29afc2f1e - Handle updating cluster metadata search attributes info when using SQL DB (#3861)
2023-01-31 - c257d84c1 - Implement default mapper for SQL visibility (#3875)
2023-02-01 - de0b04906 - Fix visibility processor panic on add after stop (#3830)
2023-02-01 - 2bc48d21f - Upgrade otel to 0.34 (#3850)
2023-02-01 - af0071934 - Merge custom search attributes from index name with empty string key (#3877)
2023-02-01 - d9bac92f2 - Return error if cluster metadata is invalid (#3879)
2023-02-01 - cbd73e01e - Synchronous workflow update (#3822)
2023-02-01 - dbeab313e - clean replication task processor after disconnect (#3882)
2023-02-01 - 6ef774953 - Eager workflow dispatch (#3835)
2023-02-01 - 8b6b2fbab - Keep queue factory group tag (#3880)
2023-02-02 - b5f64f2f6 - Protect UpdateWorkflowExecution API with dynamic config feature flag (#3884)
2023-02-02 - 5721cf897 - Add MySQL 8 admin interface (#3886)
2023-02-02 - 19d36fb49 - Fix docker config template to set only one visibility (#3889)
2023-02-02 - cf9e84adf - Overwrite operation for admin and operator APIs (#3890)
2023-02-02 - e43e5b500 - Advanced visibility for MySQL (#3878)
2023-02-02 - f94e57545 - Update admin handler to handle search attributes operations with SQL DB (#3894)
2023-02-02 - 52d1ba08f - Advanced visibility for PostgreSQL (#3896)
2023-02-02 - 32a468689 - Pass workflowTask object to completion methods of mutable state (#3897)
2023-02-03 - bd2d3b3cb - Advanced visibility for SQLite (#3895)
2023-02-03 - 82f911ea5 - Disable unused-parameter linter (#3891)
2023-02-03 - 75cf92b4d - Ignore unhandled errors from fmt.Printf (#3892)
2023-02-03 - c7d7a896e - Listen for loopback instead of all incoming traffic in RP test (#3893)
2023-02-03 - a4943bbd2 - Add metrics + max buffer size to schedule workflow (#3872)
2023-02-03 - 30264ca61 - Update SDK dependency to v1.21.0 (#3902)
2023-02-03 - d43e112d9 - Fix fail of speculative workflow task (#3898)
2023-02-03 - e95e082e7 - Fix namespace interceptor to allow search attributes operations (#3899)
2023-02-03 - 20a5ce339 - Remove search attributes with nil values (#3903)
2023-02-03 - 3ade2631d - Allow setting cluster Id explicitly (#3883)
2023-02-03 - 008f056d6 - Improve task scheduler rate limiter (#3860)
2023-02-03 - ca5be2995 - Reimplement DC redirection handler (#3887)
2023-02-03 - a070fb511 - Fix clientFactory build issue (#3904)
2023-02-04 - 157cd1142 - Bugfix: conditional variable broadcast channel size (#3906)
2023-02-06 - 9972e7738 - Fix backward compatibility with std visibility when doing search attribute operations (#3907)
2023-02-06 - a65c7d47a - Fix MySQL visibility select query (#3909)
2023-02-06 - 52c3a9eef - Fix silly bug in matching client (#3910)
2023-02-07 - a840d7529 - Strict validation when using SQL DB for visibility (#3905)
2023-02-07 - 793dd7e71 - Use max join time as start timeout (#3911)
2023-02-08 - 9186ca763 - Fix visibility full text search query in MySQL (#3913)
2023-02-08 - 9e37fece5 - Refactor failWorkflowTask method (#3915)
2023-02-08 - 908a672d3 - Fail workflow task with BadSearchAttributes cause if search attributes mapper returned an error (#3919)
2023-02-08 - 0da3d990e - Default authorizer always allow health checks (#3920)
2023-02-08 - 158a70eb7 - Reduce unnecessary log noise (#3918)
2023-02-09 - 304bc9861 - Fix MySQL visibility index for full text search (#3921)
2023-02-09 - 2d17cb83f - Replicate custom search attributes aliases with namespace replication (#3922)
2023-02-09 - d4498d1a8 - Expose history size to workflows (#3055)
2023-02-09 - 75c05a2ba - Call RedirectionInterceptor only for WorkflowHandler (#3924)
2023-02-09 - 77a4487b9 - Fix MySQL visibility indexes with close_time column (#3927)
2023-02-09 - 0da29e6cd - Fix typo in visibility schema for SQLite (#3932)
2023-02-09 - de6ca9c90 - Tokenize and build FTS query string for postgres and sqlite (#3933)
2023-02-09 - 7faf6f89d - Handle namespace division filter in SQL queries (#3931)
2023-02-10 - f3327486a - Schedules start workflows through frontend (#3935)
2023-02-10 - 1fb069706 - Make frontend drain traffic time configurable (#3934)
2023-02-10 - 2117c81dc - Add integration tests for eager workflow start (#3928)
2023-02-10 - 74bfaffa8 - Enable Elasticsearch functional tests for SQL advanced visibility (#3923)
2023-02-10 - 2d9bb1608 - Frontend health check waits until membership initialized (#3936)
2023-02-10 - 8501b046d - Remove foreign key from MySQL visibility schema (#3937)
2023-02-10 - 4f5b1b06d - Remove unnecessary aliasing/unaliasing of scheduled workflow search attributes (#3943)
2023-02-10 - 3b27372d0 - Support mysql8 and postgres12 dbs in docker config template (#3941)
2023-02-10 - df9ec3423 - Fix TestUpdateWeight (#3942)
2023-02-10 - 85c438143 - Address #3923 feedback (#3938)
2023-02-10 - e96b529a7 - Update failure detect default time (#3940)
2023-02-13 - 75a4c0f99 - Fix recreate custom search attribute after upgrade to advanced visibility with SQL (#3945)
2023-02-13 - 677890bc9 - Separate metrics on read history and read raw history (#3925)
2023-02-13 - 654bc3ffd - Fix create custom search attribute with SQL DB (#3951)
2023-02-13 - 4b78d28d3 - Fix ExecutionTime rounding when using PostgreSQL (#3952)
2023-02-14 - 0a22e1c8e - Fix Elasticsearch index creation in functional tests (#3954)
2023-02-14 - d353dcc43 - Emit inordered buffered events metric (#3949)
2023-02-14 - 0b698c1a0 - Wrap user's visibility query string in parenthesis (#3956)
2023-02-14 - f4e4ea2a8 - Remove isRecordValid check from ES visibility store (#3948)
2023-02-14 - 099cef3e2 - Support CountWorkflowExecutions for SQL DB (#3955)
2023-02-14 - 5d6d93c1c - Refactor SQL query converter (#3950)
2023-02-15 - c3578127d - Propagate SDK Metadata & add capability (#3944)
2023-02-15 - 8917ecc95 - Update Go SDK to v1.21.1 (#3960)
2023-02-15 - a23489b96 - Set namespace on API if not present (#3953)
2023-02-15 - 98a52d197 - Quick fix for generated mock file name (#3959)
2023-02-15 - 868b2d16c - Update IsAdvancedVisibilityActivity to include SQL DB (#3957)
2023-02-15 - 0bb688437 - Handle namespace not found case in redirection interceptor (#3947)
2023-02-15 - 2db41752a - Fix IWRR scheduler UpdateWeight test (#3961)
2023-02-16 - 7ab7e9c5d - Metering metadata propagation (#3965)
2023-02-16 - cbed0fe41 - Fix wrap user query string in parenthesis (#3967)
2023-02-16 - f9bd5b083 - DeleteExecutions workflow: pass nextPageToken with ContinueAsNewError (#3966)
2023-02-16 - 93fb0ec66 - Populate history size in describe workflow and list workflow response (#3964)
2023-02-16 - 06188f3bc - Suport dual visibility in visibility persistence checks (#3968)
2023-02-16 - b313b7f58 - Delete workflow execution from visibility store even if it was deleted from main store (#3962)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes a few minor bugs related to task processing and ES bulk processor. It also enables a security check feature by default to enforce the task token check to prohibit caller from modifiying namespace info in the task token.

All Changes

2022-12-29 - 6b2e44863 - Fix NPE in task channelWeightFn (#3766)
2023-01-12 - 23d8cf9a4 - Fix ES bulk processor commit timeout (#3696)
2023-01-12 - 7b64aa0a0 - Remove internal bulk processor retries (#3739)
2023-01-12 - c46fccbad - Capture task processing panic (#3779)
2023-01-12 - ea8af5056 - Disable eager activities for Python 0.1a1, 0.1a2, and0.1b1 too (#3793)
2023-01-12 - 5237e7f4a - Prioritize worker pool shutdown (#3795)
2023-01-12 - c22128835 - Capture panic in replication task processing (#3799)
2023-01-12 - a8da3587b - Fix recover call (#3804)
2023-01-12 - 5e784b48b - Drop task on serialization error (#3803)
2023-01-13 - 96d6b0064 - Turn on frontend.enableTokenNamespaceEnforcement by default
2023-01-13 - 95a40941b - Upgrade version to 1.19.1

Details about the v1.19.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.19.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

Schema changes

Before upgrading to your Temporal Cluster to release v1.19, you must upgrade your storage schema version to the following:

• MySQL schema version 1.9
• PostgreSQL schema version 1.9
• Cassandra schema version 1.7

Use the appropriate schema upgrade tool to upgrade your schema version.
For details, see https://docs.temporal.io/cluster-deployment-guide#upgrade-server.

Note that schema version is not the same as the storage type (database) version.

Golang version

• Upgraded golang to 1.19.

tctl

• tctl release with v1.17.2 (https://github.com/temporalio/tctl/releases/tag/v1.17.2).

Metric

A major refactoring has been done in the metrics code. Both metrics.Client and metrics.Scope are replaced by metrics.metricsHandler.

Customized metricsHandler is also supported by using temporal.WithCustomMetricsHandler .

Deprecated metrics

All metrics that have the suffix _per_tl are deprecated.

Improvements and fixes

Clean up mutable state when Workflow is closed

When the Workflow is closed, you can trigger a cleanup to remove Memos and Search Attributes from the mutable state. This helps reducing the size of the mutable state for closed Workflows. However, because the cleanup deletes data, it’s best to pair Advanced Visibility with Elasticsearch to ensure that Search Attributes can be retrieved. This feature is controlled by the dynamic config history.visibilityProcessorEnableCloseWorkflowCleanup.

Count limits for pending Child Workflows, Activities, Signals, and cancellation requests

This change adds a new system protection limit for the maximum number of pending Child Workflows, pending Activities, pending Signals to external Workflows, and pending requests to cancel external Workflows.
Operators can set different values for those new limits via the dynamic config. The default is currently 50,000 for each value.

limit.numPendingChildExecutions.error
limit.numPendingActivities.error
limit.numPendingSignals.error
limit.numPendingCancelRequests.error

Maximum number of concurrent pollers

The dynamic config frontend.namespaceCount controls how many concurrent pollers are allowed to connect to Temporal Server. Currently, this limit applies to the total number of pollers; that is, Activity plus Workflow pollers. In v1.19.0, this limit now applies to the number of pollers per type; that is, Activity and Workflow pollers won’t compete with each other for a connection to Temporal Server.

Batch deletion

The batch delete operation is now support in the batch operation API. Use BatchOperationDeletion option in the StartBatchOperationRequest for a batch delete operation.

History task processing

Default implementation

• Host-level Task scheduler is now enabled by default, meaning the default value of history.[timer|transfer|visibility]ProcessorEnablePriorityTaskScheduler is true.
• Multi-cursor queue implementation is now enabled by default, meaning the default value of history.[timer|transfer|visibility]ProcessorEnableMultiCursor is true.

Rate limiting

• Added priority rate limiter for Task processing. By default this rate limiter is disabled but can be enabled by setting history.taskSchedulerEnableRateLimiter to true. The rate limiter should be enabled only when multi-cursor queue implementation is enabled.
• The rate is controlled by history.taskSchedulerMaxQPS for per-host limit and history.taskSchedulerNamespaceMaxQPS for per-Namespace limit. All queues (timer, transfer, visibility) share the same limit. Please note that deletion Tasks have lower priority when consuming rate limit tokens and do not count against the per-Namespace limit.
• The default value for the preceding two configurations is 0, which means they will fall back to and use the value specified by history.persistenceMaxQPS and history.persistenceNamespaceMaxQPS.

Metrics

• task_latency_processing_nouserlatency, task_latency_userlatency, task_latency_nouserlatency, and task_latency_queue_nouserlatency are removed.
• New task_latency_user metric, which captures the latency for acquiring Workflow lock in a single Task Execution attempt.

History Scavenger

A retention validation on workflow has been added in the history scavenger. This feature is enabled by fault with worker.historyScannerVerifyRetention. A default grace period on this retention validation is 90 days + namespace retention time. The grace period is configurable by worker.executionDataDurationBuffer.

All changes

2022-09-19 - 9703d3363 - Post-release: bump version and upgrade dependencies (#3408)
2022-09-19 - 91c31f1b6 - Fix merge map of payload (#3412)
2022-09-19 - 1088382bd - Fix task reschedule policy (#3413)
2022-09-20 - c672c37c0 - Adds a retryable error for when we try to delete open executions (#3405)
2022-09-20 - d890a2f17 - Propagate CloseVisibilityTaskId to DeleteExecutionVisibilityTask (#3404)
2022-09-20 - be727b37a - Retry attempts to delete open visibility records (#3402)
2022-09-21 - 8f24d1fcb - Revert "Retry attempts to delete open visibility records" (#3420)
2022-09-21 - 9dfdf757a - Rename parameters of MergeMapOfPayload (#3418)
2022-09-22 - ff40b89e1 - Retry attempts to delete open visibility records (#3402) (#3421)
2022-09-23 - c6fb6b833 - Fixes an issue where last-heartbeat time was set to the first event's timestamp (#3361)
2022-09-26 - f4af2d5ff - Fix list batch operation to include division (#3431)
2022-09-27 - b4b61ff39 - Reorder grpc interceptors (#3423)
2022-09-27 - 85f400e57 - Add postgres es development script (#3429)
2022-09-27 - e3e1ccea6 - Retry attempts to delete open workflow executions (#3411)
2022-09-27 - 769b8658f - Add cluster ID into ringpop app (#3415)
2022-09-27 - 6530847a2 - Use original branch token instead of deserializing as branch info and then re-serializing (#3384)
2022-09-27 - d27ea8936 - Fix action metrics (#3434)
2022-09-28 - 8929def00 - Namespace replication for failover history (#3439)
2022-09-28 - c15047428 - Validate structured calendar specs and improve error messages (#3425)
2022-09-28 - e0bbabe21 - Per-namespace workers should only run on active cluster (#3426)
2022-09-29 - b497033e8 - Ensure urfave/cli accepts flag values with comma (#3440)
2022-09-29 - 363d4c0de - Log warning only when there is an error in SA size validation (#3443)
2022-09-29 - 97d0ec352 - Supply optionally configured workflow settings as hints (#3442)
2022-09-29 - 47bb15510 - Update dependencies and pin otel (#3444)
2022-09-29 - 38159152f - Implement GetAllHistoryTreeBranches for SQL persistence backends (#3438)
2022-09-30 - 3099274b4 - Fix reset workflow in replication reapply (#3449)
2022-10-03 - 175b916a4 - Use safer default TIMESTAMP for MySQL. (#3424)
2022-10-04 - a86d45534 - Index history size when workflow closes (#3451)
2022-10-05 - fc37cd4ba - Update ns version history in handover state (#3456)
2022-10-06 - 0ae4cc520 - Add config filter by task type (#3455)
2022-10-06 - 0df719d2d - Update replication use branch token (#3447)
2022-10-10 - fa51f5d24 - Move record child workflow completed to api package (#3350)
2022-10-10 - 62cd1439d - Move verify child workflow completion recorded to api package (#3351)
2022-10-10 - 71a1de37d - Use separate metric for resource exhausted error in task processing (#3463)
2022-10-10 - bc451dbac - Compare task type enum (#3464)
2022-10-10 - c6b472ef6 - Fix exclude tags with withTags method (#3466)
2022-10-10 - 158737abd - Remove old logic for checking workflow deletion task dependencies from delete_manager (#3427)
2022-10-10 - d33559a54 - Properly handle min task ID > max task ID case during shard re-balancing (#3470)
2022-10-11 - 0e70cf77f - Move get / poll mutable state to api package (#3467)
2022-10-11 - 7278168ad - Move describe workflow to api package (#3469)
2022-10-11 - 0dddd3cd6 - Fix timer task visibility timestamp for workflow refresh (#3460)
2022-10-11 - 429c0afc6 - Move replication APIs to api package (#3472)
2022-10-11 - d90f3abe0 - Move NDC logic to ndc package, move workflow reset to api package (#3465)
2022-10-11 - 740c7a3b1 - Move reapply events to api package (#3476)
2022-10-11 - d25265617 - Move remove signal from mutable state to api package (#3475)
2022-10-11 - b292c222d - Move delete workflow to api package (#3473)
2022-10-11 - 32ea0bf01 - Move refresh workflow to api package (#3477)
2022-10-12 - 16184da74 - Fix scheduled queue max read level update for single processor (#3474)
2022-10-12 - ceae24181 - Handle sync workflow state task in replication dlq handler (#3482)
2022-10-13 - bf8e1f170 - Bump UI to v2.7.0 (#3480)
2022-10-13 - 46dcb4fdb - Turns on the history scavenger for SQL backends (#3462)
2022-10-13 - 86fde8eff - Move query workflow to api package (#3486)
2022-10-13 - ef2140507 - Fix sanitize mutable state after replication (#3479)
2022-10-13 - 457eb05e6 - Clean up duplicate empty task id (#3490)
2022-10-13 - a0ea1772e - Add execution scavenger for retention (#3457)
2022-10-15 - a4fe3a7cf - Remove UI v1 from development environment (#3485)
2022-10-17 - dc84e87c5 - Update replication timestamp with no task (#3487)
2022-10-17 - 8e5ccb5a0 - schama -> schema (#3501)
2022-10-17 - b8d46a578 - Do not add version 0 to failover history (#3483)
2022-10-17 - cebbde9e5 - Sync proto API 1.12.0 (#3500)
2022-10-17 - 968c5765f - Create visibility GetWorkflowExecution API (#3488)
2022-10-18 - ecde5435b - Remove now parameter from task generator interface (#3478)
2022-10-18 - 9cb646be3 - Rewrite mysql PaginateBranchesFromHistoryTree query (#3509)
2022-10-18 - 8a7538ad1 - Warning log on new event during set workflow (#3508)
2022-10-19 - 8d645e85c - Increase visibility RPS and burst dynamic configs for integration tests (#3221)
2022-10-19 - 4b4493ba9 - Bump Golang version to 1.19 (#3507)
2022-10-19 - 3aff85ed6 - Fix scanner start dep (#3513)
2022-10-19 - c7713b14c - Support arbitrary history task category in SQL persistence (#3489)
2022-10-19 - d5edb0e61 - Plumb in timeouts instead of referencing the uninitialized execution info during workflow creation (#3512)
2022-10-20 - 6ee5ac524 - Check shard ownership by assertion (#3416)
2022-10-20 - c21ffad95 - initialize otelMetricsHandler logger (#3518)
2022-10-20 - af187569f - Simplify interface & remove unused vars (#3516)
2022-10-22 - 39ee37211 - Fix otel gauge metric type (#3525)
2022-10-24 - 827a59f82 - Support batch operation for delete workflow (#3497)
2022-10-25 - 42fca45ca - add metricstest package (#3528)
2022-10-26 - 5fac4d4eb - Add ArchiveExecutionTask to our set of tasks (#3524)
2022-10-26 - b42bc8593 - Add task version mismatch metrics (#3491)
2022-10-26 - 119478a59 - Change execution scavenger to call admin delete (#3526)
2022-10-26 - 275b62b64 - Fix replication task nil case (#3531)
2022-10-27 - 3a0d879ab - Remove SA and memo after close visibility task is completed (#3504)
2022-10-27 - ddc33bc3d - Skip archival if it was already done before deleting history data (#3523)
2022-10-28 - d4d7684e7 - Add service name tag to metrics emitted using metrics handler (#3537)
2022-10-31 - 4eec379f4 - Add retention validation in history scavenger (#3541)
2022-10-31 - 6d9c59528 - Increase timeouts in debug mode (#3542)
2022-10-31 - 88d34110d - Take rpc address from config for local cluster (#3546)
2022-10-31 - 1f697219e - Add deadlock detector (#3492)
2022-11-01 - 176cd42a9 - Add a new archiver client which doesn't retry (#3544)
2022-11-03 - 4a9d6ab38 - Queue processor handle shard ownership lost (#3553)
2022-11-03 - e2ef735f6 - Add a CanSkipVisibilityArchival flag to the CloseExecutionTask (#3529)
2022-11-03 - 7860ec059 - Fix shard info serialization (#3555)
2022-11-04 - 53000fc1d - Fix namespace handover log (#3559)
2022-11-04 - c2d5496e9 - Generate ArchiveExecutionTask instead of DeleteHistoryEventTask whenever an execution is closed (#3533)
2022-11-04 - 6721acec6 - Improve integration test framework (#3547)
2022-11-06 - b300010dd - Add some config values for the archival queue (#3554)
2022-11-07 - 02b8cca7b - Make shard more resilient to append history timeout (#3564)
2022-11-07 - 9fa1a9043 - Replace metrics client/scope with MetricsHandler (#3561)
2022-11-08 - bccf128f9 - Handle namespace not found in replication (#3557)
2022-11-08 - bbd386e03 - Add jitter for retention timer (#3569)
2022-11-09 - 86833988a - Fix task serialization in persistence tests (#3576)
2022-11-10 - 44154d324 - bump ui to v2.8.3 (#3573)
2022-11-11 - 5e466c30e - Fix metrics tag after metrics refactoring (#3579)
2022-11-14 - 574aee6b4 - Move queue metrics to metrics package (#3587)
2022-11-14 - 50936e1d5 - Add a tool to limit the number of pending child workflows (#3575)
2022-11-14 - ef05ac8bd - Fix scheduled queue lookahead (#3563)
2022-11-14 - 3a08eed68 - Limit Cassandra memory appetite to 1Gb in development configuration (#3581)
2022-11-14 - 76b58a74e - Fix resilient shard (#3584)
2022-11-14 - b74b0e3aa - Upgrade dev grafana version used in docker (#3593)
2022-11-14 - dde2afb3c - Update api and sdk references (#3595)
2022-11-14 - 4f2b98e89 - Wrap gocql.Iter with custom wrapper (#3577)
2022-11-14 - e7f99d074 - Fix history scavenger bug on delete mutable state (#3588)
2022-11-15 - c82d43c6d - Return an error whenever we exceed the number of pending child executions (#3586)
2022-11-15 - c14f64346 - Fix visibility queue log message and level (#3600)
2022-11-15 - 25e4a8fb3 - Don't log a warning when the number of child workflows is high (#3598)
2022-11-15 - 02b913fd4 - Clean up task latency metrics (#3510)
2022-11-16 - 3d32ebc82 - Add tdbg to goreleaser (#3609)
2022-11-16 - e96c17ae4 - Initialize new branch token based on replication target (#3604)
2022-11-17 - be30bab1a - Disable eager activity dispatch for buggy clients (#3610)
2022-11-17 - ee1abe1c7 - Fix scheduled task precision (#3591)
2022-11-18 - d34998708 - Add request validation on batch APIs (#3621)
2022-11-18 - be7daec26 - Add limits on other workflow fields (#3599)
2022-11-18 - e26ea3893 - Retry CreateIndex in integration tests request upon failure (#3618)
2022-11-18 - e4d1b98ad - Add methods to check workflow size violations for several other fields (#3602)
2022-11-18 - 81b79a73d - Change max namespace count limit to per poller type (#3608)
2022-11-18 - 66db3aebe - Remember retention timer task state in memory for retry (#3620)
2022-11-18 - 21118ed8b - Enforce the per-workflow pending activity constraint (#3611)
2022-11-18 - b13ed161f - Add flag to enable workflow closed clean up (#3603)
2022-11-18 - 4483f1023 - Handle unknown cluster during replication (#3619)
2022-11-18 - 0c735e496 - Fix function name typos (#3623)
2022-11-18 - df160db9f - Enforce the pending cancel request limit (#3615)
2022-11-18 - 32212eabd - Task scheduler rate limiter (#3606)
2022-11-18 - 91a67f41e - Enforce the per-workflow pending signal limit (#3617)
2022-11-18 - 954137483 - Keep multi cursor default reader always running (#3626)
2022-11-18 - b43ca639a - Fix immediate task fire time equality check (#3628)
2022-11-21 - 86631e9ce - Fix typo in dynamic config for VisibilityProcessorEnsureCloseBeforeDelete (#3632)
2022-11-21 - ded2f6761 - Improve test log format (#3634)
2022-11-22 - 3520803e2 - Extract duplicated getTemporalPackageDir into common function (#3637)
2022-11-22 - b0647ae55 - Randomization and configuration for scheduler throttle duration (#3636)
2022-11-22 - 13fa5f7bc - Handle unknown cluster in replication (#3639)
2022-11-22 - 473d48772 - Revert change 'De-duplicate reapply event' (#3640)
2022-11-22 - e40a1df2e - Enable history scavenger to scan expired workflow (#3643)
2022-11-22 - 13047df9b - Fix persistence GetHistoryTask (#3645)
2022-11-22 - 17b121a00 - Fix fail workflow batch id (#3646)
2022-11-22 - a4002ab04 - Fix pending tasks metric (#3648)
2022-11-22 - cd1872545 - Fix bug in GetRepoRootDirectory code (#3644)
2022-11-23 - d11074d0b - Make replication task reading logic more resilient (#3596)
2022-11-23 - d54cd1df9 - Simplify ServerOptions applyFunc (#3649)
2022-11-23 - 788b2b76a - Use fx group for deadlock detector (#3650)
2022-11-23 - 5658d5e79 - Disable visibility verify close before delete (#3651)
2022-11-23 - 9bbaa2731 - Handle unknown cluster with a unique string (#3652)
2022-11-23 - 5dd8cddb1 - Log when force replicate failed (#3653)
2022-11-23 - f46f2a466 - Fix one metrics name (#3654)
2022-11-23 - 306797b3b - Update persistence error metrics (#3655)
2022-11-24 - b717bd530 - Enforce min interval for continue as new runs (#3659)
2022-11-25 - 7a2fab580 - Fix a flaky persistence integration test (#3664)
2022-11-25 - 53f461577 - Enable host level pool and multi-cursor by default (#3660)
2022-11-28 - 001ec0e6a - Consolidate service name constants (#3656)
2022-11-28 - c2f21fe58 - Fix grpc listener initialization (#3657)
2022-11-28 - 0547245e7 - Set default catchup window eagerly (#3658)
2022-11-28 - 82cbe647a - make some s3 utility functions public (#3668)
2022-11-28 - 997434cc5 - Fix replication task execution latency (#3671)
2022-11-28 - b3812938f - Do not panic if passive processor is missing (#3672)
2022-11-28 - 362987f9b - Return an error when we fail to instantiate the server (#3642)
2022-11-28 - 937c38f8e - Update workflow state validator (#3673)
2022-11-28 - ee4648e43 - Set limit on DescribeSchedule query/signal loop (#3669)
2022-11-28 - 318749b75 - Don't wake up scheduler workflow when paused (#3670)
2022-11-29 - f38961142 - Fix defer not able to capture return error issue (#3675)
2022-11-29 - f6a340d23 - Fix slice pending task count (#3676)
2022-11-30 - 0101924db - Improve execution scavenger (#3674)
2022-12-01 - ff3f95cd3 - Remove empty visibility tag (#3681)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.19.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes minor bugs in history scavenger, gocql client recovery logic, and internal queue processing.

All Changes

2022-11-15 - 00bb51389 - Update version to 1.18.5
2022-11-15 - f103b1920 - Fix history scavenger bug on delete mutable state (#3588)
2022-11-15 - ec332ef75 - Wrap gocql.Iter with custom wrapper (#3577)
2022-11-15 - 077c5e0d7 - Fix resilient shard (#3584)
2022-11-15 - 0f68d2687 - Queue processor handle shard ownership lost (#3553)

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.5)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes a few minor bugs and upgrades the gocql dependency. It also upgrades the expat package in the admin-tools Docker image for CVE-2022-40674 (this doesn't affect the Temporal server itself).

All Changes

2022-10-27 - 0616c5c89 - Port util.CloneMapNonNil to release branch (#3530)
2022-10-31 - 327d46e45 - Upgrade gocql dependency to v1.2.1
2022-10-31 - 69e2bb949 - Change execution scavenger to call admin delete (#3526)
2022-10-31 - c6cfe6d52 - Fix replication task nil case (#3531)
2022-10-31 - 26782e9f5 - Add retention validation in history scavenger (#3541)
2022-10-31 - fe6a873b1 - Take rpc address from config for local cluster (#3546)
2022-10-31 - 03d6c0336 - Update version to 1.18.4

In docker-builds:

2022-10-31 - 85d405a - Use larger GitHub-hosted runners for Docker builds (# 74)
2022-10-31 - e85d0cc - Upgrade expat in admin-tools (# 75)
2022-10-31 - 07e4b80 - Update temporal submodule for branch release/v1.18.x

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.4)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release fixes a bug that was introduced in 1.18.2 that would prevent the server from starting in a single-node configuration.

All Changes

2022-10-19 - 570909087 - Fix scanner start dep (#3513)
2022-10-20 - c0eb02c6c - Update version to 1.18.3

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.3)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

This release has a bug that prevents server startup when running in a single-node configuration. Please use v1.18.3 instead.

Release Highlights

This patch releases fixes a few small bugs in namespace migration.
It also has two changes to scavengers: the history scavenger is now enabled on SQL persistence, and the execution scavenger now finds mutable state that's past its retention timeout.

All Changes

2022-10-17 - 1b8e28f64 - Implement GetAllHistoryTreeBranches for SQL persistence backends (#3438)
2022-10-17 - 2127976bf - Properly handle min task ID > max task ID case during shard re-balancing (#3470)
2022-10-17 - d602bff22 - Fix timer task visibility timestamp for workflow refresh (#3460)
2022-10-17 - 8cd481ad7 - Fix scheduled queue max read level update for single processor (#3474)
2022-10-17 - 45bd55d02 - Turns on the history scavenger for SQL backends (#3462)
2022-10-17 - bb7b1f432 - Fix sanitize mutable state after replication (#3479)
2022-10-17 - ff47b2c3c - Add execution scavenger for retention (#3457)
2022-10-17 - 78366f1e5 - Update replication timestamp with no task (#3487)
2022-10-17 - b57b9307b - Do not add version 0 to failover history (#3483)
2022-10-18 - 9a427c26a - Rewrite mysql PaginateBranchesFromHistoryTree query (#3509)
2022-10-18 - 07ab3e111 - Warning log on new event during set workflow (#3508)
2022-10-18 - 0aa594887 - Update version to 1.18.2

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This patch release fixes a few minor issues with search attribute/memo upsert, task rescheduling, schedule api validation, replication, and batch operations. We recommend that everyone upgrade to it.

All Changes

2022-09-28 - 060353584 - Fix merge map of payload (#3412)
2022-09-28 - ac2593cae - Fix task reschedule policy (#3413)
2022-09-28 - ca5d0fdb5 - Reorder grpc interceptors (#3423)
2022-09-28 - 88d6fc982 - Validate structured calendar specs and improve error messages (#3425)
2022-09-28 - 02e2edf11 - Per-namespace workers should only run on active cluster (#3426)
2022-09-28 - 3c0105b2a - Fix list batch operation to include division (#3431)
2022-09-28 - adc4cc82c - Fix action metrics (#3434)
2022-09-28 - 70ca71baf - Namespace replication for failover history (#3439)
2022-09-29 - f84be211d - Log warning only when there is an error in SA size validation (#3443)
2022-10-10 - 082713b14 - Fix reset workflow in replication reapply (#3449)
2022-10-10 - 1e462372a - Add config filter by task type (#3455)
2022-10-10 - 2f6df416f - Use separate metric for resource exhausted error in task processing (#3463)
2022-10-10 - e2e7474c8 - Update replication use branch token (#3447)
2022-10-10 - f089fda9f - Update ns version history in handover state (#3456)
2022-10-10 - cb2f57453 - Compare task type enum (#3464)
2022-10-10 - 78a5e36e7 - Fix exclude tags with withTags method (#3466)
2022-10-10 - 1a6bae268 - Prepare 1.18.1 release

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

Upgrade action item summary

• Before upgrade
  • If you’re using ES v6: do not deploy this version until after moving to ES v7 or later
  • If you’re using ES v7+: upgrade schema to include new builtin search attributes (see Schedules section below)
  • If you used the experimental Schedules feature in 1.17: delete all schedules and recreate them after upgrading
• After upgrade
  • Change dynamic config matching.useOldRouting to false on all nodes as close in time as possible. This will cause temporary matching disruption during the propagation.
  • Consider enabling host-level worker pool and multi-cursor task processing (see Task processing section below)
  • Consider setting persistence rate limits (see Persistence rate limiting section below)

Elasticsearch v6 support is removed

Elasticsearch v7 became the default supported version in 1.12.0 release. In 1.18.0, Elasticsearch v6 support is completely removed. If you are still using Elasticsearch v6, don't upgrade to 1.18.0. Upgrade Elasticsearch first following the migration guide.

Along with v6 support removal, we added Elasticsearch v8 support. Elasticsearch v8 doesn’t have breaking changes that affect Temporal.

Batch API

This release introduces new batch operation APIs in the Frontend Service. The feature is enabled by default.

• StartBatchOperation: Start a batch operation.
• StopBatchOperation: Stop a running batch operation.
• DescribeBatchOperation: Get detail information about a batch operation.
• ListBatchOperations: List all the batch operations.

By default, the APIs support one concurrent operation per Namespace. This value is configurable by dynamic config: frontend.MaxConcurrentBatchOperationPerNamespace.

Use DescribeBatchOperation or metrics batcher_processor_requests and batcher_processor_errors to monitor the progress of batch operations.

Cluster API

The following Cluster APIs moved from the admin service to the operator service. Those APIs will be deprecated in the admin service in a future release.

• AddOrUpdateRemoteCluster: Add or update a connection configuration to a remote Cluster.
• RemoveRemoteCluster: Remove a connection to a remote Cluster.
• ListClusters: List the configuration of all connected Clusters.

Task processing

Host level task scheduler

• Host level task scheduler now uses separate task channels for each namespace for better resource isolation.
• Deprecated the dynamic config history.[timer|transfer|visibility]TaskHighPriorityRPS.

Multi-cursor queue

New multi-cursor queue implementation for better resource isolation and handling cases not covered by host-level task scheduler: too many pending tasks, stuck queue ack level, etc.

• The new implementation can be enabled by setting the value of dynamic config history.[timer|transfer|visibility]ProcessorEnableMultiCursor to true.
• The host-level worker pool for the corresponding queue should also be enabled, otherwise the above dynamic config won’t take effect. This can be done by setting history.[timer|transfer|visibility]ProcessorEnablePriorityTaskScheduler to true.

Task Loading

• The dynamic config history.[timer|transfer|visibility]ProcessorMaxPollHostRPS can be used to limit the throughput of the queue processor. This is very useful for recovering from a persistence outage which leads to a large task queue backlog in persistence. Set the value to a small number and gradually increase to ensure a smooth draining of the backlog. By default, value for this config is 0 and will fallback to 30% of the history.persistenceMaxQPS for transfer and timer queue and 15% for visibility queue.

Metrics

New task processing related metrics are added for better visibility.

• task_latency_load: measures the duration from task generation to task loading (task schedule to start latency for persistence queue).
• task_latency_schedule: measures the duration from task submission (to task scheduler) to processing (task schedule to start latency for in-memory queue).
• queue_latency_schedule: measures the time it takes to schedule 100 tasks in one task channel in the host level task scheduler. If there are less than 100 tasks in the task channel for 30s, the latency will be scaled to 100 tasks upon emission. NOTE: this is still an experimental metric and is subject to change.

Retry behavior

Consolidated retry logic for APIs calls to two places: service handler (by an interceptor) and service client for calling other temporal services or persistence. Limited retry maximum attempts and removed retry logic in other places to avoid potential retry storm.

Persistence rate limiting

Persistence layer rate limiter prioritizes user requests (e.g. start workflow, signal workflow, etc.) over system background requests (e.g. task loading, requests incurred by task processing, replication, etc.)

Persistence max QPS for each namespace can be set by tuning the dynamic config [frontend|history|matching|worker].persistenceNamespaceMaxQPS. By default, this value is 0 and will fallback to the overall host persistence QPS set by [frontend|history|matching|worker].persistenceMaxQPS.

Persistence metrics persistence_requests, persistence_latency and persistence_error* now also contains namespace tag for better observability.

UpsertMemo

This release adds UpsertMemo to modify existing workflow memo. It works similarly to UpsertSearchAttributes. However, on the visibility side, the memo is updated only when used with advanced visibility (i.e. Elasticsearch). When used with standard visibility (SQL databases), the memo is not being updated currently, and the workflow memo can be retrieved correctly only from the mutable state (that is, you can call DescribeWorkflowExecution to retrieve it).

Matching routing

Task queues are now distributed among matching nodes based on namespace and type, for better load balancing. This change is disabled by default to avoid disruption during the upgrade. After upgrading to 1.18, you should make a dynamic config change on all nodes simultaneously to set matching.useOldRouting to false. You can use the following snippet:

# Remove this after upgrading to 1.19:
matching.useOldRouting:
- value: false

This change will cause a short disruption to Task dispatch as nodes reload the dynamic config and then Task Queues get moved between matching nodes. You’ll also see some persistence errors in logs for a few minutes.

We plan to make this routing the default in 1.19, so if you don’t make the dynamic config change, it’ll happen during the next upgrade.

Schedules

A few incompatible changes were made to the Schedules feature, which was introduced as experimental in 1.17. If you have created any Schedules, you should delete them before upgrading to this release, and recreate them after the upgrade.

The Schedule feature is now enabled by default.

If you're using Advanced Visibility (i.e. Elasticsearch), Schedules don’t appear in Workflow lists anymore. If you’re not using Elasticsearch, Schedules continue to appear in Workflow lists for now.

There is one new builtin search attribute to support Schedule visibility. You can add it to Elasticsearch by running the upgrade script from the server repo. See the top of the script for environment variables to use to point it at your Elasticsearch. You can do this before or after the upgrade, but before creating any Schedules. (If you didn’t perform the v2 upgrade when upgrading to 1.17, which also added search attributes for Schedules, also do that now.)

schema/elasticsearch/visibility/versioned/v3/upgrade.sh

Frontend connections

Update: these release notes previously described a new method of making internal frontend connections. This new method works in many, but not all, server configurations (specifically related to mTLS and custom authorizers). To make it work for all configurations, we're going to make some more changes to the feature. In the meantime, there's no need to change any configuration. (If you've already made the config change that was suggested here and it's working, there's also no need to change it back.)

Maximum retention

The maximum Namespace retention limit of 30 days is removed. Namespaces now can use any retention as needed, as long as the persistence has enough capacity for the storage.

Dynamic config interface

The dynamicconfig.Client interface was changed and simplified.

If you’re using Temporal with our pre-build container images or binaries, there’s nothing to do.

If you’re building your own Temporal binary and only refer to dynamicconfig.Client or dynamicconfig.NewFileBasedClient in your ServerOptions, you should be able to rebuild with no code changes.

If you’ve written a custom dynamic config implementation, you’ll need to change it to the new interface. This should be pretty straightforward, but if you have any questions, please contact us; we can help.

All changes

2022-06-21 - 231655c69 - Rename queryTermination to queryCompletion (#3000)
2022-06-22 - 73881a375 - Prepare for 1.18 release (#3009)
2022-06-22 - f86b8d2c5 - Per-service fx-ified OTEL tracing (#2896)
2022-06-22 - c7d831b50 - Explicitly specify timezone for TIMESTAMP values (#3012)
2022-06-22 - 958bde7c5 - Update mysql image version to support arm64 (#3013)
2022-06-23 - 6cab7e5df - Add developer doc on tracing (#3014)
2022-06-23 - 0c0d369f5 - Some instrumentation tips for tracing (#3017)
2022-06-24 - abdc156ce - Update dependencies and checkers (#3016)
2022-06-25 - a431b8a6d - Tracing docs typo (#3018)
2022-06-27 - 8004b1675 - Update Cassandra install guide for MacBook with ARM chip (#3026)
2022-06-27 - 2d0071dde - Don't check delete workflow execution transfer task version (#3021)
2022-06-27 - 187e1ae81 - Prioritize task processor shutdown (#3024)
2022-06-27 - 3354e3510 - Multi-cursor components: queue slice (#2996)
2022-06-27 - 69ae0f4e0 - Release shard lock earlier during delete workflow execution (#3028)
2022-06-28 - e52d10e96 - Adjust shard acquire expiration (#3032)
2022-06-28 - b262b529e - Cancel activity when eager execution and request cancellation are in the same WFT (#3029)
2022-06-29 - 8a38825f8 - Switch mock sdk uses to gomock-based sdk mocks (#3030)
2022-06-29 - 6c0f53adc - Perf optimize interleaved weighted round robin scheduler (#3034)
2022-06-29 - d09213a4b - Proto definition for multi cursor queue persistence states (#3035)
2022-06-29 - 23a6bec73 - Rename task predicates definition (#3037)
2022-06-29 - 30a89202a - Generate most of the gRPC wrappers in client (#3001)
2022-06-30 - 1b330e115 - Run task rescheduler in background (#3038)
2022-06-30 - e7c02d9ad - Make queue max read level always exclusive (#3036)
2022-06-30 - ce47ea923 - Revert "Adjust shard acquire expiration (#3032)" (#3041)
2022-06-30 - 5d9c892bb - Unit tests for scheduler workflow + bug fixes (#3020)
2022-06-30 - c7866bba4 - add support for multiple pragma statements and driver syntax (#3031)
2022-06-30 - d968a39db - Remove reporter and userscope interfaces (#3044)
2022-06-30 - 2b31d0dd8 - Update backfill history in sync workflow state (#3042)
2022-07-01 - 2d7072e16 - Clean up history branch ancestor operation (#3046)
2022-07-01 - 08ce28c4a - Move external-mocks to go:generate (#3039)
2022-07-01 - da4de6583 - Implement foundation of versioning API (#2980)
2022-07-01 - 8d570d685 - Fix potential npe in scheduler workflow update signal (#3049)
2022-07-01 - de1164621 - Fix error shadowing in standby executors (#3053)
2022-07-01 - ca22f60ef - Improve error handling for standby transfer task verification (#3050)
2022-07-05 - 72ff3dac9 - Add convertion func for persistence multi-cursor queue state (#3051)
2022-07-05 - d02851fac - Handle workflow already completed when verifying first workflow task scheduled (#3061)
2022-07-05 - 7c57f96f1 - Remove unnecessary debug log (#3063)
2022-07-06 - bab298f4b - Fix namespace failover callback (#3066)
2022-07-06 - 7bb531926 - Do not hold lock when running namespace change callback (#3067)
2022-07-06 - 7d6be15cd - Move replication task cleanup to processor manager (#3068)
2022-07-07 - 1f4c05fc9 - Resource aware operation retry (#3069)
2022-07-07 - 05575d9b6 - Add ability to clear search attributes (#3047)
2022-07-07 - 080aed3de - Fix schedule jitter calculation bug (#3059)
2022-07-07 - 7cde1d013 - Do not double execute close execution task if visibility archival is disabled (#3072)
2022-07-08 - 50a2f3aff - Rename queue processor (#3075)
2022-07-10 - d138aefaa - Reorder frontend rate limiter (#3080)
2022-07-10 - 92c866c5e - Remove rds iam auth plugin (#3079)
2022-07-11 - 987fac1ad - Get and set queue states in shard info (#3074)
2022-07-11 - 9d632d7af - Change a bunch of code to use generics (#3078)
2022-07-11 - 29f0ab2b4 - Move search attributes empty slice check to DecodeValue (#3073)
2022-07-11 - cbf6cb167 - Fix merge collision between #3078 and #3074 (#3084)
2022-07-12 - 4f016648d - Add protection against system namespace delete (#3087)
2022-07-13 - 04988f6cb - Add optional --defaultdb flag to sql tool (#2950)
2022-07-13 - 9a9798b6f - Support history task cancellation (#3052)
2022-07-13 - e06979960 - Do fx logging properly (#3098)
2022-07-13 - 76e123cf0 - Use backoff.ThrottleRetry in ringpop (#3099)
2022-07-13 - 1af66ac71 - Start services in parallel (#3100)
2022-07-14 - e969dd8bb - change custom datastore config map type (#3082)
2022-07-14 - 8d536bce6 - Expand comment on shard context acquire behavior (#3095)
2022-07-14 - a1ac36faf - Refactor: rename termCh to completionCh (#3101)
2022-07-14 - 7f1627666 - Fix deleteexecutions iterator (#3083)
2022-07-14 - 1b95eafc8 - Update DeleteWorkflowExecution description (#3048)
2022-07-14 - 1ef4de4c3 - Wait to acquire lease in matchingEngine (#3033)
2022-07-15 - 8d82a8cee - Make staticcheck great again (#3103)
2022-07-15 - ae5cad888 - Fix workflow ID reuse when running on ScyllaDB (#3027)
2022-07-15 - 5425bfa1f - Use shard lifecycle context for persistence operations (#3096)
2022-07-15 - db09e2801 - Consume api v1.10.0 (WorkflowUpdate) (#3102)
2022-07-18 - d0ebe682d - Improve verification APIs error handling (#3064)
2022-07-18 - e55f32836 - Emit metric for activity eager execution (#3110)
2022-07-18 - 0b9fbba10 - Fix concurrency of shard acquisition and flaky test (#3097)
2022-07-18 - 44db941d5 - Make shard.GetEngine always take Context (#3112)
2022-07-18 - 2cf6ec2ac - Make TransientWorkflowTaskInfo more general (#3109)
2022-07-18 - 881f37d12 - Treat protoc warnings as errors (#3113)
2022-07-18 - 378759cd0 - Bump UI to v2.2.3 (#3114)
2022-07-18 - d85d67845 - Rename workflow_task_info to transient_workflow_task_info (#3115)
2022-07-18 - 219240a91 - Added history of active clusters to NamespaceDetail (#3106)
2022-07-20 - 50bc62f4a - Updated NamespaceReplicationQueue API to use specific types instead of interface{}, since it only works with one type (#3121)
2022-07-20 - ba585a97b - Create binary tool for debugging Temporal (#2932)
2022-07-20 - 2617e6570 - Don't count PermissionDenied as ServiceFailure (#3119)
2022-07-21 - e3aeee453 - Fix operation tag for admin service APIs (#3125)
2022-07-21 - eea8343d4 - Fix metrics for OperatorService (#3129)
2022-07-22 - 85a5f1e64 - Fix queue rate limiting busy loop (#3111)
2022-07-22 - d56e1d6fe - Update sql tools readme (#3090)
2022-07-22 - b8ca07b3d - Fix task rate limiting (#3137)
2022-07-25 - 54bf185e3 - Fix timer metric bucket (#3136)
2022-07-25 - 9d9536129 - Randomize shard ownership assertion starting position (#3058)
2022-07-25 - 0dfc7f945 - Use default consistency level for visibility DeleteWorkflowExecution (#3132)
2022-07-25 - 16df6dd9b - Add config to disable ORDER BY clause (#3128)
2022-07-26 - e8576590f - Consolidate rpc client metrics logic (#3147)
2022-07-26 - c2a3a2bdf - Add fuzz-style test for shard controller (#3124)
2022-07-26 - 8309427c9 - Multi-cursor executable reader implementation (#3118)
2022-07-27 - 8c2482557 - Bring back the task complete loop (#3151)
2022-07-27 - 942041fc8 - Fix task rescheduler flaky test (#3152)
2022-07-27 - 77aaf844e - Remove RunId from visibility query sort for ES 7 (#3140)
2022-07-27 - 7f480149b - Core persistence priority rate limiting (#3139)
2022-07-27 - 87847e065 - Run one per-ns worker per namespace instead of namespace × component (#3116)
2022-07-27 - b43f51690 - Multi-cursor: reader group (#3153)
2022-07-28 - 234296415 - Don't dispatch expired tasks from taskReader buffer (#3161)
2022-07-28 - c9b9c674a - Implement tdbg AdminDeleteWorkflow (#3157)
2022-07-28 - 61108e71e - Add metrics to detect no poller issue (#3146)
2022-07-28 - e96d485c2 - Update shard replication timestamp with max generated task (#3158)
2022-07-28 - 4392952a6 - Config worker use public frontend endpoint (#3162)
2022-07-31 - 4d56df95e - add address translation to Cassandra persistence (#3076)
2022-08-01 - c0fbdf96c - Adds ability to specify sanitizeOptions for prometheus metrics (#3170)
2022-08-02 - 66852ea94 - Fix tally timer metric (#3173)
2022-08-02 - c5aad0192 - Multi-cursor: queue implementation (#3167)
2022-08-03 - 3ab0bae5c - Add admin workflow rebuild command (#3175)
2022-08-03 - ab56cdfd0 - Add version check for delete workflow transfer task (#3159)
2022-08-03 - 04e4c6520 - Fix Cassandra fixed address translator unit test (#3179)
2022-08-03 - 98ae4978b - Code restructure (#3181)
2022-08-04 - 3182dc2cd - Define & use shard controller interface (#3180)
2022-08-04 - 0b4bf4792 - Simplify system retry logic: Part 1 (#3172)
2022-08-04 - 9a2fde6a9 - Multi cursor: wire up implementation (#3176)
2022-08-04 - cae7cbab7 - Remove namespace max retention (#3148)
2022-08-04 - a8fb66afb - Batch operation per namespace worker (#3094)
2022-08-04 - bed5d1fcc - Make branch token initialization part of data persistence interface (#3187)
2022-08-05 - ae435989c - additional history scavenger heartbeat (#3186)
2022-08-05 - 68b3d7e9b - Add missing NewHistoryBranch (#3188)
2022-08-05 - 9e0b2a109 - Fix task predicates operation (#3190)
2022-08-05 - 7825eadfc - Refactor transient workflow task creation (#3189)
2022-08-05 - 875a9763a - Use detached context for shard context operations (#3194)
2022-08-08 - bd68cc000 - Fix queue processor throttling logic (#3195)
2022-08-08 - ef19550d4 - Simplify system retry logic: Part 2 (#3191)
2022-08-08 - de694c6d2 - Use separate mutex for shard context state (#3108)
2022-08-08 - b27975deb - Hide scheduler workflows with new search attribute (#3123)
2022-08-09 - 206100b41 - Add DeleteNamespace integration test (#3206)
2022-08-10 - 42168bf6d - Push serialization of history branch from history manager down to history store (#3205)
2022-08-10 - 9be1e188a - Only load complete event if need to report to parent (#3207)
2022-08-10 - 3c6abda58 - Only load complete event for standby close task if needed (#3209)
2022-08-10 - c8c2777f1 - Use a prefix for scheduler workflow ids (#3201)
2022-08-10 - c5ccbd596 - Add read-only APIs that've been added since the original list was created (#3208)
2022-08-10 - aadbbacf1 - Default to unix time 0, instead of time.Time{} (#3213)
2022-08-11 - db5f14f3e - Fix shard close error type (#3215)
2022-08-11 - 6fe8f3eb7 - Fix inline history archival (#3216)
2022-08-11 - 55e50fbed - Improve task processing retry behavior (#3218)
2022-08-11 - cf74ca4c0 - Add comment for newDetachedContext (#3219)
2022-08-11 - 0f1b823f1 - Multi-cursor: metrics (#3198)
2022-08-12 - 85bb4e76a - Versioning metadata propagation between partitions (#3166)
2022-08-12 - 2f8487968 - Chain methods for setting retry policy parameters (#3220)
2022-08-15 - e9d2ba582 - remove maxshards, switch from random to sequential (#3200)
2022-08-15 - b2949909b - Upgrade gocql version to v1.2.0 (#3226)
2022-08-15 - 4873231b2 - Fix archival activities error handling (#3227)
2022-08-15 - 53d11aeac - Delegate branch token serialization/deserialization to store layer (#3225)
2022-08-15 - e52d5bb3e - Enhanced persistence priority rate limiting (#3163)
2022-08-15 - cd876eef1 - Fix failover queue shutdown (#3232)
2022-08-16 - cd9dec9b2 - Update go.temporal.io/sdk to v1.16.0 (#3231)
2022-08-16 - 298ccf99b - Use namespace and type for task queue routing (#3202)
2022-08-16 - 8c77bde29 - OTEL tracing updates (#3107)
2022-08-16 - f5730ce6d - Propagate original branch token to persistence history store layer (#3234)
2022-08-16 - f4d0290f2 - Structured defaults in dynamicconfig (#3199)
2022-08-16 - 0f99d82f9 - Remove GetRingpopChannel from integration tests (#3235)
2022-08-16 - 9af6e3c7f - Revert "OTEL tracing updates (#3107)" (#3236)
2022-08-16 - bcc440bca - Prevent concurrent workflow execution deletion (#3237)
2022-08-17 - 83aa3b713 - Address context.TODO in history service (#3242)
2022-08-17 - ef78ea97d - Completely move signal with start logic into its own package (#3182)
2022-08-18 - 9e0cc833d - Add EncodedFailureAttributes to supported capabilities (#3244)
2022-08-18 - 77be7a96c - Evenly distributed frontend global rps limit (#3251)
2022-08-19 - f960b347c - Switch to github.com/temporalio/tchannel-go (#3250)
2022-08-19 - 95e24d75d - Add support for ES8 (#3185)
2022-08-19 - 1b210d8ee - Remove support for ES6 (#3184)
2022-08-19 - 6688c2505 - Do not enqueue replication task to DLQ if shard shutdown (#3245)
2022-08-19 - 47c141bdb - Sync API and add dummy implementations of batch operations (#3249)
2022-08-19 - 2c40ed5bf - Make mocksdk/generate.sh compatible with macos (#3254)
2022-08-22 - 5f4f30d3e - Multi-cursor: fix scheduled task precision issue (#3255)
2022-08-22 - 9274c34bd - Add generic time-bound and space-bound flusher (#2970)
2022-08-22 - 8d696e889 - Small update to workflow resetter confusing error message (#3257)
2022-08-22 - 5f89f38a5 - Add resource exhausted cause persistence limit (#3260)
2022-08-23 - 86bda4bcb - Move client version check to SDK interceptor (#3262)
2022-08-23 - 780bb6f12 - Use separate task channel for different namespaces (#3252)
2022-08-24 - eba842b72 - Merge ES ClientV7 with base client interface (#3259)
2022-08-24 - 15e313e42 - Add --allow-no-auth flag to allow noopAuthorizer (#3263)
2022-08-24 - e08606c22 - Fix Makefile to match temporal-sql-tool's instructions (#3264)
2022-08-25 - c059c694b - Support upsert memo (#3091)
2022-08-25 - 6c638c09f - Shorten timeout used for recording task started (#3268)
2022-08-25 - de9757cea - Multi-cursor: reader watermark monitoring and mitigation (#3253)
2022-08-25 - e084babb1 - Task channel aware rescheduling (#3266)
2022-08-25 - 48a5af01f - Remove redundant error metrics (#3269)
2022-08-25 - 3c0f03f1f - Make task scheduler TrySubmit always non-blocking (#3267)
2022-08-29 - e35d6969f - Wire up batcher APIs (#3233)
2022-08-29 - 49b709827 - Send ringpopServiceResolver events on diffs instead of ringpop events (#3238)
2022-08-30 - cbfaefb71 - Add a unit test for future.Get behavior (#3278)
2022-08-30 - 3c8fa0cc3 - Init internal cluster client using membership info (#3160)
2022-08-30 - 94abd6c64 - Remove frontend handler retries (#3285)
2022-08-30 - 7c5967247 - Increase page size for namespace registry refresh (#3283)
2022-08-30 - bd4e42006 - Refactor dynamic config Client and Collection (#3271)
2022-08-30 - 04013e523 - Multi-cursor: queue reader priority rate limiting (#3273)
2022-08-31 - 878846150 - Add more queue and task processing metrics (#3274)
2022-08-31 - d1afec25e - Multi-cursor: slice count monitoring & mitigation (#3276)
2022-08-31 - 5ab8ab804 - Multi-cursor: pending task count monitoring & mitigation (#3275)
2022-09-01 - 948e303a1 - Multi-cursor: improve namespace failover (#3279)
2022-09-01 - 5ed995b61 - fix cassandra tool consistency readme to reflect the config default (#3290)
2022-09-01 - da4784f7a - Share grpc connections across sdk clients (#3239)
2022-09-01 - 27e8bcbc5 - Use custom grpc resolver for frontend connections (#3280)
2022-09-01 - 2bd34e0d3 - Enable schedules by default and add integration tests (#3289)
2022-09-01 - e12217b51 - Improve task scheduler namespace change callback (#3293)
2022-09-01 - 853269a0d - Multi-cursor: stop empty readers (#3294)
2022-09-01 - 6a354ee73 - Add metrics tags back to task executor response (#3298)
2022-09-02 - 331573389 - Multi-cursor: fix timer task precision (#3297)
2022-09-02 - e32488e7e - Multi-cursor: shrink slice predicate (#3291)
2022-09-02 - 1c02b2308 - Multi-cursor: fix task deletion (#3295)
2022-09-02 - ae3385196 - Slow down task scheduling upon resource exhausted (#3305)
2022-09-06 - aaafa05c0 - Move DeleteWorkflowExecution to workflow service (#3300)
2022-09-06 - c74881b91 - Improve batcher workflow metrics (#3306)
2022-09-06 - f97debb4e - Use 60 days as history scavenger min age (#3310)
2022-09-06 - e4bf183a6 - Multi-cursor: move alert deduping to queue monitor (#3315)
2022-09-06 - 2739f2fdc - Multi-cursor: slice predicate action (#3312)
2022-09-06 - 25c2594f0 - Set upsert memo capability (#3314)
2022-09-06 - ba13e7380 - Fix task executable metrics tags (#3307)
2022-09-06 - 1504c8bf0 - Multi-cursor: Improve slice count action (#3313)
2022-09-07 - e96b58c8e - Remove ES6 client (#3317)
2022-09-07 - a91898d6b - Fix describe completed batch operation counter (#3316)
2022-09-07 - 3c2ddeac3 - Clone request if search attributes are mapped (#3299)
2022-09-07 - c0577e094 - Revert 'Remove frontend handler retries (#3285)' (#3321)
2022-09-07 - 44ee560d5 - Move commonly used get & update function to api package (#3311)
2022-09-07 - dc328c7fe - Tweak schedule test timeouts (#3320)
2022-09-07 - 15cf86983 - Skip ratelimiting for some persistence APIs (#3322)
2022-09-07 - 163d94b57 - Move cluster management api to operator handler (#2985)
2022-09-07 - 22e082569 - Fix flaky buffered event test (#3318)
2022-09-07 - 8fa94ed3b - Simplify worker fx (#3324)
2022-09-07 - 8e64fa881 - Move start workflow to api package (#3326)
2022-09-07 - 0c4a0e7da - Move reset sticky task queue to api package (#3327)
2022-09-07 - 7cbff9e5d - Add refresh to force update dynamic rate limiter (#3335)
2022-09-07 - 0a042d188 - Increase default sendArchiveSignal timeout and make it configurable (#3325)
2022-09-07 - 34c219aff - Remove deprecated namespace fields (#3329)
2022-09-07 - 8251b5b5a - Update go.temporal.io/sdk to latest version (#3331)
2022-09-07 - 6d8c6a968 - Update base CI image version (#3334)
2022-09-07 - d2cc4a407 - Remove deprecated namespace fields from parentclosepolicy.Request (#3330)
2022-09-07 - 8db40a43e - Move record activity task started to api package (#3328)
2022-09-08 - 8be957ef0 - Fix snake case to camel case (#3339)
2022-09-08 - b21ac2a53 - Update Makefile to start server with --allow-no-auth (#3342)
2022-09-08 - 57b07b157 - Bypass namespace validator for DescribeNamespace API (#3336)
2022-09-08 - 4899a21ca - Move respond activity task completed to api package (#3333)
2022-09-08 - 72da3baee - Move respond activity task failed to api package (#3344)
2022-09-08 - d2a9fe9f5 - Move record activity task heartbeat to api package (#3346)
2022-09-08 - 978f9322b - Move request cancel workflow to api package (#3347)
2022-09-08 - 09c205176 - Cleanup leftover after "mutable state builder" to "mutable state" rename (#3343)
2022-09-08 - 8338211d4 - Move terminate workflow to api package (#3349)
2022-09-08 - cb3f1bc30 - Move respond activity task canceled to api package (#3345)
2022-09-08 - e80a691cc - Delay namespace deletion (#3341)
2022-09-08 - 9f93c4493 - Move signal workflow to api package (#3348)
2022-09-09 - 55c254317 - Return a "not found" error code for empty archival namespaces (#3340)
2022-09-09 - e5d81447e - Update Go SDK to v1.17.0 (#3354)
2022-09-09 - ec325c53d - Enforce memo size limit during StartWorkflow (#3352)
2022-09-09 - 9624e81a4 - Use multi-cursor queue state for workflow deletion check (#3332)
2022-09-09 - 4b2c62412 - Add namespace tag for persistence metrics (#3355)
2022-09-09 - 30bf25054 - Prioritize task loading persistence requests (#3217)
2022-09-09 - 2925d4ff8 - Add namespace tag for service client metrics (#3356)
2022-09-09 - 33263db1c - Fix memo/search attribute size validation when upserting (#3353)
2022-09-09 - 5389cd520 - Use memo for schedule list info (#3359)
2022-09-12 - f84e4c973 - Add separate metrics for memo size (#3365)
2022-09-12 - 642d035e1 - Multi-cursor: fix scheduled queue look-ahead (#3364)
2022-09-12 - 9493ec3f7 - Fail integration tests on shutdown errors (#3301)
2022-09-12 - b3897aa6b - Skip task namespaceNotActive check in single queue mode (#3369)
2022-09-12 - f50d84c14 - Attaches success/error statistics to the batch operation workflow's memo (#3360)
2022-09-12 - cb2cc6f76 - Implement new schedule API features (#3337)
2022-09-12 - 5e6b2d7dd - Remove unused history event check (#3373)
2022-09-13 - 877c2f526 - Update task processing dynamic config defaults (#3374)
2022-09-13 - b812e8ebe - Calculate schedule times in sideeffect (#3375)
2022-09-13 - 574228e8e - Exclude taskAlreadyStarted error from history client logging (#3376)
2022-09-13 - 81a92f144 - Use mock DB client in namespace unit test (#3363)
2022-09-13 - cf754414d - Move build proto dependencies to separate go.mod (#3377)
2022-09-14 - fcec8f99b - Update api version to v1.12.0 (#3379)
2022-09-14 - 6494f88bc - Allow NewHistoryBranch optionally specify branch id (#3386)
2022-09-14 - 57424088d - Retry after sdk post-Start errors in per-namespace worker (#3385)
2022-09-15 - bba148cf1 - Prevent scheduled task max read level update when shard status invalid (#3382)
2022-09-15 - 0a27533a6 - Make TestShardControllerFuzz less flaky (#3394)
2022-09-15 - 65dd13eb1 - Move notifyQueueProcessor to acquireShard (#3392)
2022-09-15 - 160ba6973 - Fix fail workflow due to attributes size violation (#3388)
2022-09-15 - 2ec223414 - Fix task schedule latency metric (#3395)
2022-09-15 - ce9a66fb5 - Fix timerGate race condition in rescheduler (#3396)
2022-09-15 - c91d596ba - Add CloseVisibilityTaskID to DeleteExecutionVisibilityTask (#3391)
2022-09-15 - f361405fd - Remove shardOwnershipLost error from fault injection execution store (#3399)
2022-09-16 - c9e8f4977 - Dynamic config (#3390)
2022-09-16 - 569433ffe - Revert "Remove unused history event check (#3373)" (#3400)
2022-09-16 - de91ae491 - Change search attribute validator to expect field names instead of alias (#3398)
2022-09-16 - 86966c5d2 - Process signals later in scheduler workflow (#3397)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes a fix for an issue where a task queue could fail to process backlogged tasks if it's loaded or reloaded while persistence is unavailable.

All changes

2022-09-13 - 0fcc1b618 - Prepare 1.17.6 patch
2022-09-13 - f4b94d25c - Move build proto dependencies to separate go.mod (#3377)
2022-09-13 - 6e592f636 - Wait to acquire lease in matchingEngine (#3033)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.6)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes a bug fix for Retry Policy logic in Frontend Service handler. If you are not using the custom Search Attributes mapper, you are not affected.

All changes

2022-08-30 - 9bbd336de - Prepare 1.17.5 patch
2022-08-30 - 4e50c8fe2 - Remove frontend handler retries (#3285)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.5)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes bug fixes for Archival and Retry logic, and removes 30 days max Retention Period limit.

IMPORTANT NOTE FOR ARCHIVAL USERS
If you are using the Archival feature and if v1.17.4 of the History Service runs with an older version of the Worker Service during the upgrade, it is possible to have a "dangling Workflow mutable state" where the Event History has been deleted.
To avoid that when upgrading to this release, perform the following steps:

1. Stop the old version of the Worker Service.
2. Deploy this patched release.
3. Start the Worker Service after the upgrade.

All changes

2022-08-16 - 7f7210638 - Prepare 1.17.3 patch
2022-08-16 - 06d641f3b - Fix formating
2022-08-16 - 44267026f - Fix failover queue shutdown (#3232)
2022-08-15 - 0bd7435d7 - Fix archival activities error handling (#3227)
2022-08-12 - f8a59f7ae - Use detached context for shard context operations (#3194)
2022-08-12 - dae8a5152 - Fix queue processor throttling logic (#3195)
2022-08-12 - cea07099c - Remove namespace max retention (#3148)
2022-08-12 - cdb9ae0c9 - Add version check for delete workflow transfer task (#3159)
2022-08-12 - c61450f49 - Fix inline history archival (#3216)
2022-08-12 - ba452a2b9 - additional history scavenger heartbeat (#3186)
2022-08-12 - 5b05741f6 - Simplify system retry logic: Part 1 (#3172)
2022-08-12 - 51817aa60 - Use a prefix for scheduler workflow ids (#3201)
2022-08-12 - 4b7507d77 - Improve task processing retry behavior (#3218)
2022-08-12 - 23e40a9f1 - Fix shard close error type (#3215)
2022-08-12 - bd22047c0 - Run one per-ns worker per namespace instead of namespace × component (#3116)
2022-08-12 - 6aaddf32f - Run task rescheduler in background (#3038)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.4)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes fixes for metrics, rate limiters and Elasticsearch based visibility performance optimizations.

All changes

2022-08-02 - a39ba312d - Prepare 1.17.2 patch
2022-08-02 - 3b8f5c939 - Fix tally timer metric (#3173)
2022-08-02 - 26af874e0 - Cherry Pick: Adds ability to specify sanitizeOptions for prometheus metrics (#3170) (#3174)
2022-07-29 - 7d9b2626f - Core persistence priority rate limiting (#3139)
2022-07-29 - d18625913 - Don't dispatch expired tasks from taskReader buffer (#3161)
2022-07-29 - 28cd79b68 - Remove RunId from visibility query sort for ES 7 (#3140)
2022-07-29 - 0b28e9a2b - Consolidate rpc client metrics logic (#3147)
2022-07-29 - 478582c32 - Generate most of the gRPC wrappers in client (#3001)
2022-07-29 - efc3203ab - Add config to disable ORDER BY clause (#3128)
2022-07-29 - 4a5f6edc4 - Use default consistency level for visibility DeleteWorkflowExecution (#3132)
2022-07-29 - b48cf2a32 - Fix timer metric bucket (#3136)
2022-07-29 - 84149d348 - Fix task rate limiting (#3137)
2022-07-29 - 4d7772d60 - Fix queue rate limiting busy loop (#3111)
2022-07-28 - a1d1650bd - Config worker use public frontend endpoint (#3162)
2022-07-28 - 8dea8807b - Update shard replication timestamp with max generated task (#3158)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes fixes for replication stack as well as for shard lock scope.

All changes

2022-07-11 - 83c2b6a54 - Prepare 1.16.3 patch
2022-07-11 - afd374d49 - Cherry pick #3080 for 1.16 (#3081)
2022-07-08 - 082d7c9cc - Release shard lock earlier during delete workflow execution (#3028)
2022-06-01 - 05a90f834 - Move cluster ack level on ack level update (#2927)
2022-06-01 - 021d5015d - Fix dlq retry policy
2022-06-01 - 360062b60 - Fix failover queue creation after shard reload (#2862)
2022-06-01 - 79102bf9c - Handle NDC active -> passive transition with transient workflow & no buffered event
2022-06-01 - bd87aa388 - Minor tweak to migration workflow wait replication check (#2839)
2022-06-01 - aad0f850c - Misc changes to migration workflow (#2838)
2022-06-01 - b7562a4f9 - Check workflow task after reapply events (#2840)
2022-05-20 - 3fff3f039 - Fix record child completion mutable state stale check (#2885)

Details about v1.16.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.16.3)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes fixes for replication stack, delete namespace functionality, and potential deadlock.

All changes

2022-07-11 - e88aa24b8 - Prepare 1.17.1 patch
2022-07-11 - 13cd5a5a8 - Reorder frontend rate limiter (#3080)
2022-07-08 - 0b9f15c79 - Resource aware operation retry (#3069)
2022-07-08 - 56c529571 - Do not double execute close execution task if visibility archival is disabled (#3072)
2022-07-08 - 646f12d0b - Fix namespace failover callback (#3066)
2022-07-08 - 6737693cd - Handle workflow already completed when verifying first workflow task scheduled (#3061)
2022-07-08 - 12c173c7a - Fix schedule jitter calculation bug (#3059)
2022-07-08 - a95c2fcaa - Fix error shadowing in standby executors (#3053)
2022-07-08 - 0b01a721e - Improve error handling for standby transfer task verification (#3050)
2022-07-08 - 32781119a - Clean up history branch ancestor operation (#3046)
2022-07-08 - fb208d57f - Cancel activity when eager execution and request cancellation are in the same WFT (#3029)
2022-07-08 - 6513af873 - Release shard lock earlier during delete workflow execution (#3028)
2022-07-08 - 4657f0fd0 - Don't check delete workflow execution transfer task version (#3021)
2022-07-06 - 972af2bed - Do not hold lock when running namespace change callback (#3067)
2022-06-30 - 0f13b2358 - add support for multiple pragma statements and driver syntax (#3031)
2022-06-30 - c6810d588 - Update backfill history in sync workflow state (#3042)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

IMPORTANT

This release contains a potential shard lock issue. Do not use this release. Use v1.17.4 instead.

Release highlights

Elasticsearch schema changes

Elasticsearch schema v2 has been added.

The Elasticsearch schema v2 supports new built-in Search Attributes. This schema only adds new mappings, it is not a full reindex.

Use the v2 schema upgrade script schema/elasticsearch/visibility/versioned/v2/upgrade.sh. This can be done before or after upgrading to this release, but it must be done before using the experimental Scheduled Workflow feature.

Metrics

Multiple interfaces in the metrics package are deprecated. The following interfaces are scheduled to be removed in v1.18.0:

• common/metrics/interfaces.go
  • UserScope
  • Scope
  • Client
  • Reporter

Initial implementation to replace these interfaces is now:

• common/metrics/metrics.go
  • MetricsHandler

Histogram metrics use per unit bucket instead of latency unit bucket to emit metrics. In previous releases, the histogram metrics have to x1000 as they are using milliseconds unit. With the per unit bucket update, there is no need to have the x1000 compensation for Dimensionless unit and Bytes unit metrics.

• common/metrics/config.go
  • defaultPerUnitHistogramBoundaries

New metrics service_error_with_type is added for all service errors. It included error_type as tag. The existing metrics service_errors_<FOR_EACH_SPECIFIC_ERROR_TYPE> are deprecated and will be removed from future release. Example of such existing metrics.
New metrics persistence_error_with_type is added for all persistence errors. It included error_type as tag. The existing metrics persistence_errors_<FOR_EACH_SPECIFIC_ERROR_TYPE> are deprecated and will be removed from future release. Example of such existing metrics.

Workflow Execution deletion

Operator Service now exposes the new DeleteWorkflowExecution API, which accepts the name of a Namespace and a Workflow Execution. The Workflow Execution can be in any state. If it is Running, it will be terminated first. The API is exposed via tctl (version >=1.17.0-alpha.2):

$ tctl config set version next
$ tctl workflow delete --workflow-id my_workflow
$ tctl workflow delete --workflow-id my_workflow --run-id my_run_id

If --run-id is not specified, the latest Workflow Execution will be deleted.

Namespace deletion

Operator Service now exposes the new DeleteNamespace API, which accepts the name of a Namespace to delete. Be very careful with this API: it can’t be undone. The API is exposed via tctl:

$ tctl config set version next
$ tctl namespace delete --name my_namespace_name

Namespace deletion is implemented using a system Workflow that is run by a system Worker. It is an asynchronous process, but the Namespace name can be reused as soon as API returns:

1. The Namespace is marked as deleted and renamed to a temporary name. After the DeleteNamespace API returns, the name can be reused (that is, a Namespace with the same name can be created).
2. All Workflow Executions in the deleted Namespace are deleted asynchronously from the database using the DeleteWorkflowExecution API described earlier.
3. The Namespace is deleted from the database.

Please note: this feature is still under active testing.

Host-level priority Task processing

• New Task processing framework in History service for multi-tenancy. This feature is disabled by default in v1.17 and will be enabled in a future release.
• Improved Task retry mechanism to avoid Tasks occupying a Worker goroutine for an extended period of time and blocking the entire queue.
• Configuration for enabling host-level Task Worker pool. Example of dynamic config to enable them:

history.timerProcessorEnablePriorityTaskScheduler:
- constraints: {}
  value: true
history.transferProcessorEnablePriorityTaskScheduler:
- constraints: {}
  value: true
history.visibilityProcessorEnablePriorityTaskScheduler:
- constraints: {}
  value: true

• Configuration for Worker pool size (history.timer/transfer/visibilityProcessorSchedulerWorkerCount) for controlling total traffic sent downstream.
• Configuration for adjusting high-priority Task quota allowed for each Namespace (history.timer/transfer/visibilityTaskHighPriorityRPS).
• Configuration for Task processing round-robin weights (history.timer/transfer/visibilityProcessorSchedulerRoundRobinWeights) among various priorities.
• New task_priority tag for Task processing metrics.

Scheduled Workflows (experimental)

Scheduled Workflows are a new feature that's more flexible than the existing CronSchedule. They're disabled by default in this release, but you can enable them with dynamic config settings. See more docs [here].

SQLite persistence

The SQLite module was replaced with a purego implementation. CGO is no longer required to use SQLite persistence. This means that you can use binaries attached to this release to run local server for development. Just download archive corresponding to your OS and architecture (Mac M1 works!), unzip it, and run server locally without docker-compose and any other dependencies with simple command:

$ ./temporal-server --env development-sqlite start

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag v1.17.0)

Server
Server With Auto-Setup
what is Auto-Setup?
Admin-Tools

What's Changed

2022-06-17 - 79844fd70 - use correct scope constructor (#3005)
2022-06-17 - 0b0fea021 - Correct tag exclusion logic and add tests. (#3003)
2022-06-16 - b7262fe3c - Metrics perf investigation (#3002)
2022-06-15 - 9eb72eac9 - Rename dt and di variables to workflowTask/wt (#2999)
2022-06-15 - ba0c0635f - Update docker-compose.cdc.yml (#2998)
2022-06-15 - 461005f87 - Remove unneeded duration/seconds conversion (#2994)
2022-06-15 - 49cb949a5 - Rename scheduleID to scheduledEventID and startedID to startedEventID (#2992)
2022-06-14 - 1dade46e9 - improve error message (#2937)
2022-06-14 - 24539467e - add default metricoptions where nil (#2993)
2022-06-14 - b3c281088 - Add equality check for task predicate (#2988)
2022-06-14 - 5cfbdc11f - Use metricProvider in task scheduler related components (#2989)
2022-06-13 - c7e9e938b - Update docker-compose.yml (#2984)
2022-06-13 - d4e763243 - mitigate performance issue with userscope (#2983)
2022-06-13 - a09f7ce1e - Do not retry while holding the shard lock (#2990)
2022-06-13 - d250e0f0b - Use metricProvider in task executable and rescheduler (#2930)
2022-06-13 - bd33793ce - Error with tag metrics (#2987)
2022-06-13 - 946b1cef4 - Move task range definition to queues package (#2986)
2022-06-12 - 3c5b09f2c - Add statsd tag separator capability (#2763)
2022-06-10 - b571edfdd - Multi-cursor components: queue iterator (#2974)
2022-06-09 - 3f302aa18 - Remove extra namespace checks (#2975)
2022-06-09 - 1e4a76d50 - Multi-cursor components: queue scope (#2973)
2022-06-08 - 5135a6875 - check for nil config and return noop (#2979)
2022-06-08 - 8114a4fbd - Remove unindexed fields from index schema template (#2976)
2022-06-08 - d16cc46fd - Adding serveroption for metrics.MetricHandler (#2978)
2022-06-08 - 49b57b7c2 - Don't index search attribute used only for passing data through visibility (#2972)
2022-06-08 - f0f1793bc - Fix error details bug in deletenamespace workflow (#2954)
2022-06-08 - 9de6b453c - Fix getRemoteClusterInfo race condition (#2971)
2022-06-07 - 86109455c - Update SDK to v1.15.0 (#2964)
2022-06-07 - d3dc17e06 - Remove caching objects to prevent leak (#2966)
2022-06-06 - e4918d93b - Adding reporter to shard context for access to UserScope (#2961)
2022-06-06 - 81e7a13a3 - moving handler instantiation to allow noop reporter for autosetup (#2960)
2022-06-06 - b4f2ff513 - also lookup scopedef by common if not found by service (#2959)
2022-06-06 - 48cc8d1b6 - Revert "Refactor authorization interceptor by extracting utilities (#2945)" (#2958)
2022-06-06 - 14556ddb2 - Fill in fields in ListSchedulesResponse (#2946)
2022-06-06 - ba8ce6533 - Multi-cursor queue components: task predicates (#2957)
2022-06-06 - e16e6e820 - Multi-cursor queue components: task key and range (#2956)
2022-06-06 - bef27908f - Add schedule-related search attributes (#2953)
2022-06-06 - 96a0bcd6a - Tune host level task processing performance (#2955)
2022-06-06 - e5b4e14a6 - Update docker-compose.yml (#2952)
2022-06-04 - 732d43a34 - Fix task parallel processor test (#2951)
2022-06-04 - fa2e06cb6 - Update UI readme to v2 (#2936)
2022-06-04 - 8444848ba - Fix host level task scheduler start/stop (#2948)
2022-06-03 - bd5e72225 - Updating dependency due to redaction (#2947)
2022-06-03 - ebe226256 - Updates to metrics interface, road to deprecate metrics.Reporter (#2935)
2022-06-03 - e3a5e7872 - Resetting open child workflow should not sent Terminated Failure to the parent #725 (#2913)
2022-06-03 - e71a51fbc - Refactor authorization interceptor by extracting utilities (#2945)
2022-06-03 - 8de7333b5 - Handle deleted namespaces in perNamespaceWorkerManager (#2939)
2022-06-03 - d1f73ad7f - Ensure timer max query level always above min level (#2944)
2022-06-03 - dfbcdce80 - Fix task rescheduler metric scope (#2943)
2022-06-03 - c3d172d42 - Clean up reset reapply default value (#2938)
2022-06-02 - 4c11b96c0 - Update replication processor/executor init logic (#2926)
2022-06-02 - 58d2964e5 - Fix cluster metadata remove (#2940)
2022-06-02 - 9e69052e8 - Add unit tests for perNamespaceWorkerManager (#2933)
2022-06-02 - 803276bca - Improve deletenamespace logging (#2934)
2022-06-02 - a347c299f - Add gomock mocks for SDK Client and Worker (#2931)
2022-06-01 - 4c62f56f3 - Handle visibility task timeout in bulk operation (#2895)
2022-06-01 - 9ad9e7237 - Add priority tag for task processing metrics (#2921)
2022-06-01 - 556b60dd3 - Update shard persistence test (#2928)
2022-05-31 - e7905786a - Limit max number of shards used by tests (#2925)
2022-05-31 - 118aae5fb - Generate replication task when updating cluster list (#2924)
2022-05-31 - 775d1fb87 - Use MutableSideEffect for scheduler workflow tweakables (#2906)
2022-05-31 - 78beb81db - History queue processor task loading host RPS limit (#2920)
2022-05-31 - 5d88844d5 - Fill in SupportsSchedules field (#2922)
2022-05-31 - 0f349353c - Fix npe for dlq retry policy (#2923)
2022-05-31 - 7af6cda77 - Delete global namespace (#2867)
2022-05-31 - 00394e9d1 - Add close time in mutable state (#2917)
2022-05-31 - 96796769a - Dynamic config for scheduler server worker (#2904)
2022-05-31 - f5ba752e8 - Add dynamic config to disable frontend schedule rpcs (#2908)
2022-05-27 - 3855a36bb - moving fork to temporal (#2916)
2022-05-27 - 3dbabab40 - Use namespace Id for replication APIs (#2914)
2022-05-27 - a7e3520bb - metric interface refactor, consumer and producer decoupled (#2883)
2022-05-27 - df4d519bb - Make task processing worker count config truly dynamic (#2911)
2022-05-27 - 4e7098384 - Improve deletenamespace workflow errors (#2909)
2022-05-26 - 738b36f01 - Add namespace_id field to events with namespace field (#2903)
2022-05-26 - 9b8cadc4c - Fix timer task completion (#2910)
2022-05-26 - 9bfadcdbf - Properly initialize task key fire time (#2907)
2022-05-26 - 3beaf63ad - Add schedule rpc handlers (#2857)
2022-05-26 - 7ab7ef237 - Add scheduler server worker (#2856)
2022-05-25 - 2209c4df6 - Update comments for RequestCancelWorkflow (#2889)
2022-05-25 - 89c0ef323 - Rename legacy history replication task (#2901)
2022-05-25 - 8a2dcec13 - Update vscode launch scripts (#2898)
2022-05-24 - df78a57da - Add sync workflow state task (#2853)
2022-05-24 - 4ac6e3fbe - Handle NDC active -> passive transition with transient workflow & no buffered event (#2845)
2022-05-24 - c39cfb764 - Misc. improvements for priority task processing (#2897)
2022-05-24 - 9e07a3478 - Use workflow consistency checker for all APIs (#2873)
2022-05-23 - 6b2b3f29e - Remove deprecated shard info ack level fields (#2884)
2022-05-20 - df43fbaac - Changed RegisterNamespace to initialize new Namespaces in REPLICATION_STATE_NORMAL (#2887)
2022-05-20 - 04f209418 - Fix typo (#2888)
2022-05-20 - 563510185 - Fix bug and simplify dynamicconfig (#2875)
2022-05-20 - b530a356a - Add String method to ContextImpl to fix a race (#2879)
2022-05-20 - b94ddc1b3 - Update base-ci-builder to 1.5.0 (#2880)
2022-05-20 - ad573c06e - Update tctl details in contributing.md (#2881)
2022-05-20 - d3f3a2989 - add ContinueAsNew as action command (#2876)
2022-05-20 - 548dd6c4c - Check visibility ack level in standby cluster for DeleteWorkflowExecution (#2870)
2022-05-20 - faa215f4d - Use --env instead of --zone to start the server in development environment (#2878)
2022-05-20 - 98f452596 - Clean up dynamic configs (#2877)
2022-05-20 - a40214d94 - Guarantee history task execution (#2864)
2022-05-20 - 2758e2940 - Allow dynamic config filter by namespace or task queue name only (#2858)
2022-05-19 - 150516aeb - Replicate workflow state (#2852)
2022-05-19 - 1eed3a72f - Disable cgo by default (#2874)
2022-05-19 - 148630ef8 - Add cluster ID to clock (#2871)
2022-05-19 - 5eda9075d - Add task count metric (#2869)
2022-05-19 - d36291f7e - Refactor ndc history resender to handle multiple remote clusters (#2866)
2022-05-19 - b618a945e - Remove lock on current workflow ID (#2872)
2022-05-19 - 65ea1072a - Return error when cluster info is not found (#2868)
2022-05-18 - b8b58b156 - Remove custom retries for ReclaimResourcesWorkflow (#2865)
2022-05-18 - 29de7f9bc - Bypass ack level check for delete workflow execution in standby cluster (#2859)
2022-05-18 - 9edda593a - Add metering action metrics (#2863)
2022-05-18 - b9c335e76 - Admin workflow deletion API (#2841)
2022-05-17 - 08de0ff19 - Guarantee shard shutdown (#2861)
2022-05-17 - b7aa833f2 - Fix failover queue creation after shard reload (#2862)
2022-05-16 - 7d4265457 - Add namespace argument to PerNSWorkerComponent.Register (#2847)
2022-05-16 - b385075d7 - Fix transfer standby queue processor locking (#2854)
2022-05-16 - e562b4452 - Delete workflow executions in global namespace (#2855)
2022-05-16 - b93f87870 - Development environment for multi cluster setup (#2848)
2022-05-16 - ec73b4889 - Add skeleton for Schedule rpcs (#2846)
2022-05-16 - 4a47b8139 - Do not load stiky queue for add task (#2850)
2022-05-13 - 5acbdd70b - Fix auth plugin configuration to match the new SQLAuthPlugin config block (#2844)
2022-05-12 - 208391f71 - Remove fossa from buildkite (#2843)
2022-05-12 - f95945189 - Drop namespace replication task if it does not live in current cluster (#2842)
2022-05-12 - 690ad5427 - Add RDS IAM auth plugin for SQL drivers (#2830)
2022-05-12 - 7361f8beb - Check workflow task after reapply events (#2840)
2022-05-12 - aba6e1f8e - Add scheduler time logic (#2666)
2022-05-12 - ab6e338d4 - Add per-namespace worker manager (#2799)
2022-05-12 - 87c3bebdb - Remove TerminateWorkflowExecution call when deleting namespace (#2837)
2022-05-12 - d95f94daf - Add first_execution_run_id to Get/PollMutableStateResponse (#2836)
2022-05-12 - 3c50e834e - Minor tweak to migration workflow wait replication check (#2839)
2022-05-12 - 85978d589 - Misc changes to migration workflow (#2838)
2022-05-11 - 2b860099a - Resend history for pending standby activity workflow task (#2796)
2022-05-11 - cf4153c8e - Delete running workflow executions (#2819)
2022-05-11 - a089ae5a1 - Fix WorkflowTaskAttempt metric definition (#2835)
2022-05-10 - d91d16426 - Enforce timeout when updating shard (#2833)
2022-05-10 - 6ef79ea8a - Remove global StickyTTL (#2832)
2022-05-10 - 19972ae61 - Fix scheduler NPE for timer failover processor (#2831)
2022-05-10 - 668c85115 - Fix query (#2826)
2022-05-10 - 153fb240d - Update docker-compose.yml (#2828)
2022-05-09 - 256560cf3 - Fix build (#2827)
2022-05-09 - 5da9b57e1 - Refactor replication related components (#2816)
2022-05-09 - be024bf0a - Use public WorkflowNotReady (#2825)
2022-05-09 - 99fdfb61f - Allow refreshing closed or standby workflow tasks (#2809)
2022-05-09 - bffb7559a - Fix potential deadlock in shard addTask (#2823)
2022-05-09 - 2f43b88bd - Fix mutable state stale check for recordChildExecutionCompleted (#2821)
2022-05-09 - c118f6b96 - Verify child first workflow task scheduled (#2822)
2022-05-09 - 0c7541816 - Add appendRawHistoryNodes api (#2797)
2022-05-09 - 3dc5143ab - Deprecate StatsTypeTag (#2820)
2022-05-09 - 9c9b4c92e - Fix sporadic duplicate key errors in mysql queue implementation (#2802)
2022-05-08 - aa2c98ee4 - Verify child completion recorded (#2806)
2022-05-07 - f68d9f700 - update slack link (#2818)
2022-05-07 - 4767cd4e5 - Fix error handling in processParentClosePolicy (#2814)
2022-05-07 - e3e48c95c - Rename clockpb to clocksbp import alias (#2815)
2022-05-07 - b31e51233 - Remove obsolete nil CloseTime check (#2813)
2022-05-06 - 95dfa072f - Fix 5s delay on sticky queue (#2811)
2022-05-05 - 1f442652f - Return NamespaceNotFound error from RespondWorkflowTaskCompleted handler (#2810)
2022-05-05 - a8a4d27ef - Remove UNSPECIFIED failed cause (#2807)
2022-05-05 - f1dbddc61 - Use part of namespace ID for delete namespace name (#2803)
2022-05-05 - f1fb6350d - Expose namespace data (#2804)
2022-05-05 - 3912a3e31 - Always schedule first workflow task for started abandoned child (#2414)
2022-05-04 - aa257cd70 - Pass child initiated event version to child workflow (#2801)
2022-05-03 - 6614ea64b - Remove legacy execution table tests (#2793)
2022-05-03 - eca0a0fe6 - Remove unused method from MutableState (#2798)
2022-05-02 - b7064c3b3 - Lazy init task executable logger (#2795)
2022-05-02 - 08a02265b - Ensure queue processor cluster ack level is below failover ack level (#2794)
2022-05-01 - e69d63983 - Simplify SignalWithStart API (#2791)
2022-04-30 - 603530406 - Add NamespaceInvalidState and NamespaceNotFound errors (#2785)
2022-04-29 - 92ff8c89d - Fix flaky TestClusterMembershipReadFiltersCorrectly (#2790)
2022-04-29 - 723b7a5e4 - Remove task version and visibility timestamp override (#2789)
2022-04-29 - 2a938b8f1 - Limit number of delete workflow executions retries (#2768)
2022-04-29 - f84d6d0bb - Update proto linters (#2773)
2022-04-29 - 37dcdeadd - Move history API logic into its own package (#2783)
2022-04-29 - 4faf1f4b0 - Use same root tags for system scope and user scope (#2784)
2022-04-28 - 08d02325a - Wire up host level task scheduler (#2779)
2022-04-28 - fcf7d19dd - Limit task processing time (#2770)
2022-04-28 - c302cb3ca - Move history API logic into its own package (#2780)
2022-04-28 - 130d36eb5 - Move history API logic into its own package (#2778)
2022-04-27 - 55405056d - Fix set default per unit histogram boundaries (#2775)
2022-04-27 - 6ff2f59d3 - Support UI as a client in version checker (#2751)
2022-04-27 - 07d8a41ac - Fix conditional variable test (#2774)
2022-04-27 - 724ce2a37 - Expose DeleteWorkflowExecution API on operatorservice (#2761)
2022-04-27 - 8f98374a8 - Deprecate history task processor (#2764)
2022-04-27 - b747b1798 - Update docker-compose.yml (#2769)
2022-04-26 - f7efbad6b - Do not wake up workflow in retry backoff upon signal (#2771)
2022-04-26 - 8e3fb9721 - Update mutable state consistency check logic (#2747)
2022-04-26 - 631e27d0d - Add dynamic config for namespace refresh interval (#2766)
2022-04-26 - 02542f9d3 - Don't fail ContinueAsNew command if parent namespace doesn't exist (#2767)
2022-04-26 - ca586ddfc - Slow down workflow task retry (#2765)
2022-04-24 - cd88c6171 - Improve history client retry logic (#2762)
2022-04-22 - 58103d710 - Remove namespace field from ScheduleActivityTaskCommandAttributes message (#2753)
2022-04-22 - fa39d1408 - Utilize shard clock (#2746)
2022-04-22 - 5ce1a63d7 - Use metrics unit in user scope (#2759)
2022-04-22 - 4b44fd243 - Add failure event if target namespace is missed during transfer task processing (#2752)
2022-04-22 - 8951ead4c - Make shard accessible to history handler (#2745)
2022-04-22 - 18399f2ad - Update docker-compose.yml (#2758)
2022-04-22 - 10aa1c404 - Disable cgo by default (#2760)
2022-04-21 - 041eaa362 - Task executable scheduler implementation (#2750)
2022-04-21 - 58456a474 - Utilize shard clock for workflow / activity task (#2744)
2022-04-21 - 8426ac5a9 - Update upper histogram boundary (#2756)
2022-04-20 - 04ac068f7 - Adding error type tag to peristence errors (#2748)
2022-04-20 - 7d46dc628 - Remove duplicated require section from go.mod (#2754)
2022-04-20 - a65e0caef - Fail workflow task with BadSearchAttributes cause if search attributes are wrong (#2742)
2022-04-20 - d84a19c87 - Task executable implementation (#2738)
2022-04-20 - 0458cadad - Upgrade OpenTelemetry to v0.29.0 (#2717)
2022-04-20 - 1336d46b7 - Task rescheduler implementation (#2739)
2022-04-20 - dd4a42720 - Task priority assigner implementation (#2740)
2022-04-20 - ddf8abe50 - Use better retry logic in reclaimresources workflow (#2689)
2022-04-19 - a8d47d0ec - Add shard clock proto definition (#2743)
2022-04-19 - d98c7f0ce - Revert back to using file URI for sqlite dsn (#2731)
2022-04-19 - 762bf913c - Always report service error as additional metric with error-type tag (#2741)
2022-04-19 - 6deeed5a6 - Use caller object as callback listener ID (#2734)
2022-04-19 - c6ae43d8e - Host level worker pool components interface (#2736)
2022-04-18 - 78defe19e - Use uuid as namespace notification ID (#2733)
2022-04-14 - 67e34a342 - Simplify history engine init & wait logic (#2725)
2022-04-14 - 86c9da432 - Remove CLI version information (#2715)
2022-04-13 - ea2723338 - Do not drop workflow task if it failed due to unhandled events (#2720)
2022-04-13 - 6409090d6 - Simplify history engine task read ID logic (#2724)
2022-04-13 - d541fc027 - Log NotFound error for transfer task processing (#2723)
2022-04-13 - 490d18f67 - Cancel context earlier (#2721)
2022-04-13 - 57aa72c8f - Add broadcast shutdown channel (#2647)
2022-04-13 - 8fae0c7dc - Update future to use generics (#2722)
2022-04-13 - b8b9eea59 - replace sqlite3 with modernc (#2704)
2022-04-13 - 25e2cd754 - Improve mem efficiency when mutating workflow (#2706)
2022-04-13 - ca4fa9ecb - Persistence context part 9: elasticsearch visibility store (#2713)
2022-04-13 - 168646164 - Persistence context part 8: cassandra store (#2712)
2022-04-13 - 2e55dd385 - Persistence context Part 7: SQL store implementation (#2711)
2022-04-13 - 49a9ebd87 - refactor ringpop interfaces (#2680)
2022-04-12 - 23480fce3 - Update dependencies (#2716)

New Contributors

• @Alex-Tideman made their first contribution in https://github.com/temporalio/temporal/pull/2758
• @nagl-temporal made their first contribution in https://github.com/temporalio/temporal/pull/2802
• @alexandrevilain made their first contribution in https://github.com/temporalio/temporal/pull/2835
• @gnz00 made their first contribution in https://github.com/temporalio/temporal/pull/2830
• @aaronjheng made their first contribution in https://github.com/temporalio/temporal/pull/2888
• @arnesenfamily made their first contribution in https://github.com/temporalio/temporal/pull/2892
• @feihuang made their first contribution in https://github.com/temporalio/temporal/pull/2945

Full Changelog: https://github.com/temporalio/temporal/compare/v1.16.0...v1.17.0

Release Highlights

This release includes fixes for:

• Workflow stuck issues during namespace failover
• Missing first workflow task when starting an abandoned child workflow
• SignalWorkflowExecution waking up a workflow in retry backoff period

The release also improves the configuration around namespace cache refresh.

All changes

2022-05-05 - e12a51cb8 - Prepare 1.16.2 patch
2022-05-05 - 705cbe9c6 - Do not wake up workflow in retry backoff upon signal (#2771)
2022-05-05 - ddb29dc98 - Expose namespace data (#2804)
2022-05-05 - ae8477693 - Always schedule first workflow task for started abandoned child (#2414)
2022-05-05 - 91b099be0 - Ensure queue processor cluster ack level is below failover ack level (#2794)
2022-05-05 - f62095d6e - Remove task version and visibility timestamp override (#2789)
2022-05-05 - aa566f764 - Add dynamic config for namespace refresh interval (#2766)

Details about v1.16.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.16.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes fixes for duplicated callback listener ID on namespace registry and cluster metadata. It also improves the logic for dropping failed workflow task and logging for history task processing logic.

All changes

2022-04-19 - 1c77018e5 - Prepare 1.16.1 patch
2022-04-19 - 254a3815d - Use caller object as callback listener ID (#2734)
2022-04-19 - 0b2721246 - Use uuid as namespace notification ID (#2733)
2022-04-15 - ca6b7ac1d - Do not drop workflow task if it failed due to unhandled events (#2720)
2022-04-15 - 0d0fc8732 - Log NotFound error for transfer task processing (#2723)

Details about v1.16.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.16.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

Schema changes

Before upgrading to your Temporal Cluster to this release 1.16, you must upgrade your storage schema version to the following:

• MySQL schema version 1.8
• PostgreSQL schema version 1.8
• Cassandra schema version 1.7

Use the Schema Upgrade Tool to upgrade your schema version.
How to use the Schema Upgrade Tool: Upgrade Server version.

Note that schema version is NOT the same as the storage type (database) version, see “Latest SQL database support” below.

Docker Version

• This version (1.16) will be the last version that we build our images based on Alpine 3.13. In the next release (1.17), we are going to upgrade our base image to use Alpine 3.15, which may not be able to run on docker versions lower than 20.10.0. Please check more details here.

Golang version

• Upgraded golang to 1.18

Tctl

• tctl implementation has been moved to a new repository

API

• New Operator service for managing namespace and search attribute
• New DeleteNamespace API in OperatorService. Please note that the delete namespace feature is still under active development and not production ready
• New TERMINATE_IF_RUNNING workflowIdReusePolicy. If a workflow is running using the same workflow ID, this policy will terminate it and start a new one; otherwise, the behavior is the same as ALLOW_DUPLICATE
• RequestCancelWorkflowExecution and RequestCancelExternalWorkflowExecutionCommand now supports reason field. The reason will also be recorded in WorkflowExecutionCancelRequestedEvent and RequestCancelExternalWorkflowExecutionInitiatedEvent
• NonRetryableErrorTypes now supports TemporalTimeout:StartToClose and TemporalTimeout:Heartbeat for customizing retry behavior upon timeout failure
• New last_heartbeat_details field in RespondActivityTaskFailedRequest and RespondActivityTaskFailedByIdRequest
• New activity_failure_include_heartbeat system capability indicating if RespondActivityTaskFailed API supports including heartbeat details
• New ListHistoryTasks API in admin service for listing transfer, timer, replication and visibility tasks
• New RebuildMutableState API in admin service

Configuration

• New frontend.namespaceRPS.visibility and frontend.namespaceBurst.visibility dynamic config for per instance visibility API rate limit
• Removed per service Metrics config
• New RemoteClusters field for controlling TLS settings for takling to remote cluster
• New SKIP_DB_CREATE environment variable for skipping DB creation in auto-setup
• If you use SQL persistence layer, please add the following configuration to your dynamicconfig file so that the ID length limit matches the actual limit used in sql schema. A sample configuration file can be found here

  limit.maxIDLength:
  - value: 255
      constraints: {}
  

Metric

• New matching service task_lag_per_tl metric for estimated task lag
• New service_name metrics tag
• New matching task_write_throttle_count and task_write_latency metrics
• New worrkflow_task_attempt metric
• New task_skipped metrics for number of skipped history tasks

Tool

• New disable-initial-host-lookup flag in temporal-cassandra-tool for disabling initial host lookup

Namespace deletion

• System workflows for marking namespace as deleted and reclaiming resources
• Exclude deleted namespaces from ListNamespace response
• Allow delete namespace by NamesapceId
• Delete namespace feature is still under active development and not production ready

Activity local dispatch

• An optimization for skipping matching service and task polling for running activity tasks
• This optimization requires temporal SDK support and it's still under development

Improvements and fixes

• Buffer event as local namespace after promotion
• Use type from type map before type from Metadata
• Add sqlite persistence tests
• Fix history client NPE for invalid shardID
• Fix timer queue cluster ack level update
• Return ErrExecutionsStillExist if executions are still exist
• Fix record child workflow complete mutable state stale check
• Fix namespace not active error metric
• Unify error handling of CRUD operation of workflow within shard context
• Ensure minimal context timeout for shard context APIs
• Handle workflow not found in replication
• Handle activity retry timer in passive
• Allow delete open workflows from passive sides
• De-duplicate reapply event
• Fix replication error handling for reseting workflow with pending child
• Change operator service error codes if Search attributes are not found or already exist
• Limit timer look ahead
• Use ListWorkflow instead of ScanWorkflow for batch operation
• Remove replication start event sanity check
• Copy version history before returning
• Fix shard context error state check
• Add SetWorkflowExecution persistence API
• Pass useTLS in GetFrontendClientConfig's callback
• Handle delete history branch when mutable state does not exist
• Rework health check
• Support pagination for admin get task queue tasks
• Fix unloading shard after task id sanity check
• Fix get history scheduled tasks query
• Do not generate sync activity task for deleted activity
• Support pagination for SQL GetTransfer/VisibilityTasks
• Fix potential shard deadlock issue
• Update error code and message for missing activity task
• Return NOT_FOUND for invalid workflow ID
• Fixing syntax in first migration to the Cassandra schema
• Backoff failed workflow task
• Add task queue to SQL visibility
• Check if global ns enabled in handler
• Preserve the info when updating is_global_namepsace_enabled
• Add dynamic config change diff logging
• Force metrics.Reporter as metrics extension point
• Improve extensibility of optentelemetry metrics
• Fix DeleteWorkflowExecution API when delete non current execution
• Generate namespace replication only with 2+ clusters
• Add tag support to sdkLogger
• Expose metrics reporters constructors to end-user
• Treat enablement of TLS separately for server and client config
• Add support for GetWorkflowHistory in reverse order
• Fix timerMaxReadLevel map init and update
• Do not blindly retry timer task read
• Exit retry for long poll if context is near deadline
• Convert panic when it is not error

Refactoring

• Delete bootstrap params
• Add context parameter to persistence interface and implementations
• Update iterator to use go generics
• Remove custom build info script
• Replacing SdkClient injection with SdkClientFactory injection
• Make matching persistence range queries [inclusive, exclusive)
• Remove deprecated admin list tasks APIs
• Refactor task filter function and interface
• Unify history task persistence range query behavior
• Refactor persistence task serializer and factory
• Unify history queue processor and task generation implementation
• Provide serverOptions as fx.Out struct
• Make queue processor depend on history engine interface
• Refactor handleWorkflowTaskCompleted retry loop

Others

• Upgrade web images to the latest versions
• Remove docker files
• Removing vim from docker images due to CVE
• Added mockgen to base-ci-builder Docker image
• [docs] improve documentation for building images
• Add Temoporal Web 2.0 to development docker-compose
• Split goreleaser archive in two:cgo/nocgo

All changes

2022-04-11 - bcf9e57a0 - Client error log level (#2701)
2022-04-06 - 659c05253 - Change Invalid search attribute value message (#2705)
2022-04-01 - c1d63b713 - buffer event as local namespace after promotion (#2699)
2022-04-01 - 232ea210c - Use type from type map before type from Metadata (#2697)
2022-03-30 - 306eda804 - Add sqlite persistence tests (#2681)
2022-03-30 - ad56ef249 - Upgrade web images to the latest versions (#2684)
2022-03-30 - 2ebc73eed - Delete bootstrap params (#2652)
2022-03-29 - b440f924c - Fix history client NPE for invalid shardID (#2679)
2022-03-29 - ccba09b1f - Fix timer queue cluster ack level update (#2678)
2022-03-29 - 6b15af388 - Return ErrExecutionsStillExist if executions are still exist (#2676)
2022-03-29 - c6243eb79 - Set WorkflowID for child workflows (#2675)
2022-03-29 - a274580ed - Fix record child workflow complete mutable state stale check (#2673)
2022-03-29 - 3af96c64e - Persistence context part 6: visibility store interfaces (#2674)
2022-03-29 - 163234048 - Update iterator to use go generics (#2672)
2022-03-29 - 22f58a91d - Add DeleteNamespace API unit tests (#2670)
2022-03-29 - 7bff5ee6b - Improve deletenamespace workflow logging (#2667)
2022-03-28 - 946961847 - Store namespaces in DELETED state in registry (#2665)
2022-03-28 - b87a3ead8 - Fix parent close policy processor for deleted namespace (#2664)
2022-03-28 - c6a0cfc7a - Move namespace cache delay to reclaim resources workflow (#2658)
2022-03-28 - 30c286787 - Fix namespace not active error metric (#2663)
2022-03-28 - bfaee75cd - Unify error handling of CRUD operation of workflow within shard context (#2662)
2022-03-28 - f066d97df - Ensure minimal context timeout for shard context APIs (#2660)
2022-03-28 - d6f85df4f - Persistence context part 5: persistence store interfaces (#2659)
2022-03-28 - 642d64a11 - Merge SQL schema changes for 1.16 release (#2661)
2022-03-25 - e66040395 - Handle workflow not found in replication (#2633)
2022-03-25 - 6b8986c05 - Add reason field for workflow cancellation (#2653)
2022-03-25 - cc5171143 - Add DeleteNamespace API (#2645)
2022-03-25 - 752ff502b - Exclude deleted namespaces from ListNamespace response (#2646)
2022-03-25 - 4152b17dd - Select the correct proto commit (#2651)
2022-03-25 - 7c8ac02ac - Suppress nonuseful namespaceReplicationMessageProcessor log (#2648)
2022-03-25 - 12f47312b - Handle activity retry timer in passive (#2640)
2022-03-25 - 82ab54f6e - Allow delete namespace by Id (#2643)
2022-03-24 - f0d20df45 - Allow delete open workflows from passive sides (#2636)
2022-03-24 - 8a8f0aac1 - Fix merge conflit (#2644)
2022-03-24 - d663e2cb8 - Persistence Context Part 4: Visibility Manager (#2639)
2022-03-23 - 90b749427 - Fix merge confict (#2642)
2022-03-23 - 559679073 - Rename activity local dispatch request field (#2641)
2022-03-23 - b6de7d55f - Delete namespace workflow (#2569)
2022-03-23 - e60d6bde8 - Persistence Context Part 3: Namespace Replication Queue (#2638)
2022-03-23 - a0c900120 - Persistence Context Part 2: Shard, Task, Metadata, ClusterMetadata Manager (#2632)
2022-03-22 - 7fb1bbea6 - Activity local dispatch (#2618)
2022-03-22 - af9130dcd - Remove custom build info script (#2635)
2022-03-22 - 715d9ce55 - Upgrade go version to 1.18 (#2626)
2022-03-22 - 32977ed8a - De-duplicate reapply event (#2629)
2022-03-22 - 5f776150d - Remove tctl code (#2461)
2022-03-21 - 856089a0b - Handle reset workflow with pending child wf (#2624)
2022-03-21 - f7ed39491 - Change operator service error codes if Search attributes are not found or already exist (#2627)
2022-03-21 - f4c08526e - Persistence Context Part 1: Execution Manager (#2622)
2022-03-21 - b7100aeab - Remove docker files (#2456)
2022-03-21 - cd67ba3d9 - TerminateIfRunning workflowID reuse policy (#2608)
2022-03-20 - b577e89fb - Limit timer look ahead (#2623)
2022-03-18 - 32410dcce - Use ListWorkflow instead of ScanWorkflow for batch operation (#2620)
2022-03-18 - e2d6b31c0 - Bump SDK. (#2601)
2022-03-17 - 60bb20771 - Update server capabilities for sdk usage (#2619)
2022-03-17 - 2bcfb231c - Replacing SdkClient injection with SdkClientFactory injection. (#2611)
2022-03-17 - 650d33704 - Remove replication start event sanity check (#2616)
2022-03-16 - fa3610a82 - Copy version history before returning (#2613)
2022-03-15 - 28aebc21e - Fix shard context error state check (#2612)
2022-03-15 - c8960b527 - Revert "Added mockgen to base-ci-builder Docker image (#2574)" (#2588)
2022-03-15 - bb7b984ef - Make matching persistence range queries [inclusive, exclusive) (#2599)
2022-03-14 - 9732f77eb - Emit estimated matching task lag metric (#2605)
2022-03-14 - 3a3057e03 - Misc. improvements for error checking (#2602)
2022-03-12 - cafdda9ec - removing vim from docker images due to CVE (#2604)
2022-03-11 - da95e7abb - Add a test for #2563 (#2607)
2022-03-11 - 750c61a4b - Add SetWorkflowExecution to workflow context (#2606)
2022-03-11 - ee8dda326 - Updating sql schema to expand VARCHAR (#2568)
2022-03-10 - 4c4652dfe - Ignore parent close policy if child namespace is deleted (#2596)
2022-03-10 - c04f879f9 - Continue to delete workflow execution if namespace is deleted (#2576)
2022-03-10 - f994ea5e6 - Rename namespace persistence API (#2595)
2022-03-09 - 7d1d1c114 - Pass useTLS in GetFrontendClientConfig's callback (#2563)
2022-03-09 - 169d03bcf - Adds flag to temporal-cassandra-tool to disable Initial Host Lookup #2594
2022-03-08 - 1e3bc68f4 - Remove deprecated admin list tasks APIs (#2592)
2022-03-08 - 4375f9428 - Handle delete history branch (#2591)
2022-03-08 - 6fa5b4a00 - Add dynamic config for per instance visibility API rate limit and burst (#2585)
2022-03-08 - 5fb4f1c23 - Rework health check (#2579)
2022-03-08 - 9d69465c3 - Support pagination for admin get task queue tasks (#2583)
2022-03-08 - 4054170bb - Admin list history tasks API (#2580)
2022-03-08 - 72fb979e2 - Match history and proto task category definition (#2586)
2022-03-08 - 11c67aafa - Adding heartbeat details in RespondActivityTaskFailed apis (#2560)
2022-03-07 - 0a2d2e5c4 - Add test for an expected format of Datetime search attribute (#2589)
2022-03-07 - ac1ba9839 - Export cassandra test setup definitions for reuse (#2581)
2022-03-07 - df18e3d8e - Fix unloading shard after task id sanity check (#2571)
2022-03-07 - 810302264 - Fix get history scheduled tasks query (#2578)
2022-03-04 - 9cb3f5a61 - Added mockgen to base-ci-builder Docker image (#2574)
2022-03-04 - 2c6b76f32 - Refactor task filter function and interface (#2556)
2022-03-04 - db40f0217 - Do not generate sync activity task for deleted activity (#2572)
2022-03-04 - 441f7e0ed - Support pagination for SQL GetTransfer/VisibilityTasks (#2564)
2022-03-04 - 1a894659a - Remove deleted namespace from registry (#2567)
2022-03-04 - 40e9fe098 - Fix potential shard deadlock issue (#2570)
2022-03-02 - 19cee07b6 - Update error code and message for missing activity task. (#2561)
2022-03-01 - 8433f9f43 - Update execution tests (#2562)
2022-02-28 - 716b3cd23 - Return NOT_FOUND for invalid workflow ID (#2559)
2022-02-28 - c3e7dbec0 - Unify history task persistence range query behavior (#2547)
2022-03-01 - b431869d6 - [docs] improve documentation for building images (#2557)
2022-02-28 - 4c3810885 - Refactor persistence task serializer (#2555)
2022-02-28 - 3b070308a - Refactor persistence factory implementation (#2544)
2022-02-28 - 9778fa361 - Bump Web to v1.14.0 (#2550)
2022-02-26 - aa817b204 - #2551 Fixing syntax in first migration to the Cassandra schema (#2552)
2022-02-25 - 87a48bb9c - Backoff failed workflow task (#2548)
2022-02-25 - 6107df5ab - reduce unnecessary logging (#2549)
2022-02-25 - f7952b427 - Deprecate per service metrics config (#2546)
2022-02-25 - 9964786cd - Add service type tag to metrics (#2545)
2022-02-24 - f92cc9382 - Sql task queue visibility (#2537)
2022-02-24 - 9e47a655f - Queue refactor part 7: injection (#2534)
2022-02-24 - f79eeef65 - Check is global ns enabled in handler (#2541)
2022-02-24 - 60e5a8b29 - Preserve the info when updating is_global_namepsace_enabled (#2540)
2022-02-24 - 4d129d1e6 - Operator API (#2503)
2022-02-23 - 4f0135572 - provide serverOptions as fx.Out struct (#2536)
2022-02-23 - 4c212e5bb - Add dynamic config change diff logging (#2494)
2022-02-23 - 0c402433d - Queue refactor part 6: shard context (#2511)
2022-02-23 - 3dfccf02c - Queue refactor part 5: persistence layer (#2507)
2022-02-23 - c479ea309 - Update interfaces and structs for admin use (#2533)
2022-02-22 - a5a91ffac - Add RebuildMutableState API (#2532)
2022-02-22 - 253c820ea - Force metrics.Reporter as metrics extension point (#2529)
2022-02-22 - ef1ca2a57 - Fix flaky matching AddTaskStandby test (#2527)
2022-02-22 - acb96fd97 - Adds ExecutionInfo reference to InternalWorkflowMutation #2530
2022-02-22 - 4d06e1aca - Make queue processor depend on history engine interface (#2514)
2022-02-22 - f6acc48e8 - Allow customizing retry behavior for timeout failure (#2524)
2022-02-22 - 0a76b2ae2 - Log build CGO_ENABLED state (#2509)
2022-02-20 - 61b9b6574 - Improve extensibility of optentelemetry metrics (#2521)
2022-02-18 - 73f40bf3e - Add SetWorkflowExecution persistence API (#2523)
2022-02-18 - 2e69c936c - Define type and name for task categories (#2522)
2022-02-18 - faa1dc135 - Clean up auto-setup.sh (#2516)
2022-02-18 - ede203533 - Add Temoporal Web 2.0 to development docker-compose (#2520)
2022-02-17 - 305a6fc97 - Add config and handling for remote cluster cert (#2475)
2022-02-17 - cbd1b5e2d - Fix DeleteWorkflowExecution API when delete non current execution (#2484)
2022-02-16 - 1d33c3ed9 - Refactor handleWorkflowTaskCompleted retry loop (#2483)
2022-02-16 - d2f41f050 - Add SKIP_DB_CREATE env to auto-setup.sh (#2495)
2022-02-16 - ed29a48b7 - Fix metrics scope misuse (#2515)
2022-02-16 - 1f2b29162 - Generate namespace replication only with 2+ clusters (#2513)
2022-02-16 - 24f6dc36c - Emit matching task writer latency and throttle count (#2504)
2022-02-15 - b0f9d3a68 - Add tag support to sdkLogger (#2512)
2022-02-15 - 615fb91d2 - Trigger Docker builds on release branches (#2510)
2022-02-15 - eed1fde32 - Expose metrics reporters constructors to end-user (#2481)
2022-02-14 - 632c4f7a8 - Treat enablement of TLS separately for server and client config (#2501)
2022-02-14 - f6e84df10 - Move goreleaser scripts to Makefile (#2508)
2022-02-14 - dee4abc26 - Update go version of goreleaser to to 1.17.6 (#2506)
2022-02-14 - f0d98a89d - Add support for GetWorkflowHistory in reverse order (#2442)
2022-02-14 - f5a39b7fd - Split goreleaser archive in two:cgo/nocgo (#2505)
2022-02-14 - 27cc0b164 - Fix timerMaxReadLevel map init and update (#2498)
2022-02-14 - 2a57ba95c - Upgrade dependencies (#2499)
2022-02-13 - be1f9084e - Do not blindly retry timer task read (#2500)
2022-02-11 - b99986f05 - Suppress two info logs that are not useful (#2493)
2022-02-11 - 3afd57dff - Queue refactor part 4: task serializer (#2490)
2022-02-11 - 131563b03 - Queue refactor part 3: task notification (#2489)
2022-02-11 - 855ef8e15 - Replace Cyrillic c with c in caCert (#2492)
2022-02-11 - 1552628ce - Emit workflow task attempt stats (#2487)
2022-02-10 - 61dac76ed - Exit retry for long poll if context is near deadline (#2488)
2022-02-10 - d54dd19bb - Queue refactor part 2: task generation (#2479)
2022-02-10 - 0f0c853db - convert panic when it is not error (#2486)
2022-02-09 - ff18623ec - Emit history task skipped metric (#2485)
2022-02-09 - 6a25df34b - Queue refactor part 1: define processor interface and category (#2478)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.16.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools