Release Highlights

This release includes minor bug fixes for adding remote cluster. See full list of patches below.

All changes

2022-01-21 - 5f418b916 - Update version checker to 1.14.4
2022-01-21 - 75bf59f55 - Favor local config value on enable global namespace (#2368)
2022-01-21 - 4dd443267 - Add validation for invalid starttime filter for standard visibility (#2401)
2022-01-21 - 2e94883ac - Fix c symbol in caCert variable name (#2397)
2022-01-21 - 2dbf144ac - Wire up cluster metadata in start fx DI (#2398)
2022-01-20 - fc9390e61 - Add logging to matching / history client (#2340)

Details about v1.14.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.14.4)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes a series of fixes:

• Fixed bugs in history service that could potentially miss tasks in failure mode when history pods are bouncing around.
• Updated docker config to allow different DB address for main and visibility DB.
• Support visibility dual write to 2 advanced visibility persistence.
• Apply trim logic for stale history events to PollWorkflowTaskQueue.

All changes

2022-01-18 - 71c697917 - Update versionChecker for 1.14.3
2022-01-13 - c2124f736 - [Docker build] fix defaults with variables in templates (#2378)
2022-01-12 - 72aba0e5b - Unload shard if loaded task ID is greater than max transfer ID (#2361)
2022-01-12 - 3c351f233 - [Docker build] add separate visibility env vars for different db instance (#2362)
2022-01-12 - 319861d4b - Support visibility dual write to different ES indices (#2359)
2022-01-12 - 2457d11b9 - Check max taskID instead of max read level when process task (#2371)
2022-01-12 - 1a3cf49e5 - Include more API to Cassandra read after write inconsistency protection (#2366)

Details about v1.14.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.14.3)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes hotfixes for matching service when dealing with large number of low traffic task queues.
See detailed change list below.

All changes

2022-01-12 - c0436c842 - Update versionChecker to 1.13.3
2022-01-12 - 429291f96 - Move ackLevel past gaps in task ids (#2281)

Details about v1.13.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.13.3)

Server
Server With Auto Setup
Admin-Tools

Release Highlights

This release contains bug that could potentially lose tasks in failure scenario when history pods are not stable and keep bouncing between multiple nodes. Please upgrade to latest patch release 1.14.3 or later.

This release includes a series of fixes:

• Disallow marking activity as cancelled without activity being request cancelled first
• Fix incorrect LRU cache eviction behavior
• Detect stale cache when RecordChildExecutionCompleted
• Eagerly unload shard when ownership is lost

All changes

2022-01-10 - b535b87fb - Update CLI and ServerVersion in version checker for v1.14.2
2022-01-07 - 9ca8598a9 - Detecting stale cache when RecordChildExecutionCompleted (#2354)
2022-01-07 - 94b29c3b9 - Disallow marking activity as cancelled without activity being request cancelled first (#2344)
2022-01-07 - 80cd5f557 - Fix LRU cache eviction behavior (#2353)
2022-01-07 - 5efdf07a4 - Unload shard when ownership lost (#2355)

Details about v1.14.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.14.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release contains a bug that could potentially lose tasks in failure scenario when history pods are not stable and keep bouncing between multiple nodes. Please upgrade to latest patch release 1.14.3 or later.

This release includes a hotfix for persistence stability at high loads.

All changes

2021-12-14 - baf3b94b4 - Use persistence config consistency for schema version check (#2291)
2021-12-20 - f9c1eba85 - Revert persistence rate limit error type (#2312) (#2315)
2021-12-20 - f7abe2db4 - Update CLI and Server versions in version checker (#2317)

Details about v1.14.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.14.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Release Highlights

This release includes a hotfix for persistence stability at high loads.

All changes

2021-12-20 - ce005f641 - Revert persistence rate limit error type (#2312) (#2314)
2021-12-20 - 5fee1718c - Update CLI and Server versions in version checker (#2316)

Details about v1.13.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.13.2)

Server
Server With Auto Setup
Admin-Tools

Schema changes

This release contains a bug that could potentially lose tasks in failure scenario when history pods are not stable and keep bouncing between multiple nodes. Please upgrade to latest patch release 1.14.3 or later.

Before upgrading to your Temporal Cluster to this release 1.14, you must upgrade your storage schema version to the following:

• MySQL schema version 1.7
• PostgreSQL schema version 1.7
• Cassandra schema version 1.6

Use the Schema Upgrade Tool to upgrade your schema version.
How to use the Schema Upgrade Tool: https://docs.temporal.io/docs/server/versions-and-dependencies/#upgrade-your-version-of-temporal.

Note that schema version is NOT the same as the storage type (database) version, see “Latest SQL database support” below.

tctl

• tctl batch - Added a flag to limit amount of goroutines: --concurrency.
• tctl admin - Removed direct database connection and switched to Admin API. Removed flags related to database connection.
• Added a cluster command to upsert or remove remote cluster. This feature is still in development and may change in future release.
• Renamed --min/max_read_level flag to --min/max_task_id when listing cluster tasks.
• tctl namespace - Added a flag to promote a local namespace to a global namespace: --promote-namespace.
• Removed commands related to Elasticsearch.
• Removed cluster metadata command.

Metrics

• Added option to configure histogram buckets per unit type (milliseconds, bytes, dimensionless). Existing metrics now have assigned relevant unit types. Example.
• Added config option to exclude reported metric tags. Example.
• Upgraded uber-go/Tally to v4.0.1.
• Upgraded opentelemetry version to v1.1.0.

Latest SQL database support

• Latest version of MySQL and PostgreSQL were tested. Temporal now officially supports MySQL 8 starting from version 8.0.19 and PostgreSQL 13 starting from version 13.5.

Multi-Cluster

• Cluster metadata definition is migrating from application static config file to a new DB table. Cluster metadata will be initialized once when the service first starts up. Future remote cluster metadata modifications need to be done via tctl admin cluster tool. Example.

Other

• Server will fail to start if dynamic config is malformed. Previously, default config was used if errors occurred.
• Upgraded golang to 1.17.
• Added safety check for empty JWT source URIs.
• Added Handover ReplicationState for graceful failover. This feature is still in development and will change in a future release.

Improvements/Fixes

• Fix unexpected matching GetTasks call growth on low-traffic task queues.
• Reduced overhead due to unloading history shards during short database unavailability.
• Limited Elasticsearch document _id to 512 bytes (#2112).

Internal refactoring

• Adopted uber/fx for server initialization and dependency injection.

All changes

2021-12-13 - 074d8ebb1 - Fix rollup metrics aggregation (#2298)
2021-12-10 - 29e81571b - New future implementation (#2265)
2021-12-09 - f684386fe - Rename simpleStore to standardStore (#2278)
2021-12-09 - 7feb6b3a5 - Replication handover state (#2280)
2021-12-09 - 2eaadeaad - Move ackLevel past gaps in task ids (#2281)
2021-12-08 - c4976c9ea - Only update maxReadLevel after successful re-acquire of shard (#2272)
2021-12-08 - 0e844bc3b - Add additional UTF-8 validation (#2276)
2021-12-07 - 88c6eb5d7 - patch tally histogram (#2270)
2021-12-06 - f430f8573 - Stop server start up on malformed dynamic config (#2268)
2021-12-06 - be0df0d3e - Use correct histogram type for dimensionless (#2264)
2021-12-06 - 93de50aa4 - Add custom buckets to rollup metrics (#2266)
2021-12-06 - 3f4de5243 - Fix Elasticsearch bulk processor metrics units (#2267)
2021-12-03 - beb9a5a15 - Remove namespace APIs from adminservice (#2261)
2021-12-03 - 8a409cd25 - Add option to configure per metric unit histogram buckets. (#2251)
2021-12-03 - 827c92225 - Fix flaky TestVisibility (#2259)
2021-12-03 - 3e19fff5c - Fix batcher (#2260)
2021-12-02 - f338d10d1 - Remove DB record version config (#2253)
2021-12-02 - afc775a7b - Fix onebox worker shutdown (#2255)
2021-12-02 - 822eed7f5 - handover workflow (#2258)
2021-12-02 - 725730697 - Use history event as signal payload data (#2254)
2021-12-02 - 68498e122 - Remove --state from namespace update command (#2257)
2021-12-02 - 27c97af32 - Add shard lifetime context (#2218)
2021-12-02 - 06bd292b5 - Add UpdateNamespace tests for DELETED namespace state (#2185)
2021-12-01 - c9315c3e4 - Add dedicated persistence execution mutable state integration tests (#2244)
2021-12-01 - c099c7963 - Support limited list workflow queries for standard  visibility (#2197)
2021-12-01 - aa17a4aba - Matching cleanup (#2246)
2021-12-01 - 873f28085 - Add handover namespaces to GetReplicationStatus response (#2248)
2021-12-01 - 815bfd7d0 - Fix error messages (#2247)
2021-12-01 - 799a3c1e3 - Inserted retries to avoid flaky persistence (#2232)
2021-12-01 - 4f28e32b4 - Add dedicated persistence execution mutable state integration tests (#2245)
2021-12-01 - 2a47cdf0e - Replace ldflags with go:embed (#2085)
2021-12-01 - 0d9fcd3eb - Deprecate static cluster metadata (#2231)
2021-11-30 - d49000975 - Upgrade golang to 1.17.3 (#2240)
2021-11-30 - be7482878 - Add dedicated persistence execution mutable state integration tests (#2241)
2021-11-30 - b5e343cea - Remove resource.Resource from temporal services (#2229)
2021-11-30 - 4a4cc58b4 - Add Makefile targets for arm64 (#2201)
2021-11-30 - 1edb0a6b1 - Add dedicated persistence execution mutable state integration tests (#2242)
2021-11-30 - 171d228fc - Remove duplicated history size from execution info (#2239)
2021-11-29 - f888c2840 - Namespace Handover (#2228)
2021-11-29 - a1a237022 - Fix forward / backward compatibility issue (#2237)
2021-11-29 - 6daf3b6fb - Remove unnecessary CreateWorkflowModeContinueAsNew mode (#2235)
2021-11-29 - 6c139012d - Fix ParentClosePolicy not respected cross-namespace (#2224)
2021-11-29 - 1c52e90b6 - Remove unnecessary start version from current execution table (#2236)
2021-11-24 - e741533d4 - Add testhelper package to create temp file and dir (#2214)
2021-11-24 - a0e896524 - Add interleaved weighted round robin scheduler (#2209)
2021-11-24 - 876f75ce7 - Introduce dynamic cluster metadata component (#2208)
2021-11-24 - 182195552 - Namespace state update fix (#2227)
2021-11-24 - 0f2da592b - Add enable remote connection tctl (#2213)
2021-11-24 - 0baca26c8 - Use proper logger for worker service (#2226)
2021-11-23 - fc4024331 - Switch to Admin API in tctl list task queue tasks (#2222)
2021-11-23 - c3fdcb813 - Extract common struct for service provider args (#2223)
2021-11-23 - 90e89dd7c - Create admin API to list shard tasks (#2200)
2021-11-23 - 78acd4d7b - Use context for shard acquisition timeout (#2219)
2021-11-23 - 769b69f41 - Switch to admin API in tctl list shard tasks by category (#2217)
2021-11-23 - 6c86649b9 - Create admin API to get task queue tasks (#2221)
2021-11-23 - 1bbc3fbd7 - Refactor tctl admin show workflow raw history, switch to admin API (#2225)
2021-11-22 - 69959bc5e - Allow namespace state update (#2220)
2021-11-22 - 17b9d4cae - Pin MySQL version to 8.0.19 (#2210)
2021-11-19 - c9579c2ef - Handle missing metrics config (#2205)
2021-11-19 - c24eaa4be - Add List Cluster Metadata API (#2206)
2021-11-19 - 67da8a3b9 - Use gopkg.in/yaml.v3 instead of v2 (#2203)
2021-11-19 - 21531bab5 - Cleanup serverOptions depency and add server metrics client (#2187)
2021-11-19 - 1dd76d3f7 - Wait replications (#2202)
2021-11-18 - e5740fa22 - Use t.Setenv function to set env in unit tests (#2196)
2021-11-18 - 66780893b - Update development environment to MySQL 8.0 (#2193)
2021-11-18 - 6257ac319 - Change shard persistence interface to get-or-create (#2199)
2021-11-18 - 590d9dc98 - Implement GetReplicationStatus (#2195)
2021-11-18 - 39bd317c7 - Update development environment to PostgreSQL 13.5 (#2194)
2021-11-18 - 2eb40fe73 - Add fan out to history client for GetReplicationStatus (#2198)
2021-11-18 - 00cb5194c - Add tiered storage queue processor (#2189)
2021-11-17 - ca556868a - Add GetReplicationStatus to history service (#2186)
2021-11-17 - 9a4982999 - Remove Elasticsearch admin commands (#2181)
2021-11-17 - 14fe930b3 - Switch to admin API in tctl register namespace (#2176)
2021-11-16 - e6757b9ed - Add try lock APIs to priority mutex (#2183)
2021-11-16 - de83ad266 - Add NamespaceValidator interceptor (#2169)
2021-11-16 - 407184d92 - Add IsLocked to priority mutex (#2180)
2021-11-16 - 196b57afc - Create admin API to update namespace (#2182)
2021-11-15 - b9b0615a5 - Create admin API to register namespace (#2175)
2021-11-15 - 58e380a9e - Make cluster metadata schema backward compatible (#2177)
2021-11-15 - 58af5da4d - Header support for signals and queries (#2149)
2021-11-15 - 43ac47ffb - Report acked replication task timestamp (#2173)
2021-11-15 - 170bde6ac - Use admin API for tctl list cluster members (#2162)
2021-11-14 - 6bfe955d7 - Support concurrency flag in the tctl batch command (#2168)
2021-11-12 - f1cbac2de - Disable fx logs (#2091)
2021-11-12 - b9edf16e6 - Create admin API to list namespaces (#2170)
2021-11-12 - a439a679e - Wire up cluster metadata APIs (#2174)
2021-11-12 - 5d6bb62b1 - Update opentelemetry version and add gauge cache. (#2161)
2021-11-12 - 59c571530 - Use admin API for tctl list namespaces (#2171)
2021-11-12 - 519bffb7d - Move DeleteWorkflowExecution into shard context (#2138)
2021-11-12 - 1198962e9 - Create admin API to list cluster members (#2159)
2021-11-11 - f141698a0 - Initialize SignalRequestedIDs correctly (#2167)
2021-11-11 - dc60fffe3 - Fix test behavior (#2166)
2021-11-11 - 725a4de90 - Add new admin APIs to manage remote cluster info (#2163)
2021-11-11 - 60fb3cd65 - Add protection for empty JWT key source URIs (#2142)
2021-11-11 - 3f1eb310e - Set VisibilityTimestamp on ReplicationTask (#2165)
2021-11-11 - 13082f322 - Migrate cluster metadata store to use cluster metadata info table (#2155)
2021-11-11 - 0f70d6192 - Matching GetTasks call reduction (#2164)
2021-11-10 - 8c308a0d7 - Add custom grpc inspectors (#2156)
2021-11-10 - 62ce8c584 - Move namespace name validation to RegisterNamespace (#2160)
2021-11-10 - 590b6eaeb - Add GetAllKeys to keyResolver (#2152)
2021-11-09 - e25f5dc7b - Remove command tctl admin cluster metadata (#2151)
2021-11-08 - f35895c9b - Add RequestID validation for ResetWorkflow (#2141)
2021-11-08 - e9f8078d3 - Extend admin Describe Cluster API response with cluster metadata (#2150)
2021-11-08 - c5baafa11 - Add new table for cluster metadata info and update cluster metadata struct (#2147)
2021-11-08 - 5561f2a48 - Add support exclusion of tags from metrics (#2137)
2021-11-08 - 31a576749 - Fix potential nil pointer use in shard context (#2148)
2021-11-07 - 4fb8100a9 - fix flaky TestActivityHeartbeatFailover (#2145)
2021-11-06 - f8fc78efd - Refactor worker for addsearchattributes using fx (#2144)
2021-11-05 - 62e85b420 - Update error type (#2135)
2021-11-05 - 1e161ebc9 - Add gen-replication workflow (#2139)
2021-11-04 - 6ad2edaf0 - Fix GenerateLastHistoryReplicationTask to use shard.AddTasks() (#2136)
2021-11-04 - 203ccb085 - Refactor shard context lifecycle (#2102)
2021-11-04 - 0917c302b - Remove sampled retention (#2133)
2021-11-03 - f463f0dbc - Move generation of replication task logic to task generator (#2127)
2021-11-03 - 851aaec89 - Use correct absolute paths to scripts in dockerfile (#2123)
2021-11-03 - 67b606425 - Add generic task processor (#2120)
2021-11-03 - 60a52ee1d - update cli usage (#2124)
2021-11-03 - 5acdf8654 - Add GenerateLastHistoryReplicationTasks to history service (#2129)
2021-11-03 - 577de1ad8 - Add worker manager to worker service (#2126)
2021-11-01 - ea45be2d7 - Move timer gate to common package (#2121)
2021-11-01 - c28665dd1 - Promote local namespace to global namespace (#2105)
2021-11-01 - 7e8b9650d - Specify the entrypoint absolutely (#2118)
2021-10-29 - f253ddfc0 - Patch tally import (#2115)
2021-10-29 - ed0743d90 - Remove dependency on resource from worker (#2098)
2021-10-29 - ae959113d - Add http status metric tag to bulk processor errors (#2113)
2021-10-29 - 744f26d28 - Use strong types for namespace Name and Id (#2082)
2021-10-29 - 48f78b781 - Add metrics for loaded task queue count as gauge (#2111)
2021-10-29 - 181ad20ae - Bump Web to v1.13.0 (#2110)
2021-10-28 - fc422185f - Limit Elasticsearch document _id to 512 bytes (#2112)
2021-10-28 - d2405acd2 - Make SQL CLI tool more idempotent (#2104)
2021-10-28 - 3f6253069 - Extract SearchSource object in Elasticsearch client (#2106)
2021-10-28 - 28db61ac2 - Added ability to specify dataconverter port via flag (#2093)
2021-10-28 - 0e67cd904 - Use PPROF_PORT env to configure pprof port in config_template.yaml (#2109)
2021-10-27 - e93f4b23c - Finish task serialization / deserialization (#2083)
2021-10-27 - 88473687e - Update submodules and go.temporal.io/api (#2103)
2021-10-27 - 1503e924e - Upgrade SDK and Tally (#2100)
2021-10-26 - ede54d90f - Add in flight poll activity / workflow task queue metrics (#2095)
2021-10-26 - cb9462ab2 - Shard context cleanups: private eventsCache, remove SetEngine (#2086)
2021-10-26 - af181b404 - Add per namespace burst limit control (#2067)
2021-10-26 - 9c64e2e8d - Remove deprecated task ID field from replication task (#2078)
2021-10-26 - 98814a00e - Remove half done task processing restructure from Cadence (#2074)
2021-10-26 - 8b4936a24 - Remove unnecessary method on history tasks (#2072)
2021-10-26 - 3ce382a90 - Shard context cleanups: [R]Lock/Unlock should be private (#2087)
2021-10-26 - 03fde05be - Move history tasks definition to history package (#2070)
2021-10-26 - 00388db7f - Rename workflow identifier to workflow key (#2071)
2021-10-25 - d83e601c0 - Return deserialization tasks as is from persistence layer (#2042)
2021-10-25 - d7a2d5609 - Post-release tasks (#2092)
2021-10-25 - d6771efef - Pass exit signals to temporal-server (#2069)
2021-10-25 - bdeff26cf - Move from io/ioutil to io and os package (#2062)
2021-10-25 - b3731a43b - Shard context cleanups: split two remove shard paths, don't hold engine in workflow context (#2088)
2021-10-25 - 9a3c57741 - Admin API GetShard (#2075)
2021-10-25 - 92a42adb8 - Allow port forwarding to the tctl web dataconverter (#2089)
2021-10-25 - 779f64591 - Init server via uber/fx (#2034)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.14.0)

Server
Server With Auto Setup
Admin-Tools

Release Highlights

This release includes a hotfix for long Elasticsearch document _id field values. You don't need to upgrade if your Workflow Ids are shorter than 475 bytes.

2021-10-28 - fc422185f - Limit Elasticsearch document _id to 512 bytes (#2112)

Details about v1.13.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.13.1)

Server
Server With Auto Setup
Admin-Tools

Release Highlights

This release does not includes any schema changes.

Notable changes

Expose Closed status of Temporal Cron Workflow Executions and Workflow Executions that retry

Previously Temporal Cron Workflow Executions and Workflow Executions that retried always ended with a "Continued-As-New" status, now they have a final event like any other, with a "Completed" or "Failed" or "Timed out" status. If you have code that depends on the "Continued-As-New" status you'll need to update it.

Custom search attributes changes

String search attribute type was renamed to Text to better reflect its nature and match Elasticsearch type. String is still supported as deprecated alias to Text and will be removed.

Visibility changes

Many small improvements and bug fixes in visibility.

The docker container now runs as a non-root user

You should make sure you're using https://github.com/temporalio/helm-charts/releases/tag/v1.13.0 which contains a corresponding change to the Kubernetes securityContext.

Dynamic config validator

If dynamic config is malformed an error message will be written to the log. In the next version, a malformed dynamic config will prevent the server from starting. If you see error message about dynamic config, please fix it.

SQLite persistence

Many thanks to Jacob LeGrone (DataDog) for his contribution of the SQLite driver (#1968, #2029). NOTE: The SQLite driver is still under development.

Metrics

• Added task_schedule_to_start_latency metric.

Bug fixes

• Correctly set shard ID when range complete replication tasks (#2056)
• Correctly set last event task ID as part of events replication (#2055)
• Fix match task queue rate calculation issue (#1910)

All changes

2021-10-20 - 6cbfa2a3a - Avoid acquiring shared lock recursively (#2076)
2021-10-18 - 92081a5a8 - Handle case of empty string CAs in TLS config (#2063)
2021-10-18 - 87a89e682 - Adding snipsync wrappers to setup commands (#2051)
2021-10-18 - 83c060ac7 - Remove read-through from namespace.Registry (#2066)
2021-10-18 - 7f6a954e0 - Rename INDEXED_VALUE_TYPE_STRING to INDEXED_VALUE_TYPE_TEXT (#2061)
2021-10-18 - 7c83fa4b0 - Avoid generating empty string in tls cert array (#2064)
2021-10-18 - 077d39c77 - Remove awk dependency in auto-setup.sh (#2068)
2021-10-15 - f955633ed - Make shard info to 0 unix nano time, instead of default time.Time (#2052)
2021-10-15 - c121a7d3b - Correctly set shard ID when range complete replication tasks (#2056)
2021-10-15 - 1d7b63b10 - Added php and typescript sdks to supportedClients (#2058)
2021-10-14 - 13fef8a74 - Correctly set last event task ID as part of events replication (#2055)
2021-10-14 - 10df97be0 - Adding temporal-cla and update link (#2053)
2021-10-14 - 0a92fc6d7 - Sync Elasticsearch index create script between Makefile and auto-setup.sh (#2047)
2021-10-14 - 043c97498 - add optional explicit ForceTLS option to ClientTLS config (#2036)
2021-10-13 - 861df80f1 - Unify workflow completion metrics (#2050)
2021-10-12 - cbe6c18c5 - Add metrics for workflow_continued_as_new counter (#2048)
2021-10-12 - c18e7cc30 - Use uber/fx to init worker service (#2041)
2021-10-12 - 4c2fa1794 - Remove duplicate code (#2046)
2021-10-12 - 467e5291a - Persistence rate limit error (#2049)
2021-10-12 - 1a811e079 - Refactor visibility manager (#2045)
2021-10-11 - 4ee976855 - Use time.Unix(0, 0) instead of time.Time (#2044)
2021-10-11 - 00f8c7cb6 - Use Match query instead of MatchPhrase for Elasticsearch (#2037)
2021-10-09 - e08cd0541 - Revert "Use uber/fx to init worker service (#2032)" (#2039)
2021-10-08 - f687b5b34 - Use time.Unix(0, 0) instead of time.Time (#2038)
2021-10-08 - a3649987e - Use uber/fx to init worker service (#2032)
2021-10-08 - 86d715f22 - Fix buffered query timeout issue (#2033)
2021-10-08 - 53f438a5f - Use uber/fx to init frontend (#2023)
2021-10-08 - 3e5196b1b - Log error message during server startup if dynamic config is misconfigured (#2019)
2021-10-08 - 08b48711b - Add schema setup and namespace registration utilities for SQLite (#2029)
2021-10-07 - fdea07050 - Remove video.png (#2028)
2021-10-07 - b621a6546 - Convert namespace.Registry to read-through (#2020)
2021-10-07 - 99355bfb1 - Bump Web to v1.12.0 (#2031)
2021-10-07 - 6848bf1ba - Remove unused field (#2027)
2021-10-07 - 25fec58e8 - Use di for matching integration tests (#2024)
2021-10-05 - ad4dc3f1f - Remove task polling recursion (#2022)
2021-10-04 - bf2e5a653 - Remove namespace/ dependence on cluster metadata (#2011)
2021-10-04 - 2c52fd617 - Increase add-search-attributes tctl admin command timeout to 30s (#2021)
2021-10-04 - 25f551b59 - Separate Cassandra persistence related structs (#2016)
2021-10-04 - 06b863741 - Namespace type renames (#2018)
2021-10-03 - 8fdc8992e - Escape password in connection url (#2015)
2021-10-01 - fd4e2ca2d - Use DI for matching service initialization (#1995)
2021-10-01 - f831481fc - Fix bulk processor NPE (#2013)
2021-10-01 - e80e8c1a0 - unlimited MaxConnectionAge (#1998)
2021-10-01 - d9bc815d9 - Make compare function easy to use for task key (#2008)
2021-10-01 - c7b568e97 - Allow setting workflow run timeout in tctl (#2009)
2021-10-01 - 531cc9664 - Use StartedIdentity as LastWorkerIdentity for PendingActivity if no retry (#2007)
2021-10-01 - 37799989a - Fix elasticsearch_bulk_processor_wait_add_latency metric (#2012)
2021-10-01 - 27e9ff249 - Namespace Cache refactor to take advantage of immutability (#2006)
2021-10-01 - 16b126699 - Revert "unlimited MaxConnectionAge (#1998)" (#2010)
2021-09-30 - fff6fbb85 - Add elasticsearch_bulk_processor_duplicate_request metric (#2003)
2021-09-30 - f5ad4da3c - Add SDK commands sequence validation (#1993)
2021-09-30 - e0eb82daf - Use DI to inject new cache provider (#1997)
2021-09-30 - dc4cd740d - Remove namespace.CacheEntry.GetInfo() (#2000)
2021-09-30 - 6d8338aa2 - Remove advanced visibility specific errors (#2002)
2021-09-30 - 6120b46c4 - Remove namespace.CacheEntry.GetConfig() (#1991)
2021-09-30 - 57985cbdf - Improve mapper tests and comment message (#2005)
2021-09-30 - 45a3b798c - Correctly set non timer task key (#2004)
2021-09-30 - 26882f188 - Remove cache.Cache from workflow.Cache (#2001)
2021-09-29 - ae8c7978a - Remove CacheEntry.GetReplicationConfig (#1976)
2021-09-29 - 8df4cc86e - Improve message on injected errors (#1973)
2021-09-29 - 64ef4e223 - Add SQLite storage driver (#1968)
2021-09-29 - 3f29b8f7b - Add workflow identifier to all tasks (#1989)
2021-09-29 - 2e8f38e5c - Make workflow.Cache interface (#1990)
2021-09-27 - f31bc0b8a - Unify task serialization / deserialization (#1985)
2021-09-27 - bcf518bd9 - Utilize task serialization / deserialization util (#1987)
2021-09-27 - ab32caf07 - Add persistence/visibility store info to GetClusterInfo (#1984)
2021-09-27 - a63db6682 - Improve invalid search attribute error message (#1986)
2021-09-27 - 7af4d6577 - Move task definition from persistence layer to business logic layer (#1983)
2021-09-27 - 180db9247 - Update proto submodule (#1982)
2021-09-24 - f0d4f89a5 - Make new events for retry compatible with older SDK versions (#1949)
2021-09-24 - c7994e228 - Reorganize go.mod file to follow 1.17 convention (#1974)
2021-09-24 - 60a6a61ca - Add feature flag to enable Ringpop TLS (#1967)
2021-09-24 - 57938c28b - Initialize resource via DI (#1970)
2021-09-24 - 07cae17d8 - Add more details to invalid command attributes error (#1978)
2021-09-23 - ed49957ce - Update dashboards (#1969)
2021-09-23 - cc1370f3c - Remove default page size from Elasticsearch visibility store (#1971)
2021-09-23 - a38dbd10c - Use internode mTLS for ringpop (#1902)
2021-09-23 - 3f1aaa252 - Convert Elasticsearch BadRequest error to InvalidArgument error (#1573)
2021-09-23 - 0652f4106 - Set timeout for all Elasticsearch read calls (#1972)
2021-09-22 - d79e55874 - Return memo serialization error instead of logging (#1948)
2021-09-22 - d23e605eb - Construct history.Service via DI (#1964)
2021-09-22 - 81db9bf02 - Add more metrics for mutable state update (#1960)
2021-09-22 - 6356d64e9 - Move Elasticsearch client config to elasticsearch/client package (#1950)
2021-09-22 - 3cd95674b - Plumb fault injection for use in integration tests (#1957)
2021-09-22 - 302d03ba1 - Revert "Remove deprecated metrics config code as per release 1.11 note (#1959)" (#1961)
2021-09-22 - 220795cf3 - Integrate dashboards submodule (#1965)
2021-09-21 - f769507d6 - Move SearchAttributesManager to searchattribute package (#1953)
2021-09-21 - f2c216c44 - Consistent TaskQueueManager fatal signaling (#1933)
2021-09-21 - b20a256e8 - Make ResilientSyncMatch the default behavior (#1955)
2021-09-21 - 923a45c3f - Remove standard logger dependency (#1951)
2021-09-21 - 8dfbb7405 - Clean up worker README (#1954)
2021-09-21 - 67b416cc9 - Use internal error to indicate code bug / contract breach (#1958)
2021-09-21 - 584f28272 - Don't refresh search attributes cache if database is down (#1956)
2021-09-21 - 183739dcc - Remove deprecated metrics config code as per release 1.11 note (#1959)
2021-09-20 - d9162f358 - Fix coverage profile generator (#1947)
2021-09-20 - d6c9b4af8 - Add custom logger option to schema tool (#1943)
2021-09-20 - ce4f0364b - Use internal error to indicate code bug / contract breach (#1945)
2021-09-20 - 58393935e - Refactor visibility stores and managers (#1942)
2021-09-20 - 1ec6e9f1a - Ignore goveralls exit code (#1946)
2021-09-17 - ddb37af33 - Use persistence mutable state mutation / snapshot binary size util (#1935)
2021-09-17 - 8b71d20d9 - Remove deprecated history events from mutable state (#1937)
2021-09-17 - 08a15e384 - Fix comments for NewExecutionStore (#1939)
2021-09-16 - c73460c27 - Always return nil value for NextPageToken if there are no more pages (#1936)
2021-09-16 - 8b1993b7e - Use internal error to indicate code bug / contract breach (#1924)
2021-09-16 - 7223e7a53 - Update integration tests (#1934)
2021-09-16 - 463b75820 - Remove unused code (#1938)
2021-09-15 - ebe540796 - Use internal error to indicate code bug / contract breach (#1919)
2021-09-15 - d588c3ef3 - Simplify some uses of payloads.Decode in integration tests (#1889)
2021-09-15 - ba3943e86 - Remove taskQueueManager by instance rather than queue ID (#1931)
2021-09-15 - 9bac95d6d - Merge persistence API for append history and conflict resolve (#1926)
2021-09-15 - 847c449b3 - Add persistence mutable state mutation / snapshot binary size util (#1925)
2021-09-15 - 845c4e7fc - Extract VisibilityManager from Resources (#1932)
2021-09-15 - 6ed17e325 - Reset gauge on scan complete (#1930)
2021-09-15 - 340949ff3 - Unify visibility metrics (#1928)
2021-09-15 - 1ce94933b - Update event blog size warning message (#1923)
2021-09-15 - 117fd3ae1 - Set timeout for TestCluster (#1927)
2021-09-14 - fa4a34dc7 - Add schedule to start task metric (#1915)
2021-09-14 - ba4197f40 - Use point in time with Elasticsearch v7 only if it is available (#1918)
2021-09-14 - ab8c0f36d - Update SDK to v1.10.0 (#1921)
2021-09-14 - 032ce4e05 - Remove dead code (#1920)
2021-09-13 - 96de6faec - Unload taskQueueManager by instance, not queue ID (#1917)
2021-09-13 - 469b4e616 - Use serviceerror.Unavailable error type instead of serviceerror.Internal (#1916)
2021-09-11 - ba5be8004 - Ignore InvalidArgument error for unknown custom search attributes (#1911)
2021-09-11 - 09e682a9c - Remove missing keyword support in visibility queries (#1912)
2021-09-10 - 90d08e4de - Make TestContinueAsNewWorkflow_Timeout faster (#1908)
2021-09-10 - 56d1f9aa3 - Fix match task queue rate calculation issue (#1910)
2021-09-10 - 4cb5ae03c - Wait for namespace cache refresh without sleeping in integration tests (#1907)
2021-09-09 - e84ef4084 - Adds docs and examples to the goro package (#1900)
2021-09-09 - 12e6347af - Add metrics to bulk processor (#1909)
2021-09-09 - 0d6006624 - Additional doc updates for package goro (#1906)
2021-09-08 - ef889362a - Ack existing visibility task in case of duplicates (#1905)
2021-09-08 - b113b214d - Add goro.Group and use it in taskReader (#1894)
2021-09-08 - 32b36068c - Remove openssl from base server image (#1901)
2021-09-07 - ee93ab0c7 - Switch memo flags to StringSlice (#1859)
2021-09-07 - e81657bd3 - Fix TaskWriter panic when shutdown (#1892)
2021-09-07 - 9016e5dc4 - Remove retry from DeleteExecution (#1896)
2021-09-03 - f2b317b56 - Add searchattribute.Mapper interface for custom search attributes (#1880)
2021-09-03 - 4fb055503 - Move namespace cache into namespace package (#1884)
2021-09-03 - 263797637 - Paul MS cache update (#1887)
2021-09-02 - a447f2e1a - Correctly set current version if workflow is finished (#1879)
2021-09-02 - 32fd7ddd0 - Refactor Elasticsearch visibility store (#1888)
2021-09-02 - 24a081a2b - Remove search attributes errors logs (#1886)
2021-09-01 - 9509717d8 - Remove non-retriable case from fault injection (#1877)
2021-09-01 - 42bf6ef02 - Change parseESDoc and generateESDoc to return error instead of logging (#1878)
2021-08-31 - b37a91d91 - Add non-root user to run temporal server (#1814)
2021-08-31 - 988d865f8 - Patch metrics UserScope (#1876)
2021-08-31 - 811cc8bef - Expose status of cron and retry workflow executions (#1866)
2021-08-31 - 7fc934022 - Change custom search attributes (#1870)
2021-08-31 - 67c950d70 - Update dependencies (#1874)
2021-08-31 - 52be73e49 - Bump current version to 1.13.0 (#1873)
2021-08-31 - 24898d059 - Fix TestCronWorkflow and TestCronChildWorkflowExecution integration tests (#1872)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.13.0)

Server
Server With Auto Setup
Admin-Tools

Release Highlights

• Elasticsearch v7 is now supported: #1164 (see details below).
• Internal visibility queue is used by default (not Kafka): #1235 (see details below).
• Metrics improvements: #1216, #1223, #1227, #1231, #1232, #1245.
• Schema changes.
• Bug fixes.

Schema changes

This release includes schema changes.

Before upgrading your Temporal Server cluster, run the schema upgrate tool to support version 1.4.
Schema update tool instructions are here: https://docs.temporal.io/docs/server-versions-and-dependencies/#upgrade-your-version-of-temporal.

Elasticsearch v7 Support

Upgrading to Elasticsearch v7 (ES7) from ES6 includes a number of breaking changes.
The most important one is the removal of types.
Therefore the ES6 Go client is not compatible with ES7 and upgrading the Temporal Server is a multi step process.

Fresh install

If you are installing a fresh version of the Temporal Server, use ES7.
Install the latest ES7 version, and set the ES version in the Server's static config:

persistence:
  datastores:
    es-visibility:
      elasticsearch:
        version: "v7"

Or, set ES_VERSION env to v7 if you are using pre-build docker image.

Upgrade with temporary cluster shutdown

If you can afford any Temporal Server downtime while upgrading Elasticsearch, then simply shutdown your Temporal cluster and upgrade ES6 to ES7 following the steps from the Elasticsearch documentation.
If you are on the latest 6.8.x, just perform a rolling upgrade of Elasticsearch.

Rolling upgrade of the Temporal Server

You can also upgrade the Temporal Server cluster with some minimal downtime in enhanced visibility only.

1. Update the Server to this v1.7.0 release.
2. Add the following to the dynamic config:

history.visibilityProcessorEnabled:
  - value: false

3. Restart the Server.
   Workflow visibility information won't be updated.

4. Upgrade ES6 to ES7 following steps from the Elasticsearch documentation.
   If you are on the latest 6.8.x, just perform a rolling upgrade of Elasticsearch.

5. Start ES7.
   Visibility read queries will temporarily generate errors.
   Write queries are blocked because the processor is disabled.

6. Switch to ES7 in the Server's static config:

persistence:
  datastores:
    es-visibility:
      elasticsearch:
        version: "v7"

Or, set ES_VERSION env to v7 if you are using pre-build docker image.

7. Remove this from the dynamic config:

history.visibilityProcessorEnabled:
  - value: false

8. Restart the Server one more time.

Managed AWS Elasticsearch version

During our internal testing we found out that the latest AWS Elasticsearch version 7.9 has issues and can't be used.
If you are using AWS Elasticsearch upgrade to 7.7.

Internal visibility queue is used by default

This release uses internal visibility queues by default, instead of Kafka.
You may lose unprocessed Kafka messages if you are upgrading from version other than v1.6.4.
To prevent this, upgrade to v1.6.4 first or follow the upgrade steps from the v1.5.0 release.

In the next release all remaining Kafka code will be completely removed.

Change details

2021-01-21 - 1428a3da - Set default ES bulk processor ack timeout to 1 minute (#1210)
2021-01-21 - b46fbe3f - Add ES logger (#1209)
2021-01-22 - 3aa0d683 - Update Go SDK to 1.4.0 (#1211)
2021-01-22 - 95c1b5bc - Remove support for locking down admin operations (#1194)
2021-01-22 - d2fc34b6 - Improve PostgreSQL config (#1214)
2021-01-22 - f6225dea - Fix NDC metrics reporting (#1216)
2021-01-25 - 08252071 - Add more logging to ES (#1222)
2021-01-25 - 1e7c1903 - Fix metrics scope namespace tag initialization (#1223)
2021-01-25 - a4d1428c - Bugfix: activity heartbeat timeout (#1220)
2021-01-26 - 089f746c - Improve ES logging (#1228)
2021-01-26 - 30aed765 - Fix metrics tag name (#1227)
2021-01-26 - ba7be16d - Fix replication failure metrics (#1232)
2021-01-26 - cce754ac - Fix parent close metrics definition (#1231)
2021-01-26 - ec338589 - Bugfix: close execution request goes to Kafka even when visibility queue type is set to internalWithDualProcessor (#1229)
2021-01-27 - 485ae998 - Accept empty Kafka config when visibility queue is set to internal (#1238)
2021-01-27 - 532b3181 - Cleanup DB schema (#1130)
2021-01-27 - e12c7a15 - Switch visibility queue default to internal (#1235)
2021-01-28 - d3bc2a92 - Bugfix: NoopResolver was always used for version compatibility check (#1233)
2021-01-29 - ce445321 - Add password masker (#1217)
2021-02-01 - 23e5b8b0 - Fix distribution metrics issue (#1245)
2021-02-01 - 43954fdc - Adjust number of histogram buckets (#1248)
2021-02-01 - 520de19f - Correctly initialize prometheus reporter (#1250)
2021-02-02 - 69da5619 - Bugfix: Allow enabling TLS for sql tool without requiring key/cert. (#1252)
2021-02-02 - 9d9d8bf4 - Add server option for custom Elasticsearch HTTP client (#1246)
2021-02-02 - f3e33a95 - Set search.idle.after to 365 days (#1254)
2021-02-03 - 10884f0a - Deprecate token bucket (#1253)
2021-02-03 - 3e9e452d - Replace deprecated NewExitError with Exit method (#1257)
2021-02-03 - f7f9ac1c - Simplify matching engine rate limit logic (#1260)
2021-02-04 - 10205def - Improve matching engine error type (#1261)
2021-02-05 - c5ee4750 - Add logger to matching service (#1264)
2021-02-08 - e831c9d6 - Refactor TLS tests (#1266)
2021-02-09 - 7ea0ee04 - Add background executions table scanner (#1244)
2021-02-09 - 83018043 - Fix update schema command for temporal_visibility (#1268)
2021-02-09 - f6e2c136 - Create non-transient workflow task when version changes (#1272)
2021-02-10 - cf204719 - Set connection timeout for cassandra tool (#1277)
2021-02-11 - afc00012 - Add quotes to config_tempate.yaml (#1284)
2021-02-11 - c977c8b8 - Rename ArchiveVisibilityRequest to VisibilityRecord (#1281)
2021-02-12 - 4363aca9 - Use document Id to match request and response in ES processor afterFunc callback (#1265)
2021-02-12 - ee845110 - Copy visibilityAckLevel in copyShardInfo (#1287)
2021-02-16 - 289e1de9 - Enable dynamic update of TLS certificates (#1288)
2021-02-16 - e3496b1c - Add protection on child workflow ID (#1293)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.7.0)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes hotfix for the long Elasticsearch document _id field value. You don't need to upgrade if your Workflow Ids are shorter that 475 bytes.

2021-10-28 - fc422185f - Limit Elasticsearch document _id to 512 bytes (#2112)

Details about v1.12.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.12.4)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes hotfix for the matching engine and few improvements for the visibility metrics.

2021-10-07 - 037892933 - Unify visibility metrics (#1928)
2021-10-07 - 11f35c837 - Add metrics to bulk processor (#1909)
2021-10-07 - 63fe119da - Fix elasticsearch_bulk_processor_wait_add_latency metric (#2012)
2021-10-07 - 9c5468e37 - Fix bulk processor NPE (#2013)
2021-10-07 - d10b98be7 - Add elasticsearch_bulk_processor_duplicate_request metric (#2003)
2021-10-07 - f225894c9 - Remove task polling recursion (#2022)

Details about v1.12.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.12.3)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes small bug fix for custom search attributes custom mapper. You don't need it if you are not using custom mapper.

2021-09-11 - ba5be8004 - Ignore InvalidArgument error for unknown custom search attributes (#1911)

Details about v1.12.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.12.2)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes bug fixes for matching service and advanced visibility:

2021-09-16 - d8a538da2 - Bump current version to 1.12.1
2021-09-16 - b4b0ce8dd - Use point in time with Elasticsearch v7 only if it is available (#1918)
2021-09-16 - 7704dd2ee - Remove taskQueueManager by instance rather than queue ID (#1931)
2021-09-16 - 75a1a7d5e - Unload taskQueueManager by instance, not queue ID (#1917)
2021-09-16 - eb606bba0 - Fix match task queue rate calculation issue (#1910)
2021-09-10 - 077dd42c6 - Ack existing visibility task in case of duplicates (#1905)
2021-09-10 - 7af6ad332 - Fix TaskWriter panic when shutdown (#1892)
2021-09-10 - bb10f2d97 - Refactor Elasticsearch visibility store (#1888)
2021-09-10 - 1de4b2fac - Remove search attributes errors logs (#1886)
2021-09-10 - 69515153a - Add searchattribute.Mapper interface for custom search attributes (#1880)
2021-09-10 - 59c3ccbb8 - Change parseESDoc and generateESDoc to return error instead of logging (#1878)
2021-09-10 - 2b415aa6f - Patch metrics UserScope (#1876)
2021-09-10 - d817cb1c6 - Change custom search attributes (#1870)

Details about v1.12.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.12.1)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release does not includes any schema changes.

Notable changes

Enhanced Visibility is production ready

We are happy to officially announce that Enhanced Visibility is now production ready!

A number of new updates have been added to the Enhanced Visibility feature over the past several releases.
This release adds a few more updates (see below) and brings them all together without any breaking changes.

Elasticsearch v7 is now default

If you still use Elasticsearch v6 and don't set ES_VERSION env or version property in config file explicitly, you have to do it now.

admin elasticsearch report has been removed

Please use Elasticsearch directly to build custom reports.

Query support for ColumnName=missing syntax is deprecated

Query support for ColumnName=missing syntax will be removed in the next release. Use ColumnName is null syntax instead. Use ExecutionStatus="Running" to filter only running workflows.

Elasticsearch healthcheck is disabled by default

You can turn it (and sniffing) on with new config settings added in #1865. Leave it disable if you use AWS Elasticsearch (https://github.com/olivere/elastic/issues/1091).

All changes

2021-08-31 - 6867b1c3f - Add metrics client to custom data store factory (#1871)
2021-08-27 - 59dd702de - Fix FE interceptor order to report metrics correctly (#1863)
2021-08-27 - 431127c02 - Reserve Temporal prefix for custom search attribute names (#1861)
2021-08-27 - 33f95d7b5 - Change default value of worker.ESProcessorFlushInterval from 200ms to 1s (#1860)
2021-08-27 - 2e50ce068 - Add closeIdleConnectionsInterval, enableSniff, and enableHealthcheck to Elasticsearch config (#1865)
2021-08-26 - ff069578a - Revert "Split cron and retry paths from continue-as-new (#1745)" (#1858)
2021-08-26 - ec03542e1 - Add version to event cache key (#1838)
2021-08-26 - da5bcc73e - Remove esql package (#1857)
2021-08-26 - cae0d7947 - Split cron and retry paths from continue-as-new (#1745)
2021-08-26 - acea4e9a9 - Remove query validator (#1856)
2021-08-26 - abcee70ec - Replace DSL parsing with query builder (#1855)
2021-08-26 - 9955dc5f7 - Merge AppendHistory with CreateWorkflow (#1853)
2021-08-26 - 7d423ca6b - Remove 'admin elasticsearch report' CLI command (#1854)
2021-08-24 - eaa7d7b15 - Expose cassandra.CheckCompatibleVersion() (#1849)
2021-08-24 - c11a3e169 - Allow external persistence implementations to reuse persistenceTestBase by exporting a factory method accepting PersistenceTestCluster (#1846)
2021-08-24 - 4f8c1c77e - Fixes bug where intra-node client is doing retries even if the error is context cancelled (#1845)
2021-08-24 - 2d6f48dcf - Close factory after use (#1850)
2021-08-23 - d4f3bc665 - Spin loading initial taskWriter state in Start() (#1842)
2021-08-23 - 1f24996b9 - UpdateWorkflow transaction to include append history (#1841)
2021-08-20 - 514776cfa - Add fault injection for persistence API (#1820)
2021-08-19 - 734e4a24d - Add more metrics to Elasticsearch bulk processor (#1839)
2021-08-19 - 373d0c828 - Streamline taskQueueManager startup (#1834)
2021-08-18 - d95e21cc9 - Fix crash when no pollers on task queue. Dedupe code. (#1835)
2021-08-18 - 8f6496c35 - Add custom SQL to Elasticsearch query converter (#1824)
2021-08-18 - 451004bff - Validate ids in event cache (#1837)
2021-08-18 - 182ae5a39 - Fix ExecutionTime backfill by changing init order (#1836)
2021-08-18 - 0cef628ee - Add tctl-authorization-plugin to docker images (#1832) (#1833)
2021-08-16 - aa9a9a862 - Fix execution time not set correctly during continue as new (#1828)
2021-08-16 - 39fd4ed55 - Do not exit CLI if task queue has no poller (#1827)
2021-08-14 - 3997ae6b5 - Only reapply current workflow events if current workflow is part of continue as new chain (#1822)
2021-08-13 - 4741ef1bc - Update link to buildkite badge (#1823)
2021-08-10 - 95524cb05 - Patch fossa install (#1815)
2021-08-10 - 89fe351d5 - Fix service_errors_nondeterministic merics (#1726)
2021-08-10 - 3595e6243 - Fix API doc comment for authorization.CallTarget.Request (#1816)
2021-08-09 - f9e8559b4 - Udpate fossa cli (#1811)
2021-08-09 - c4d3c34f7 - Adds flag to tctl db-based commands to override TLS ServerName setting #1813
2021-08-09 - a6e9c0567 - NewFactory with given session (#1812)
2021-08-08 - 6d1c70d68 - Revert "Add tctl config feature (#1790)" (#1810)
2021-08-07 - 438b3228c - Fix TaskQueueLock starvation (#1809)
2021-08-06 - e834f2aa6 - Update SDK to 1.9.0 (#1808)
2021-08-05 - c158bfe08 - Make sure shard acquirement channel will not be blocked (#1806)
2021-08-05 - a1b4b0673 - Upgrade gocql (#1805)
2021-08-05 - 69ddc6f82 - Enable fossa test and add delay between analyze and test (#1802)
2021-08-05 - 46a8766e9 - Handle unreliable gocql close function stuck issue (#1803)
2021-08-05 - 2e3fd6ab0 - Optimize matching ack manager loop perf (#1804)
2021-08-05 - 0f4bdf5fc - Remove unnecessary shard controller prepare to stop logic (#1807)
2021-08-04 - c630240b0 - Add retries for tctl visibility APIs (#1798)
2021-08-04 - 3e93f464c - Expose TLS RefreshInterval & ExpirationChecks via env var (#1800)
2021-08-04 - 149072c37 - Update fossa cli in base-ci-builder image (#1801)
2021-08-02 - fe0575130 - Add system worker execution & poller dynamic configs (#1796)
2021-08-02 - d7e24101f - Disallow fetching CA cert over HTTP (#1794)
2021-08-02 - a4ce3a1e2 - Merge HistoryManager into ExecutionManager (#1792)
2021-08-02 - 800768f58 - FetchCACert now supports URLs in addition to files (#1773)
2021-08-02 - 53d107c17 - Revert "Release new tctl UX (#1789)" (#1795)
2021-08-02 - 5374709fc - Release new tctl UX (#1789)
2021-08-01 - d5dbf4e51 - Disable sniffing on ES7 client used in CLI (#1784)
2021-08-01 - 25e22aee2 - Execution store no shard id2 (#1791)
2021-08-01 - 248550b32 - Rewrite persistence create workflow API error handling (#1770)
2021-07-30 - b557462ad - Report service user latency metric explicitly (#1787)
2021-07-30 - 9f9e8f8d0 - Matching deadlock fix and GC refactor (#1785)
2021-07-30 - 68e9adba8 - Add tctl config feature (#1790)
2021-07-30 - 5c777e717 - Rename WorkflowStore to ExecutionStore (#1786)
2021-07-30 - 4e7f6f9a4 - Adjust system worker poller number (#1788)
2021-07-30 - 094f860b2 - Add task latency without user induced latency (#1777)
2021-07-29 - d4b7671eb - Include root cause reasons in Elasticsearch error message (#1774)
2021-07-29 - c74834fea - WorkflowStore interface (#1781)
2021-07-29 - b765bbb97 - Revert part of matching cleanup (a18bb3abd) (#1779)
2021-07-29 - b241ae765 - Remove GetClosedWorkflowExecution method from visibility (#1782)
2021-07-29 - a18bb3abd - Matching cleanup (#1708)
2021-07-29 - 8eff0b728 - Use PIT instead of scroll for Scan API (#1772)
2021-07-29 - 7799e6c3f - Test-friendly refactorings in common (#1771)
2021-07-29 - 557b5d721 - Move visibility persistence to visibility package (#1760)
2021-07-28 - ced5fc507 - Set longer task timeout to avoid stampede on history service (#1776)
2021-07-27 - 42b8d1ee8 - Remove 1.10 backward compatibility code (#1764)
2021-07-26 - fb839418e - Switch default Elasticsearch version from v6 to v7 (#1763)
2021-07-26 - 77db3b7aa - Downgrade namespace logger messages from Info to Debug (#1755)
2021-07-26 - 2910c38ca - Correctly reapply buffered events from current workflow during workflow reset (#1762)
2021-07-26 - 18faf0078 - Tag Web v1.11.0 (#1766)
2021-07-26 - 08a4a6b24 - Delete visibility documents with admin delete command (#1767)
2021-07-23 - 6e417e27a - Extract workflow notification into dedicated util (#1761)
2021-07-23 - 32d0786fb - Extract workflow & events persistence ops into dedicated util (#1759)
2021-07-22 - eeb9adb2d - Clean up plugin interface (#1709)
2021-07-22 - 3faa5454a - Remove unused history manager reference from history engine (#1750)
2021-07-21 - ea4fd1f37 - Only unload TaskQueueManager if ownership is lost or queue traffic drops (#1730)
2021-07-21 - 9d023eef5 - Rename internal API names (#1749)
2021-07-21 - 71d548ff9 - Fix few typos (#1748)
2021-07-21 - 5d40f91f6 - Split workflow context logic into 2 files (#1751)
2021-07-20 - d2788a7c4 - Use filter context instead of query context for exact match queries (#1723)
2021-07-20 - adf692b6b - Always use search_after for Elasticsearch pagination (#1724)
2021-07-20 - a8f7cd9f7 - Add JSON print to get-search-attributes admin command (#1731)
2021-07-20 - 79fa9b3db - Bump version for next release (#1746)
2021-07-20 - 778b5059b - Extend CurrentWorkflowConditionFailedError to include more information about current workflow (#1747)
2021-07-20 - 423ef7b08 - Add skip-schema-update flag to add-search-attributes tctl admin command (#1733)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.12.0)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes bug fixes mainly for cron schedule:
2021-08-18 - 4f5d4c28a - Fix ExecutionTime backfill by changing init order (#1836)
2021-08-18 - 6ef5257fc - Fix execution time not set correctly during continue as new (#1828)

Details about v1.11.x release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.4)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes 2 bug fixes:

• Make sure shard acquirement channel will not be blocked (#1806)
• Handle unreliable gocql close function stuck issue (#1803)

Details about v1.11.x release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.3)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

IMPORTANT

The release v1.11.2 reverts two PRs to fix a deadlock issue introduced in v1.11.0 and v1.11.1.

• revert of #1699
• revert of #1697

The reason for reverting #1699 is to allow a clean revert of #1697.
#1697 introduced a deadlock issue that can cause the entire matching service to be unavailable if the database is not stable.

Schema Changes

This release includes minor schema changes for MySQL/PostgreSQL and no schema changes for Cassandra.
Before upgrading your Temporal Server cluster, run the schema upgrade tool to support version 1.6.

Instructions for the schema update tool are here:
https://docs.temporal.io/docs/server/versions-and-dependencies/#upgrade-your-version-of-temporal.

Notable Changes

Deprecating services.<service>.metrics config

We currently have two ways to configure metrics for Temporal Server: one with a global.metrics section, another in services.<service>.metrics. The second option (in services.<service>.metrics) is deprecated and will be removed in v1.13.0.

Case-insensitive dynamic config

Dynamic config is now case insensitive.

Enhanced Visibility

Important: If you use the Enhanced Visibility feature backed by Elasticsearch, manual steps are required before upgrading to version 1.11. This release includes changes to the schema of the Elasticsearch index. Follow the steps below to change your schema prior to upgrading.

If you do not use the Enhanced Visibility feature, you can skip this section.

What you should know

Version 1.11 requires a breaking change to the Elasticsearch schema used by our experimental Enhanced Visibility feature. Manual migration is required and should take between 1 and 8 hours for most users with minor service degradation.

• What migration is being done? A new schema template and associated index has to be created and all documents need to be copied from the old index to the new one.
• Why was this needed? We did not make this breaking change lightly. Enhanced Visibility is an experimental feature that does not fall under our versioning policy. This was an internal schema refactor necessary in advance of promoting this to an officially supported, production ready feature in the next Temporal release.
• Minor degradation of service. During reindexing, the visibility feature is "frozen" — no updates will be shown when queried.
• Rare upgrade. We don't plan any breaking changes to the Elasticsearch schema in the next year.
• Duration. This migration will take some time — the actual speed depends on your setup. We estimate that most migrations should take between 1 and 8 hours.
• Rollbacks. We offer a recovery strategy in case you need to roll back the upgrade.
• Optional recommendations
  • Scale up Elasticsearch. The migration will be faster if you scale up your Elasticsearch cluster before migrating.
  • Practice run. You can run on a test cluster with one of our Docker images.

How to change your Elasticsearch schema

1. Before you upgrade your server, you need some files from this 1.11.2 release handy. Also you need direct access to Elasticsearch server. If you can't access it from your local machine, the admin-tools container is a good place to run all the steps. The container has all the needed tools, and you need to copy only these files (with kubectl cp, for example).

2. Create an index template (tune it if you use more replicas or shards):

   $ curl -X PUT "http://elasticsearch:9200/_template/temporal_visibility_v1_template" -H "Content-Type: application/json" --data-binary @index_template_v7.json --write-out "\n"
   {"acknowledged":true}
   

   Use index_template_v6.json if you are still on Elasticsearch 6.

3. Create an index. You can choose any name but it must start with temporal_visibility_v1. The default is temporal_visibility_v1_dev:

   $ curl -X PUT "http://elasticsearch:9200/temporal_visibility_v1_dev" --write-out "\n"
   {"acknowledged":true,"shards_acknowledged":true,"index":"temporal_visibility_v1_dev"}
   

4. Before you start reindexing, you need to block internal visibility queue processor to prevent new documents being created or deleted during reindexing. You can achieve this with a dynamic config setting:

   history.visibilityTaskWorkerCount:
     - value: 0
       constraints: {}
   

   If you added custom search attributes, you need to also add them to your dynamic config:

   frontend.validSearchAttributes:
     - value:
         OrderAmount: "Double"
         ProductId: "Keyword"
   

   You can check your custom search attributes by using the following command:

   $ tctl admin cluster get-search-attributes
   Custom search attributes:
   +-------------+----------+
   |    NAME     |   TYPE   |
   +-------------+----------+
   | OrderAmount | Double   |
   | ProductId   | Keyword  |
   +-------------+----------+
   ...
   

   Please note, full server restart (all nodes) is required for this setting to apply. After this step, visibility updates will stop getting written.

5. Run the reindex.sh script to reindex the old index to the new one. The script has input parameters with reasonable defaults. Please review the Input parameters section in reindex.sh. For example, if you are not using default index names, you need to pass a name. If you are on Elasticsearch 6, pass ES_VERSION=v6.

   If your index has a big number of shards (such as 100) you should specify SLICES_COUNT parameter with some reasonable value (such as 10), especially if you use AWS Elasticsearch to prevent 429 errors during reindexing.

   If you added custom search attributes, you need to pass their names to reindex.sh in a JSON array format. The default is 6 Custom*Field attributes. If you don't use them, pass an empty JSON array ('[]').

   $ CUSTOM_SEARCH_ATTRIBUTES='["ProductId", "OrderAmount"]' ES_SERVER=elasticsearch ES_VIS_INDEX_V0=temporal-visibility-prod ES_VIS_INDEX_V1=temporal_visibility_v1_prod ./reindex.sh
   

   The reindex.sh script has two steps: create custom search attribute fields in the new index, and start the reindexing process itself. Both steps show you the generated JSON and ask for confirmation. These steps are idempotent; if, for some reason, reindexing fails, you can run it again.

   Reindexing is an asynchronous operation that might take a while depending on your index size, index configuration, Elasticsearch load and configuration. Average speed is approximately 500K documents per minute. Expected output should be:

   Started reindex task 8z3GRTmSRyCtGPAKmO2RRA:155347. Check status with:
       curl http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347
   Query subtasks with:
       curl http://127.0.0.1:9200/_tasks?parent_task_id=8z3GRTmSRyCtGPAKmO2RRA:155347
   Cancel with:
       curl -X POST http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347/_cancel
   
   Waiting for reindex to complete (it is safe to Ctrl+C now).
   false
   false
   false
   true
   Reindex complete:
   {
     "total": 1234567,
     "updated": 0,
     "created": 1234567,
     "version_conflicts": 0
   }
   Source index temporal-visibility-dev document count:
   1234567
   Destination index temporal_visibility_v1_dev document count:
   1234567
   

   If, for some reason, you want to start a new reindexing process, you need to cancel the previous task if it is still running. The cancel request command is in a script output. You can read more about Elasticsearch task management here. The version_conflicts value for the second run should be equal to the created value of the first run.

6. When reindexing is done, the number of documents in the source and destination indexes should match. It might be a good idea to review the new index manually:

   $ curl "http://elasticsearch:9200/temporal_visibility_v1_dev/_search?pretty"
   

   A few things to notice:

   • Attr should not appear; instead, all custom search attributes should be at the document level.
   • All *Time fields should be in human-readable time format.
   • ExecutionStatusshould be a string.
   • The new field ExecutionDuration is added.

7. Upgrade Temporal Version to 1.11. As part of this upgrade, change the visibility index in the static config to the new index name. These two changes need to happen together.

   persistence:
     datastores:
       es-visibility:
         elasticsearch:
           indices:
             visibility: temporal_visibility_v1_dev
   

8. Wait for the upgrade to complete and make sure that all nodes run the new version. Remove the history.visibilityTaskWorkerCount setting from dynamic config and restart the server one more time.

9. Verify successful migration. Visibility updates created after Step 4 were buffered and are applied after the visibility queue is unblocked (step 8).

10. It is safe to delete the old index now. (You will need old index only if you decide to roll back.)

Roll back schema changes

If something goes wrong, there is a rollback path.

If the new Temporal Server version (1.11) hasn't written anything to the new index (for example, it crashes during startup), it is safe to just switch to the old version, which will continue to use the old index. (Reindexing doesn't modify the old index.)

If, for some reason, you need to roll back after new records were written to the new index, you need to run steps 4 through 8 using rollback.sh instead of reindex.sh (assuming you have not yet deleted the old index).

Miscellaneous

• Some Elasticsearch-related metrics were renamed to follow the general pattern: https://github.com/temporalio/temporal/pull/1701.
• A new tctl command (tctl admin cluster remove-search-attributes) was added to unregister custom search attributes. It won't modify Elasticsearch schema, though, because that would require reindexing.
• You should never prefix your custom search attributes with Temporal because we might add some system search attributes in the future and would use this prefix.

All changes

2021-07-21 - ed71dcba - Revert "Separate liveness of task queue into a dedicated entity (#1699)"
2021-07-21 - ba29c478 - Revert "Enforce daemon coroutine at most once start / stop logic within matching service (#1697)"
2021-07-17 - 1135e135 - Remove docker push step from Buildkite build (#1740)
2021-07-16 - f16c5f5e - Add SLICES_COUNT param and failure report to reindex.sh (#1737)
2021-07-16 - cc47a1d2 - Disable Elasticsearch index auto creation (#1738)
2021-07-16 - 53c33545 - Elasticsearch template tuning (#1732)
2021-07-15 - b36dbeb5 - Ignore caller context cancellation error (#1734)
2021-07-15 - b123ce46 - Remove internode keep alive pings (#1735)
2021-07-14 - 746b75ed - Add rollback.sh for reindex.sh (#1728)
2021-07-13 - fa94abb4 - Add ES_VERSION parameter to reindex.sh (#1725)
2021-07-13 - 21aa499c - Handle mutable state version migration edge case (#1727)
2021-07-09 - ef4ee12f - Split memo size checks from payload; move to history service (#1691)
2021-07-09 - 1a50d49c - Fix cassandra error handling edge case (#1721)
2021-07-09 - 02f7e404 - Increase default bulk actions count from 200 to 500 (#1722)
2021-07-08 - dcc44526 - Wire Cassandra layer using new error extraction util (#1712)
2021-07-08 - a87bf44c - Add support for authorization of read-only frontend calls (#1711)
2021-07-08 - a49fa1c0 - Adds configuration flag to set DisableInitialHostLookup on gocql client driver
2021-07-07 - 9e30a3b8 - Reimplement Cassandra error extraction util (#1710)
2021-07-06 - c4fd77ae - Check shard status before workflow modification (#1707)
2021-07-05 - 28de8447 - Use an SDK client to start workflows (#1694)
2021-07-02 - cd10b355 - Refactor Elasticsearch metrics (#1701)
2021-07-02 - 8913ccad - Use mysql.Config for mysql connection string management (#1705)
2021-07-02 - 3d68d499 - Add hard minimum for retention duration (#1680)
2021-07-02 - 1aa343c7 - Update sdk to 1.8.0 (#1704)
2021-07-01 - 4d64518e - Separate liveness of task queue into a dedicated entity (#1699)
2021-06-30 - 5f1f1c32 - Enforce daemon coroutine at most once start / stop logic within matching service (#1697)
2021-06-30 - 3325b7d5 - Add metrics logging for forward error (#1695)
2021-06-29 - 226d3305 - option to use customized client factory (#1692)
2021-06-25 - 7a390510 - Modify add-search-attributes tctl command to add only missing search attributes (#1676)
2021-06-24 - db8a165b - Fix SDK cache workflow task start to close timeout behavior (#1668)
2021-06-24 - c8efea85 - Rename LastProcessedEvent to LastWorkflowTaskCompleteId (#1677)
2021-06-24 - b8c83dad - Fix to use the configured token. (#1687)
2021-06-24 - 279ba490 - Tune ES bulk processor dynamic config defaults (#1686)
2021-06-24 - 1b482d5b - Fix error prone retry time implementation (#1681)
2021-06-24 - 114c1398 - Lower-case namespace name in default authorizer (#1671)
2021-06-24 - 004cae05 - typo: uknown (#1683)
2021-06-23 - f8cc50ab - Fix enabled flag for TLS-FrontendClientConfig (#1678)
2021-06-23 - df5d49e4 - Add user metric scope (#1649)
2021-06-23 - dd13a0d6 - Refactor AddSearchAttributes to return an empty message (#1674)
2021-06-23 - a89ff408 - Add RemoveSearchAttributes API to the admin service (#1660)
2021-06-23 - a86f44e1 - Fix unable to reset workflow issue (#1673)
2021-06-23 - 09edffc8 - Persistence PayloadSerializer cleanup (#1682)
2021-06-22 - 7f4b2577 - Fix typo in encryption algorithm name ec256 -> es256 (#1664)
2021-06-22 - 56b464eb - Add optional args for DB names to install schema (#1667)
2021-06-22 - 41a953f2 - Add options to not reapply events after workflow reset (#1355)
2021-06-22 - 2661a961 - Update gocql (#1663)
2021-06-21 - 7c89d5f1 - Symlink current Elasticsearch index templates to v1 directory (#1659)
2021-06-21 - 36cd514d - Refactor search attributes GetType method (#1657)
2021-06-20 - 6b2727d4 - nil check reporters when stopping (#1656)
2021-06-17 - fc46c379 - Rename *Timestamp fields to *Time to better reflect its type (#1651)
2021-06-17 - f1352be0 - Sets Reconnection policy on gocql.ClusterObject (#1652)
2021-06-17 - b40d8f41 - Fix search attributes parsing logic (#1647)
2021-06-17 - 6459b904 - Autocreate custom search attributes fields in reindex script (#1653)
2021-06-17 - 5fb34789 - Make dynamic config case insensitive (#1635)
2021-06-17 - 3c427e6a - Add support for passing JWT audience to defaultJWTClaimMapper (#1648)
2021-06-17 - 31de6a46 - Add StateTransitionCount field to Elasticsearch index (#1650)
2021-06-17 - 2666ad92 - Restructure history service mutable state & related logics (#1641)
2021-06-17 - 0087afdf - Adds keep-alive to gRPC connections between the frontend and history/matching services #1654
2021-06-16 - f65e5b2a - Bump Web to v1.10.0 (#1609)
2021-06-16 - 9a5e9069 - Add ExecutionDuration ES field (#1645)
2021-06-16 - 7b6bcf2d - Set CGO_ENABLED to 1 for darwin and windows (#1588)
2021-06-15 - a879f7f6 - Rename Elasticsearch system search attributes (#1639)
2021-06-15 - a2aa44e4 - Refactor Elasticsearch PutMapping API (#1634)
2021-06-15 - 663c3ee4 - Use arbitrary Duration for retention (#1615)
2021-06-15 - 56e4631e - Refactor Queue for namespace replication (#1625)
2021-06-15 - 3bcfe16b - Use scaled_float instead of double for Double search attributes (#1633)
2021-06-15 - 26ac39b0 - Add mocks for data store interfaces (#1640)
2021-06-15 - 0eb0c37f - Convert Elasticsearch ExecutionStatus field to keyword (#1637)
2021-06-14 - fbb2ac2a - Fix auto-setup.sh for PostgreSQL (#1629)
2021-06-14 - f55d0bbc - Convert Elasticsearch time fields to date type (#1624)
2021-06-14 - 9fc20ca9 - Refactor Dockerfiles to better support ARM (#1575)
2021-06-14 - 9f36e826 - hookup custom data store factory (#1630)
2021-06-14 - 8e34c296 - Add minimum date validation (#1631)
2021-06-14 - 3481ed69 - Fix Elasticsearch metrics and memo encoding (#1632)
2021-06-14 - 305c18c9 - Make sync match wait time configurable (#1627)
2021-06-14 - 15666fff - Emit dedicated metrics for context cancelled (#1628)
2021-06-11 - c41ddbdb - Add ExecutionTime to WorkflowExecutionInfo (#1618)
2021-06-11 - 799ee020 - Emits warning/error logs when Temporal tries/fails to establish a new GoCQL session (#1623)
2021-06-10 - c78ea3e4 - Refactor ExecutionStore / ExecutionManager (#1614)
2021-06-10 - 79a537ec - Add logger interceptor to frontend (#1616)
2021-06-10 - 7402bd70 - Enable mutable state DB version CAS by default (#1621)
2021-06-10 - 283bb1e9 - Update SDK to 1.7.0 (#1620)
2021-06-09 - 8043f685 - Use priority mutex for mutable state cache lock (#1612)
2021-06-09 - 202e9811 - Report error for invalid retention days (#1610)
2021-06-08 - e3e7d4a5 - Introduce priority mutex & condition variable (#1611)
2021-06-08 - 80f04b52 - Remove unnecessary metrics on hisory cache (#1606)
2021-06-08 - 3311e673 - Fix error type if query task alrady gone (#1608)
2021-06-07 - ff8b2013 - Allow tctl to start workflows with infinite timeout (#1602)
2021-06-07 - 8f635379 - Remove Attr prefix from Elasticsearch schema (#1586)
2021-06-07 - 804ed608 - Task manager impl continued (#1604)
2021-06-07 - 6b0921be - Adding TaskManagerImpl (#1600)
2021-06-07 - 27456ecb - Fix opentelemetry metrics listen address (#1603)
2021-06-04 - f4bbcbc8 - Upgrade dgrijalva/jwt-go to v4.0.0-preview1 (#1597)
2021-06-04 - ea339fd6 - Remove CLI flags for security tokens (#1599)
2021-06-04 - 026a75e7 - Rename files for history manager and store (#1596)
2021-06-03 - f1c6935d - Refactor ShardManager (#1594)
2021-06-03 - d5f478b7 - Remove spew dependency (#1591)
2021-06-03 - 4951b7ee - Fix DeleteHistoryBranch (#1590)
2021-06-03 - 2eb8fac7 - Include namespace in API error log (#1593)
2021-06-02 - d36837ac - Use time.Time in internal visibility APIs (#1576)
2021-06-02 - 16584ef8 - Refactor persistence for HistoryStore (#1580)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.2)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

IMPORTANT

This release contains a serious issue (#1697). Do not use this release. Use v1.11.2 instead.

Release Highlights

The previous release did not bump the version number. This patch is to properly update the version number (#1744).

Details about v1.11.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.1)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

IMPORTANT

This release contains a serious issue (#1697). Do not use this release. Use v1.11.2 instead.

Release Highlights

Schema Changes

This release includes minor schema changes for MySQL/PostgreSQL and no schema changes for Cassandra.
Before upgrading your Temporal Server cluster, run the schema upgrade tool to support version 1.6.

Instructions for the schema update tool are here:
https://docs.temporal.io/docs/server/versions-and-dependencies/#upgrade-your-version-of-temporal.

Notable Changes

Deprecating services.<service>.metrics config

We currently have two ways to configure metrics for Temporal Server: one with a global.metrics section, another in services.<service>.metrics. The second option (in services.<service>.metrics) is deprecated and will be removed in v1.13.0.

Case-insensitive dynamic config

Dynamic config is now case insensitive.

Enhanced Visibility

Important: If you use the Enhanced Visibility feature backed by Elasticsearch, manual steps are required before upgrading to version 1.11. This release includes changes to the schema of the Elasticsearch index. Follow the steps below to change your schema prior to upgrading.

If you do not use the Enhanced Visibility feature, you can skip this section.

What you should know

Version 1.11 requires a breaking change to the Elasticsearch schema used by our experimental Enhanced Visibility feature. Manual migration is required and should take between 1 and 8 hours for most users with minor service degradation.

• What migration is being done? A new schema template and associated index has to be created and all documents need to be copied from the old index to the new one.
• Why was this needed? We did not make this breaking change lightly. Enhanced Visibility is an experimental feature that does not fall under our versioning policy. This was an internal schema refactor necessary in advance of promoting this to an officially supported, production ready feature in the next Temporal release.
• Minor degradation of service. During reindexing, the visibility feature is "frozen" — no updates will be shown when queried.
• Rare upgrade. We don't plan any breaking changes to the Elasticsearch schema in the next year.
• Duration. This migration will take some time — the actual speed depends on your setup. We estimate that most migrations should take between 1 and 8 hours.
• Rollbacks. We offer a recovery strategy in case you need to roll back the upgrade.
• Optional recommendations
  • Scale up Elasticsearch. The migration will be faster if you scale up your Elasticsearch cluster before migrating.
  • Practice run. You can run on a test cluster with one of our Docker images.

How to change your Elasticsearch schema

1. Before you upgrade your server, you need some files from this 1.11.0 release handy. Also you need direct access to Elasticsearch server. If you can't access it from your local machine, the admin-tools container is a good place to run all the steps. The container has all the needed tools, and you need to copy only these files (with kubectl cp, for example).

2. Create an index template (tune it if you use more replicas or shards):

   $ curl -X PUT "http://elasticsearch:9200/_template/temporal_visibility_v1_template" -H "Content-Type: application/json" --data-binary @index_template_v7.json --write-out "\n"
   {"acknowledged":true}
   

   Use index_template_v6.json if you are still on Elasticsearch 6.

3. Create an index. You can choose any name but it must start with temporal_visibility_v1. The default is temporal_visibility_v1_dev:

   $ curl -X PUT "http://elasticsearch:9200/temporal_visibility_v1_dev" --write-out "\n"
   {"acknowledged":true,"shards_acknowledged":true,"index":"temporal_visibility_v1_dev"}
   

4. Before you start reindexing, you need to block internal visibility queue processor to prevent new documents being created or deleted during reindexing. You can achieve this with a dynamic config setting:

   history.visibilityTaskWorkerCount:
     - value: 0
       constraints: {}
   

   If you added custom search attributes, you need to also add them to your dynamic config:

   frontend.validSearchAttributes:
     - value:
         OrderAmount: "Double"
         ProductId: "Keyword"
   

   You can check your custom search attributes by using the following command:

   $ tctl admin cluster get-search-attributes
   Custom search attributes:
   +-------------+----------+
   |    NAME     |   TYPE   |
   +-------------+----------+
   | OrderAmount | Double   |
   | ProductId   | Keyword  |
   +-------------+----------+
   ...
   

   Please note, full server restart (all nodes) is required for this setting to apply. After this step, visibility updates will stop getting written.

5. Run the reindex.sh script to reindex the old index to the new one. The script has input parameters with reasonable defaults. Please review the Input parameters section in reindex.sh. For example, if you are not using default index names, you need to pass a name. If you are on Elasticsearch 6, pass ES_VERSION=v6.

   If your index has a big number of shards (such as 100) you should specify SLICES_COUNT parameter with some reasonable value (such as 10), especially if you use AWS Elasticsearch to prevent 429 errors during reindexing.

   If you added custom search attributes, you need to pass their names to reindex.sh in a JSON array format. The default is 6 Custom*Field attributes. If you don't use them, pass an empty JSON array ('[]').

   $ CUSTOM_SEARCH_ATTRIBUTES='["ProductId", "OrderAmount"]' ES_SERVER=elasticsearch ES_VIS_INDEX_V0=temporal-visibility-prod ES_VIS_INDEX_V1=temporal_visibility_v1_prod ./reindex.sh
   

   The reindex.sh script has two steps: create custom search attribute fields in the new index, and start the reindexing process itself. Both steps show you the generated JSON and ask for confirmation. These steps are idempotent; if, for some reason, reindexing fails, you can run it again.

   Reindexing is an asynchronous operation that might take a while depending on your index size, index configuration, Elasticsearch load and configuration. Average speed is approximately 500K documents per minute. Expected output should be:

   Started reindex task 8z3GRTmSRyCtGPAKmO2RRA:155347. Check status with:
       curl http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347
   Query subtasks with:
       curl http://127.0.0.1:9200/_tasks?parent_task_id=8z3GRTmSRyCtGPAKmO2RRA:155347
   Cancel with:
       curl -X POST http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347/_cancel
   
   Waiting for reindex to complete (it is safe to Ctrl+C now).
   false
   false
   false
   true
   Reindex complete:
   {
     "total": 1234567,
     "updated": 0,
     "created": 1234567,
     "version_conflicts": 0
   }
   Source index temporal-visibility-dev document count:
   1234567
   Destination index temporal_visibility_v1_dev document count:
   1234567
   

   If, for some reason, you want to start a new reindexing process, you need to cancel the previous task if it is still running. The cancel request command is in a script output. You can read more about Elasticsearch task management here. The version_conflicts value for the second run should be equal to the created value of the first run.

6. When reindexing is done, the number of documents in the source and destination indexes should match. It might be a good idea to review the new index manually:

   $ curl "http://elasticsearch:9200/temporal_visibility_v1_dev/_search?pretty"
   

   A few things to notice:

   • Attr should not appear; instead, all custom search attributes should be at the document level.
   • All *Time fields should be in human-readable time format.
   • ExecutionStatusshould be a string.
   • The new field ExecutionDuration is added.

7. Upgrade Temporal Version to 1.11. As part of this upgrade, change the visibility index in the static config to the new index name. These two changes need to happen together.

   persistence:
     datastores:
       es-visibility:
         elasticsearch:
           indices:
             visibility: temporal_visibility_v1_dev
   

8. Wait for the upgrade to complete and make sure that all nodes run the new version. Remove the history.visibilityTaskWorkerCount setting from dynamic config and restart the server one more time.

9. Verify successful migration. Visibility updates created after Step 4 were buffered and are applied after the visibility queue is unblocked (step 8).

10. It is safe to delete the old index now. (You will need old index only if you decide to roll back.)

Roll back schema changes

If something goes wrong, there is a rollback path.

If the new Temporal Server version (1.11) hasn't written anything to the new index (for example, it crashes during startup), it is safe to just switch to the old version, which will continue to use the old index. (Reindexing doesn't modify the old index.)

If, for some reason, you need to roll back after new records were written to the new index, you need to run steps 4 through 8 using rollback.sh instead of reindex.sh (assuming you have not yet deleted the old index).

Miscellaneous

• Some Elasticsearch-related metrics were renamed to follow the general pattern: https://github.com/temporalio/temporal/pull/1701.
• A new tctl command (tctl admin cluster remove-search-attributes) was added to unregister custom search attributes. It won't modify Elasticsearch schema, though, because that would require reindexing.
• You should never prefix your custom search attributes with Temporal because we might add some system search attributes in the future and would use this prefix.

All changes

2021-07-17 - 1135e135 - Remove docker push step from Buildkite build (#1740)
2021-07-16 - f16c5f5e - Add SLICES_COUNT param and failure report to reindex.sh (#1737)
2021-07-16 - cc47a1d2 - Disable Elasticsearch index auto creation (#1738)
2021-07-16 - 53c33545 - Elasticsearch template tuning (#1732)
2021-07-15 - b36dbeb5 - Ignore caller context cancellation error (#1734)
2021-07-15 - b123ce46 - Remove internode keep alive pings (#1735)
2021-07-14 - 746b75ed - Add rollback.sh for reindex.sh (#1728)
2021-07-13 - fa94abb4 - Add ES_VERSION parameter to reindex.sh (#1725)
2021-07-13 - 21aa499c - Handle mutable state version migration edge case (#1727)
2021-07-09 - ef4ee12f - Split memo size checks from payload; move to history service (#1691)
2021-07-09 - 1a50d49c - Fix cassandra error handling edge case (#1721)
2021-07-09 - 02f7e404 - Increase default bulk actions count from 200 to 500 (#1722)
2021-07-08 - dcc44526 - Wire Cassandra layer using new error extraction util (#1712)
2021-07-08 - a87bf44c - Add support for authorization of read-only frontend calls (#1711)
2021-07-08 - a49fa1c0 - Adds configuration flag to set DisableInitialHostLookup on gocql client driver
2021-07-07 - 9e30a3b8 - Reimplement Cassandra error extraction util (#1710)
2021-07-06 - c4fd77ae - Check shard status before workflow modification (#1707)
2021-07-05 - 28de8447 - Use an SDK client to start workflows (#1694)
2021-07-02 - cd10b355 - Refactor Elasticsearch metrics (#1701)
2021-07-02 - 8913ccad - Use mysql.Config for mysql connection string management (#1705)
2021-07-02 - 3d68d499 - Add hard minimum for retention duration (#1680)
2021-07-02 - 1aa343c7 - Update sdk to 1.8.0 (#1704)
2021-07-01 - 4d64518e - Separate liveness of task queue into a dedicated entity (#1699)
2021-06-30 - 5f1f1c32 - Enforce daemon coroutine at most once start / stop logic within matching service (#1697)
2021-06-30 - 3325b7d5 - Add metrics logging for forward error (#1695)
2021-06-29 - 226d3305 - option to use customized client factory (#1692)
2021-06-25 - 7a390510 - Modify add-search-attributes tctl command to add only missing search attributes (#1676)
2021-06-24 - db8a165b - Fix SDK cache workflow task start to close timeout behavior (#1668)
2021-06-24 - c8efea85 - Rename LastProcessedEvent to LastWorkflowTaskCompleteId (#1677)
2021-06-24 - b8c83dad - Fix to use the configured token. (#1687)
2021-06-24 - 279ba490 - Tune ES bulk processor dynamic config defaults (#1686)
2021-06-24 - 1b482d5b - Fix error prone retry time implementation (#1681)
2021-06-24 - 114c1398 - Lower-case namespace name in default authorizer (#1671)
2021-06-24 - 004cae05 - typo: uknown (#1683)
2021-06-23 - f8cc50ab - Fix enabled flag for TLS-FrontendClientConfig (#1678)
2021-06-23 - df5d49e4 - Add user metric scope (#1649)
2021-06-23 - dd13a0d6 - Refactor AddSearchAttributes to return an empty message (#1674)
2021-06-23 - a89ff408 - Add RemoveSearchAttributes API to the admin service (#1660)
2021-06-23 - a86f44e1 - Fix unable to reset workflow issue (#1673)
2021-06-23 - 09edffc8 - Persistence PayloadSerializer cleanup (#1682)
2021-06-22 - 7f4b2577 - Fix typo in encryption algorithm name ec256 -> es256 (#1664)
2021-06-22 - 56b464eb - Add optional args for DB names to install schema (#1667)
2021-06-22 - 41a953f2 - Add options to not reapply events after workflow reset (#1355)
2021-06-22 - 2661a961 - Update gocql (#1663)
2021-06-21 - 7c89d5f1 - Symlink current Elasticsearch index templates to v1 directory (#1659)
2021-06-21 - 36cd514d - Refactor search attributes GetType method (#1657)
2021-06-20 - 6b2727d4 - nil check reporters when stopping (#1656)
2021-06-17 - fc46c379 - Rename *Timestamp fields to *Time to better reflect its type (#1651)
2021-06-17 - f1352be0 - Sets Reconnection policy on gocql.ClusterObject (#1652)
2021-06-17 - b40d8f41 - Fix search attributes parsing logic (#1647)
2021-06-17 - 6459b904 - Autocreate custom search attributes fields in reindex script (#1653)
2021-06-17 - 5fb34789 - Make dynamic config case insensitive (#1635)
2021-06-17 - 3c427e6a - Add support for passing JWT audience to defaultJWTClaimMapper (#1648)
2021-06-17 - 31de6a46 - Add StateTransitionCount field to Elasticsearch index (#1650)
2021-06-17 - 2666ad92 - Restructure history service mutable state & related logics (#1641)
2021-06-17 - 0087afdf - Adds keep-alive to gRPC connections between the frontend and history/matching services #1654
2021-06-16 - f65e5b2a - Bump Web to v1.10.0 (#1609)
2021-06-16 - 9a5e9069 - Add ExecutionDuration ES field (#1645)
2021-06-16 - 7b6bcf2d - Set CGO_ENABLED to 1 for darwin and windows (#1588)
2021-06-15 - a879f7f6 - Rename Elasticsearch system search attributes (#1639)
2021-06-15 - a2aa44e4 - Refactor Elasticsearch PutMapping API (#1634)
2021-06-15 - 663c3ee4 - Use arbitrary Duration for retention (#1615)
2021-06-15 - 56e4631e - Refactor Queue for namespace replication (#1625)
2021-06-15 - 3bcfe16b - Use scaled_float instead of double for Double search attributes (#1633)
2021-06-15 - 26ac39b0 - Add mocks for data store interfaces (#1640)
2021-06-15 - 0eb0c37f - Convert Elasticsearch ExecutionStatus field to keyword (#1637)
2021-06-14 - fbb2ac2a - Fix auto-setup.sh for PostgreSQL (#1629)
2021-06-14 - f55d0bbc - Convert Elasticsearch time fields to date type (#1624)
2021-06-14 - 9fc20ca9 - Refactor Dockerfiles to better support ARM (#1575)
2021-06-14 - 9f36e826 - hookup custom data store factory (#1630)
2021-06-14 - 8e34c296 - Add minimum date validation (#1631)
2021-06-14 - 3481ed69 - Fix Elasticsearch metrics and memo encoding (#1632)
2021-06-14 - 305c18c9 - Make sync match wait time configurable (#1627)
2021-06-14 - 15666fff - Emit dedicated metrics for context cancelled (#1628)
2021-06-11 - c41ddbdb - Add ExecutionTime to WorkflowExecutionInfo (#1618)
2021-06-11 - 799ee020 - Emits warning/error logs when Temporal tries/fails to establish a new GoCQL session (#1623)
2021-06-10 - c78ea3e4 - Refactor ExecutionStore / ExecutionManager (#1614)
2021-06-10 - 79a537ec - Add logger interceptor to frontend (#1616)
2021-06-10 - 7402bd70 - Enable mutable state DB version CAS by default (#1621)
2021-06-10 - 283bb1e9 - Update SDK to 1.7.0 (#1620)
2021-06-09 - 8043f685 - Use priority mutex for mutable state cache lock (#1612)
2021-06-09 - 202e9811 - Report error for invalid retention days (#1610)
2021-06-08 - e3e7d4a5 - Introduce priority mutex & condition variable (#1611)
2021-06-08 - 80f04b52 - Remove unnecessary metrics on hisory cache (#1606)
2021-06-08 - 3311e673 - Fix error type if query task alrady gone (#1608)
2021-06-07 - ff8b2013 - Allow tctl to start workflows with infinite timeout (#1602)
2021-06-07 - 8f635379 - Remove Attr prefix from Elasticsearch schema (#1586)
2021-06-07 - 804ed608 - Task manager impl continued (#1604)
2021-06-07 - 6b0921be - Adding TaskManagerImpl (#1600)
2021-06-07 - 27456ecb - Fix opentelemetry metrics listen address (#1603)
2021-06-04 - f4bbcbc8 - Upgrade dgrijalva/jwt-go to v4.0.0-preview1 (#1597)
2021-06-04 - ea339fd6 - Remove CLI flags for security tokens (#1599)
2021-06-04 - 026a75e7 - Rename files for history manager and store (#1596)
2021-06-03 - f1c6935d - Refactor ShardManager (#1594)
2021-06-03 - d5f478b7 - Remove spew dependency (#1591)
2021-06-03 - 4951b7ee - Fix DeleteHistoryBranch (#1590)
2021-06-03 - 2eb8fac7 - Include namespace in API error log (#1593)
2021-06-02 - d36837ac - Use time.Time in internal visibility APIs (#1576)
2021-06-02 - 16584ef8 - Refactor persistence for HistoryStore (#1580)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.0)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools