Release Highlights

This release does not includes any schema changes.

Notable changes

Enhanced Visibility is production ready

We are happy to officially announce that Enhanced Visibility is now production ready!

A number of new updates have been added to the Enhanced Visibility feature over the past several releases.
This release adds a few more updates (see below) and brings them all together without any breaking changes.

Elasticsearch v7 is now default

If you still use Elasticsearch v6 and don't set ES_VERSION env or version property in config file explicitly, you have to do it now.

admin elasticsearch report has been removed

Please use Elasticsearch directly to build custom reports.

Query support for ColumnName=missing syntax is deprecated

Query support for ColumnName=missing syntax will be removed in the next release. Use ColumnName is null syntax instead. Use ExecutionStatus="Running" to filter only running workflows.

Elasticsearch healthcheck is disabled by default

You can turn it (and sniffing) on with new config settings added in #1865. Leave it disable if you use AWS Elasticsearch (https://github.com/olivere/elastic/issues/1091).

All changes

2021-08-31 - 6867b1c3f - Add metrics client to custom data store factory (#1871)
2021-08-27 - 59dd702de - Fix FE interceptor order to report metrics correctly (#1863)
2021-08-27 - 431127c02 - Reserve Temporal prefix for custom search attribute names (#1861)
2021-08-27 - 33f95d7b5 - Change default value of worker.ESProcessorFlushInterval from 200ms to 1s (#1860)
2021-08-27 - 2e50ce068 - Add closeIdleConnectionsInterval, enableSniff, and enableHealthcheck to Elasticsearch config (#1865)
2021-08-26 - ff069578a - Revert "Split cron and retry paths from continue-as-new (#1745)" (#1858)
2021-08-26 - ec03542e1 - Add version to event cache key (#1838)
2021-08-26 - da5bcc73e - Remove esql package (#1857)
2021-08-26 - cae0d7947 - Split cron and retry paths from continue-as-new (#1745)
2021-08-26 - acea4e9a9 - Remove query validator (#1856)
2021-08-26 - abcee70ec - Replace DSL parsing with query builder (#1855)
2021-08-26 - 9955dc5f7 - Merge AppendHistory with CreateWorkflow (#1853)
2021-08-26 - 7d423ca6b - Remove 'admin elasticsearch report' CLI command (#1854)
2021-08-24 - eaa7d7b15 - Expose cassandra.CheckCompatibleVersion() (#1849)
2021-08-24 - c11a3e169 - Allow external persistence implementations to reuse persistenceTestBase by exporting a factory method accepting PersistenceTestCluster (#1846)
2021-08-24 - 4f8c1c77e - Fixes bug where intra-node client is doing retries even if the error is context cancelled (#1845)
2021-08-24 - 2d6f48dcf - Close factory after use (#1850)
2021-08-23 - d4f3bc665 - Spin loading initial taskWriter state in Start() (#1842)
2021-08-23 - 1f24996b9 - UpdateWorkflow transaction to include append history (#1841)
2021-08-20 - 514776cfa - Add fault injection for persistence API (#1820)
2021-08-19 - 734e4a24d - Add more metrics to Elasticsearch bulk processor (#1839)
2021-08-19 - 373d0c828 - Streamline taskQueueManager startup (#1834)
2021-08-18 - d95e21cc9 - Fix crash when no pollers on task queue. Dedupe code. (#1835)
2021-08-18 - 8f6496c35 - Add custom SQL to Elasticsearch query converter (#1824)
2021-08-18 - 451004bff - Validate ids in event cache (#1837)
2021-08-18 - 182ae5a39 - Fix ExecutionTime backfill by changing init order (#1836)
2021-08-18 - 0cef628ee - Add tctl-authorization-plugin to docker images (#1832) (#1833)
2021-08-16 - aa9a9a862 - Fix execution time not set correctly during continue as new (#1828)
2021-08-16 - 39fd4ed55 - Do not exit CLI if task queue has no poller (#1827)
2021-08-14 - 3997ae6b5 - Only reapply current workflow events if current workflow is part of continue as new chain (#1822)
2021-08-13 - 4741ef1bc - Update link to buildkite badge (#1823)
2021-08-10 - 95524cb05 - Patch fossa install (#1815)
2021-08-10 - 89fe351d5 - Fix service_errors_nondeterministic merics (#1726)
2021-08-10 - 3595e6243 - Fix API doc comment for authorization.CallTarget.Request (#1816)
2021-08-09 - f9e8559b4 - Udpate fossa cli (#1811)
2021-08-09 - c4d3c34f7 - Adds flag to tctl db-based commands to override TLS ServerName setting #1813
2021-08-09 - a6e9c0567 - NewFactory with given session (#1812)
2021-08-08 - 6d1c70d68 - Revert "Add tctl config feature (#1790)" (#1810)
2021-08-07 - 438b3228c - Fix TaskQueueLock starvation (#1809)
2021-08-06 - e834f2aa6 - Update SDK to 1.9.0 (#1808)
2021-08-05 - c158bfe08 - Make sure shard acquirement channel will not be blocked (#1806)
2021-08-05 - a1b4b0673 - Upgrade gocql (#1805)
2021-08-05 - 69ddc6f82 - Enable fossa test and add delay between analyze and test (#1802)
2021-08-05 - 46a8766e9 - Handle unreliable gocql close function stuck issue (#1803)
2021-08-05 - 2e3fd6ab0 - Optimize matching ack manager loop perf (#1804)
2021-08-05 - 0f4bdf5fc - Remove unnecessary shard controller prepare to stop logic (#1807)
2021-08-04 - c630240b0 - Add retries for tctl visibility APIs (#1798)
2021-08-04 - 3e93f464c - Expose TLS RefreshInterval & ExpirationChecks via env var (#1800)
2021-08-04 - 149072c37 - Update fossa cli in base-ci-builder image (#1801)
2021-08-02 - fe0575130 - Add system worker execution & poller dynamic configs (#1796)
2021-08-02 - d7e24101f - Disallow fetching CA cert over HTTP (#1794)
2021-08-02 - a4ce3a1e2 - Merge HistoryManager into ExecutionManager (#1792)
2021-08-02 - 800768f58 - FetchCACert now supports URLs in addition to files (#1773)
2021-08-02 - 53d107c17 - Revert "Release new tctl UX (#1789)" (#1795)
2021-08-02 - 5374709fc - Release new tctl UX (#1789)
2021-08-01 - d5dbf4e51 - Disable sniffing on ES7 client used in CLI (#1784)
2021-08-01 - 25e22aee2 - Execution store no shard id2 (#1791)
2021-08-01 - 248550b32 - Rewrite persistence create workflow API error handling (#1770)
2021-07-30 - b557462ad - Report service user latency metric explicitly (#1787)
2021-07-30 - 9f9e8f8d0 - Matching deadlock fix and GC refactor (#1785)
2021-07-30 - 68e9adba8 - Add tctl config feature (#1790)
2021-07-30 - 5c777e717 - Rename WorkflowStore to ExecutionStore (#1786)
2021-07-30 - 4e7f6f9a4 - Adjust system worker poller number (#1788)
2021-07-30 - 094f860b2 - Add task latency without user induced latency (#1777)
2021-07-29 - d4b7671eb - Include root cause reasons in Elasticsearch error message (#1774)
2021-07-29 - c74834fea - WorkflowStore interface (#1781)
2021-07-29 - b765bbb97 - Revert part of matching cleanup (a18bb3abd) (#1779)
2021-07-29 - b241ae765 - Remove GetClosedWorkflowExecution method from visibility (#1782)
2021-07-29 - a18bb3abd - Matching cleanup (#1708)
2021-07-29 - 8eff0b728 - Use PIT instead of scroll for Scan API (#1772)
2021-07-29 - 7799e6c3f - Test-friendly refactorings in common (#1771)
2021-07-29 - 557b5d721 - Move visibility persistence to visibility package (#1760)
2021-07-28 - ced5fc507 - Set longer task timeout to avoid stampede on history service (#1776)
2021-07-27 - 42b8d1ee8 - Remove 1.10 backward compatibility code (#1764)
2021-07-26 - fb839418e - Switch default Elasticsearch version from v6 to v7 (#1763)
2021-07-26 - 77db3b7aa - Downgrade namespace logger messages from Info to Debug (#1755)
2021-07-26 - 2910c38ca - Correctly reapply buffered events from current workflow during workflow reset (#1762)
2021-07-26 - 18faf0078 - Tag Web v1.11.0 (#1766)
2021-07-26 - 08a4a6b24 - Delete visibility documents with admin delete command (#1767)
2021-07-23 - 6e417e27a - Extract workflow notification into dedicated util (#1761)
2021-07-23 - 32d0786fb - Extract workflow & events persistence ops into dedicated util (#1759)
2021-07-22 - eeb9adb2d - Clean up plugin interface (#1709)
2021-07-22 - 3faa5454a - Remove unused history manager reference from history engine (#1750)
2021-07-21 - ea4fd1f37 - Only unload TaskQueueManager if ownership is lost or queue traffic drops (#1730)
2021-07-21 - 9d023eef5 - Rename internal API names (#1749)
2021-07-21 - 71d548ff9 - Fix few typos (#1748)
2021-07-21 - 5d40f91f6 - Split workflow context logic into 2 files (#1751)
2021-07-20 - d2788a7c4 - Use filter context instead of query context for exact match queries (#1723)
2021-07-20 - adf692b6b - Always use search_after for Elasticsearch pagination (#1724)
2021-07-20 - a8f7cd9f7 - Add JSON print to get-search-attributes admin command (#1731)
2021-07-20 - 79fa9b3db - Bump version for next release (#1746)
2021-07-20 - 778b5059b - Extend CurrentWorkflowConditionFailedError to include more information about current workflow (#1747)
2021-07-20 - 423ef7b08 - Add skip-schema-update flag to add-search-attributes tctl admin command (#1733)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.12.0)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes bug fixes mainly for cron schedule:
2021-08-18 - 4f5d4c28a - Fix ExecutionTime backfill by changing init order (#1836)
2021-08-18 - 6ef5257fc - Fix execution time not set correctly during continue as new (#1828)

Details about v1.11.x release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.4)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes 2 bug fixes:

• Make sure shard acquirement channel will not be blocked (#1806)
• Handle unreliable gocql close function stuck issue (#1803)

Details about v1.11.x release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.3)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

IMPORTANT

The release v1.11.2 reverts two PRs to fix a deadlock issue introduced in v1.11.0 and v1.11.1.

• revert of #1699
• revert of #1697

The reason for reverting #1699 is to allow a clean revert of #1697.
#1697 introduced a deadlock issue that can cause the entire matching service to be unavailable if the database is not stable.

Schema Changes

This release includes minor schema changes for MySQL/PostgreSQL and no schema changes for Cassandra.
Before upgrading your Temporal Server cluster, run the schema upgrade tool to support version 1.6.

Instructions for the schema update tool are here:
https://docs.temporal.io/docs/server/versions-and-dependencies/#upgrade-your-version-of-temporal.

Notable Changes

Deprecating services.<service>.metrics config

We currently have two ways to configure metrics for Temporal Server: one with a global.metrics section, another in services.<service>.metrics. The second option (in services.<service>.metrics) is deprecated and will be removed in v1.13.0.

Case-insensitive dynamic config

Dynamic config is now case insensitive.

Enhanced Visibility

Important: If you use the Enhanced Visibility feature backed by Elasticsearch, manual steps are required before upgrading to version 1.11. This release includes changes to the schema of the Elasticsearch index. Follow the steps below to change your schema prior to upgrading.

If you do not use the Enhanced Visibility feature, you can skip this section.

What you should know

Version 1.11 requires a breaking change to the Elasticsearch schema used by our experimental Enhanced Visibility feature. Manual migration is required and should take between 1 and 8 hours for most users with minor service degradation.

• What migration is being done? A new schema template and associated index has to be created and all documents need to be copied from the old index to the new one.
• Why was this needed? We did not make this breaking change lightly. Enhanced Visibility is an experimental feature that does not fall under our versioning policy. This was an internal schema refactor necessary in advance of promoting this to an officially supported, production ready feature in the next Temporal release.
• Minor degradation of service. During reindexing, the visibility feature is "frozen" — no updates will be shown when queried.
• Rare upgrade. We don't plan any breaking changes to the Elasticsearch schema in the next year.
• Duration. This migration will take some time — the actual speed depends on your setup. We estimate that most migrations should take between 1 and 8 hours.
• Rollbacks. We offer a recovery strategy in case you need to roll back the upgrade.
• Optional recommendations
  • Scale up Elasticsearch. The migration will be faster if you scale up your Elasticsearch cluster before migrating.
  • Practice run. You can run on a test cluster with one of our Docker images.

How to change your Elasticsearch schema

1. Before you upgrade your server, you need some files from this 1.11.2 release handy. Also you need direct access to Elasticsearch server. If you can't access it from your local machine, the admin-tools container is a good place to run all the steps. The container has all the needed tools, and you need to copy only these files (with kubectl cp, for example).

2. Create an index template (tune it if you use more replicas or shards):

   $ curl -X PUT "http://elasticsearch:9200/_template/temporal_visibility_v1_template" -H "Content-Type: application/json" --data-binary @index_template_v7.json --write-out "\n"
   {"acknowledged":true}
   

   Use index_template_v6.json if you are still on Elasticsearch 6.

3. Create an index. You can choose any name but it must start with temporal_visibility_v1. The default is temporal_visibility_v1_dev:

   $ curl -X PUT "http://elasticsearch:9200/temporal_visibility_v1_dev" --write-out "\n"
   {"acknowledged":true,"shards_acknowledged":true,"index":"temporal_visibility_v1_dev"}
   

4. Before you start reindexing, you need to block internal visibility queue processor to prevent new documents being created or deleted during reindexing. You can achieve this with a dynamic config setting:

   history.visibilityTaskWorkerCount:
     - value: 0
       constraints: {}
   

   If you added custom search attributes, you need to also add them to your dynamic config:

   frontend.validSearchAttributes:
     - value:
         OrderAmount: "Double"
         ProductId: "Keyword"
   

   You can check your custom search attributes by using the following command:

   $ tctl admin cluster get-search-attributes
   Custom search attributes:
   +-------------+----------+
   |    NAME     |   TYPE   |
   +-------------+----------+
   | OrderAmount | Double   |
   | ProductId   | Keyword  |
   +-------------+----------+
   ...
   

   Please note, full server restart (all nodes) is required for this setting to apply. After this step, visibility updates will stop getting written.

5. Run the reindex.sh script to reindex the old index to the new one. The script has input parameters with reasonable defaults. Please review the Input parameters section in reindex.sh. For example, if you are not using default index names, you need to pass a name. If you are on Elasticsearch 6, pass ES_VERSION=v6.

   If your index has a big number of shards (such as 100) you should specify SLICES_COUNT parameter with some reasonable value (such as 10), especially if you use AWS Elasticsearch to prevent 429 errors during reindexing.

   If you added custom search attributes, you need to pass their names to reindex.sh in a JSON array format. The default is 6 Custom*Field attributes. If you don't use them, pass an empty JSON array ('[]').

   $ CUSTOM_SEARCH_ATTRIBUTES='["ProductId", "OrderAmount"]' ES_SERVER=elasticsearch ES_VIS_INDEX_V0=temporal-visibility-prod ES_VIS_INDEX_V1=temporal_visibility_v1_prod ./reindex.sh
   

   The reindex.sh script has two steps: create custom search attribute fields in the new index, and start the reindexing process itself. Both steps show you the generated JSON and ask for confirmation. These steps are idempotent; if, for some reason, reindexing fails, you can run it again.

   Reindexing is an asynchronous operation that might take a while depending on your index size, index configuration, Elasticsearch load and configuration. Average speed is approximately 500K documents per minute. Expected output should be:

   Started reindex task 8z3GRTmSRyCtGPAKmO2RRA:155347. Check status with:
       curl http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347
   Query subtasks with:
       curl http://127.0.0.1:9200/_tasks?parent_task_id=8z3GRTmSRyCtGPAKmO2RRA:155347
   Cancel with:
       curl -X POST http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347/_cancel
   
   Waiting for reindex to complete (it is safe to Ctrl+C now).
   false
   false
   false
   true
   Reindex complete:
   {
     "total": 1234567,
     "updated": 0,
     "created": 1234567,
     "version_conflicts": 0
   }
   Source index temporal-visibility-dev document count:
   1234567
   Destination index temporal_visibility_v1_dev document count:
   1234567
   

   If, for some reason, you want to start a new reindexing process, you need to cancel the previous task if it is still running. The cancel request command is in a script output. You can read more about Elasticsearch task management here. The version_conflicts value for the second run should be equal to the created value of the first run.

6. When reindexing is done, the number of documents in the source and destination indexes should match. It might be a good idea to review the new index manually:

   $ curl "http://elasticsearch:9200/temporal_visibility_v1_dev/_search?pretty"
   

   A few things to notice:

   • Attr should not appear; instead, all custom search attributes should be at the document level.
   • All *Time fields should be in human-readable time format.
   • ExecutionStatusshould be a string.
   • The new field ExecutionDuration is added.

7. Upgrade Temporal Version to 1.11. As part of this upgrade, change the visibility index in the static config to the new index name. These two changes need to happen together.

   persistence:
     datastores:
       es-visibility:
         elasticsearch:
           indices:
             visibility: temporal_visibility_v1_dev
   

8. Wait for the upgrade to complete and make sure that all nodes run the new version. Remove the history.visibilityTaskWorkerCount setting from dynamic config and restart the server one more time.

9. Verify successful migration. Visibility updates created after Step 4 were buffered and are applied after the visibility queue is unblocked (step 8).

10. It is safe to delete the old index now. (You will need old index only if you decide to roll back.)

Roll back schema changes

If something goes wrong, there is a rollback path.

If the new Temporal Server version (1.11) hasn't written anything to the new index (for example, it crashes during startup), it is safe to just switch to the old version, which will continue to use the old index. (Reindexing doesn't modify the old index.)

If, for some reason, you need to roll back after new records were written to the new index, you need to run steps 4 through 8 using rollback.sh instead of reindex.sh (assuming you have not yet deleted the old index).

Miscellaneous

• Some Elasticsearch-related metrics were renamed to follow the general pattern: https://github.com/temporalio/temporal/pull/1701.
• A new tctl command (tctl admin cluster remove-search-attributes) was added to unregister custom search attributes. It won't modify Elasticsearch schema, though, because that would require reindexing.
• You should never prefix your custom search attributes with Temporal because we might add some system search attributes in the future and would use this prefix.

All changes

2021-07-21 - ed71dcba - Revert "Separate liveness of task queue into a dedicated entity (#1699)"
2021-07-21 - ba29c478 - Revert "Enforce daemon coroutine at most once start / stop logic within matching service (#1697)"
2021-07-17 - 1135e135 - Remove docker push step from Buildkite build (#1740)
2021-07-16 - f16c5f5e - Add SLICES_COUNT param and failure report to reindex.sh (#1737)
2021-07-16 - cc47a1d2 - Disable Elasticsearch index auto creation (#1738)
2021-07-16 - 53c33545 - Elasticsearch template tuning (#1732)
2021-07-15 - b36dbeb5 - Ignore caller context cancellation error (#1734)
2021-07-15 - b123ce46 - Remove internode keep alive pings (#1735)
2021-07-14 - 746b75ed - Add rollback.sh for reindex.sh (#1728)
2021-07-13 - fa94abb4 - Add ES_VERSION parameter to reindex.sh (#1725)
2021-07-13 - 21aa499c - Handle mutable state version migration edge case (#1727)
2021-07-09 - ef4ee12f - Split memo size checks from payload; move to history service (#1691)
2021-07-09 - 1a50d49c - Fix cassandra error handling edge case (#1721)
2021-07-09 - 02f7e404 - Increase default bulk actions count from 200 to 500 (#1722)
2021-07-08 - dcc44526 - Wire Cassandra layer using new error extraction util (#1712)
2021-07-08 - a87bf44c - Add support for authorization of read-only frontend calls (#1711)
2021-07-08 - a49fa1c0 - Adds configuration flag to set DisableInitialHostLookup on gocql client driver
2021-07-07 - 9e30a3b8 - Reimplement Cassandra error extraction util (#1710)
2021-07-06 - c4fd77ae - Check shard status before workflow modification (#1707)
2021-07-05 - 28de8447 - Use an SDK client to start workflows (#1694)
2021-07-02 - cd10b355 - Refactor Elasticsearch metrics (#1701)
2021-07-02 - 8913ccad - Use mysql.Config for mysql connection string management (#1705)
2021-07-02 - 3d68d499 - Add hard minimum for retention duration (#1680)
2021-07-02 - 1aa343c7 - Update sdk to 1.8.0 (#1704)
2021-07-01 - 4d64518e - Separate liveness of task queue into a dedicated entity (#1699)
2021-06-30 - 5f1f1c32 - Enforce daemon coroutine at most once start / stop logic within matching service (#1697)
2021-06-30 - 3325b7d5 - Add metrics logging for forward error (#1695)
2021-06-29 - 226d3305 - option to use customized client factory (#1692)
2021-06-25 - 7a390510 - Modify add-search-attributes tctl command to add only missing search attributes (#1676)
2021-06-24 - db8a165b - Fix SDK cache workflow task start to close timeout behavior (#1668)
2021-06-24 - c8efea85 - Rename LastProcessedEvent to LastWorkflowTaskCompleteId (#1677)
2021-06-24 - b8c83dad - Fix to use the configured token. (#1687)
2021-06-24 - 279ba490 - Tune ES bulk processor dynamic config defaults (#1686)
2021-06-24 - 1b482d5b - Fix error prone retry time implementation (#1681)
2021-06-24 - 114c1398 - Lower-case namespace name in default authorizer (#1671)
2021-06-24 - 004cae05 - typo: uknown (#1683)
2021-06-23 - f8cc50ab - Fix enabled flag for TLS-FrontendClientConfig (#1678)
2021-06-23 - df5d49e4 - Add user metric scope (#1649)
2021-06-23 - dd13a0d6 - Refactor AddSearchAttributes to return an empty message (#1674)
2021-06-23 - a89ff408 - Add RemoveSearchAttributes API to the admin service (#1660)
2021-06-23 - a86f44e1 - Fix unable to reset workflow issue (#1673)
2021-06-23 - 09edffc8 - Persistence PayloadSerializer cleanup (#1682)
2021-06-22 - 7f4b2577 - Fix typo in encryption algorithm name ec256 -> es256 (#1664)
2021-06-22 - 56b464eb - Add optional args for DB names to install schema (#1667)
2021-06-22 - 41a953f2 - Add options to not reapply events after workflow reset (#1355)
2021-06-22 - 2661a961 - Update gocql (#1663)
2021-06-21 - 7c89d5f1 - Symlink current Elasticsearch index templates to v1 directory (#1659)
2021-06-21 - 36cd514d - Refactor search attributes GetType method (#1657)
2021-06-20 - 6b2727d4 - nil check reporters when stopping (#1656)
2021-06-17 - fc46c379 - Rename *Timestamp fields to *Time to better reflect its type (#1651)
2021-06-17 - f1352be0 - Sets Reconnection policy on gocql.ClusterObject (#1652)
2021-06-17 - b40d8f41 - Fix search attributes parsing logic (#1647)
2021-06-17 - 6459b904 - Autocreate custom search attributes fields in reindex script (#1653)
2021-06-17 - 5fb34789 - Make dynamic config case insensitive (#1635)
2021-06-17 - 3c427e6a - Add support for passing JWT audience to defaultJWTClaimMapper (#1648)
2021-06-17 - 31de6a46 - Add StateTransitionCount field to Elasticsearch index (#1650)
2021-06-17 - 2666ad92 - Restructure history service mutable state & related logics (#1641)
2021-06-17 - 0087afdf - Adds keep-alive to gRPC connections between the frontend and history/matching services #1654
2021-06-16 - f65e5b2a - Bump Web to v1.10.0 (#1609)
2021-06-16 - 9a5e9069 - Add ExecutionDuration ES field (#1645)
2021-06-16 - 7b6bcf2d - Set CGO_ENABLED to 1 for darwin and windows (#1588)
2021-06-15 - a879f7f6 - Rename Elasticsearch system search attributes (#1639)
2021-06-15 - a2aa44e4 - Refactor Elasticsearch PutMapping API (#1634)
2021-06-15 - 663c3ee4 - Use arbitrary Duration for retention (#1615)
2021-06-15 - 56e4631e - Refactor Queue for namespace replication (#1625)
2021-06-15 - 3bcfe16b - Use scaled_float instead of double for Double search attributes (#1633)
2021-06-15 - 26ac39b0 - Add mocks for data store interfaces (#1640)
2021-06-15 - 0eb0c37f - Convert Elasticsearch ExecutionStatus field to keyword (#1637)
2021-06-14 - fbb2ac2a - Fix auto-setup.sh for PostgreSQL (#1629)
2021-06-14 - f55d0bbc - Convert Elasticsearch time fields to date type (#1624)
2021-06-14 - 9fc20ca9 - Refactor Dockerfiles to better support ARM (#1575)
2021-06-14 - 9f36e826 - hookup custom data store factory (#1630)
2021-06-14 - 8e34c296 - Add minimum date validation (#1631)
2021-06-14 - 3481ed69 - Fix Elasticsearch metrics and memo encoding (#1632)
2021-06-14 - 305c18c9 - Make sync match wait time configurable (#1627)
2021-06-14 - 15666fff - Emit dedicated metrics for context cancelled (#1628)
2021-06-11 - c41ddbdb - Add ExecutionTime to WorkflowExecutionInfo (#1618)
2021-06-11 - 799ee020 - Emits warning/error logs when Temporal tries/fails to establish a new GoCQL session (#1623)
2021-06-10 - c78ea3e4 - Refactor ExecutionStore / ExecutionManager (#1614)
2021-06-10 - 79a537ec - Add logger interceptor to frontend (#1616)
2021-06-10 - 7402bd70 - Enable mutable state DB version CAS by default (#1621)
2021-06-10 - 283bb1e9 - Update SDK to 1.7.0 (#1620)
2021-06-09 - 8043f685 - Use priority mutex for mutable state cache lock (#1612)
2021-06-09 - 202e9811 - Report error for invalid retention days (#1610)
2021-06-08 - e3e7d4a5 - Introduce priority mutex & condition variable (#1611)
2021-06-08 - 80f04b52 - Remove unnecessary metrics on hisory cache (#1606)
2021-06-08 - 3311e673 - Fix error type if query task alrady gone (#1608)
2021-06-07 - ff8b2013 - Allow tctl to start workflows with infinite timeout (#1602)
2021-06-07 - 8f635379 - Remove Attr prefix from Elasticsearch schema (#1586)
2021-06-07 - 804ed608 - Task manager impl continued (#1604)
2021-06-07 - 6b0921be - Adding TaskManagerImpl (#1600)
2021-06-07 - 27456ecb - Fix opentelemetry metrics listen address (#1603)
2021-06-04 - f4bbcbc8 - Upgrade dgrijalva/jwt-go to v4.0.0-preview1 (#1597)
2021-06-04 - ea339fd6 - Remove CLI flags for security tokens (#1599)
2021-06-04 - 026a75e7 - Rename files for history manager and store (#1596)
2021-06-03 - f1c6935d - Refactor ShardManager (#1594)
2021-06-03 - d5f478b7 - Remove spew dependency (#1591)
2021-06-03 - 4951b7ee - Fix DeleteHistoryBranch (#1590)
2021-06-03 - 2eb8fac7 - Include namespace in API error log (#1593)
2021-06-02 - d36837ac - Use time.Time in internal visibility APIs (#1576)
2021-06-02 - 16584ef8 - Refactor persistence for HistoryStore (#1580)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.2)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

IMPORTANT

This release contains a serious issue (#1697). Do not use this release. Use v1.11.2 instead.

Release Highlights

The previous release did not bump the version number. This patch is to properly update the version number (#1744).

Details about v1.11.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.1)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

IMPORTANT

This release contains a serious issue (#1697). Do not use this release. Use v1.11.2 instead.

Release Highlights

Schema Changes

This release includes minor schema changes for MySQL/PostgreSQL and no schema changes for Cassandra.
Before upgrading your Temporal Server cluster, run the schema upgrade tool to support version 1.6.

Instructions for the schema update tool are here:
https://docs.temporal.io/docs/server/versions-and-dependencies/#upgrade-your-version-of-temporal.

Notable Changes

Deprecating services.<service>.metrics config

We currently have two ways to configure metrics for Temporal Server: one with a global.metrics section, another in services.<service>.metrics. The second option (in services.<service>.metrics) is deprecated and will be removed in v1.13.0.

Case-insensitive dynamic config

Dynamic config is now case insensitive.

Enhanced Visibility

Important: If you use the Enhanced Visibility feature backed by Elasticsearch, manual steps are required before upgrading to version 1.11. This release includes changes to the schema of the Elasticsearch index. Follow the steps below to change your schema prior to upgrading.

If you do not use the Enhanced Visibility feature, you can skip this section.

What you should know

Version 1.11 requires a breaking change to the Elasticsearch schema used by our experimental Enhanced Visibility feature. Manual migration is required and should take between 1 and 8 hours for most users with minor service degradation.

• What migration is being done? A new schema template and associated index has to be created and all documents need to be copied from the old index to the new one.
• Why was this needed? We did not make this breaking change lightly. Enhanced Visibility is an experimental feature that does not fall under our versioning policy. This was an internal schema refactor necessary in advance of promoting this to an officially supported, production ready feature in the next Temporal release.
• Minor degradation of service. During reindexing, the visibility feature is "frozen" — no updates will be shown when queried.
• Rare upgrade. We don't plan any breaking changes to the Elasticsearch schema in the next year.
• Duration. This migration will take some time — the actual speed depends on your setup. We estimate that most migrations should take between 1 and 8 hours.
• Rollbacks. We offer a recovery strategy in case you need to roll back the upgrade.
• Optional recommendations
  • Scale up Elasticsearch. The migration will be faster if you scale up your Elasticsearch cluster before migrating.
  • Practice run. You can run on a test cluster with one of our Docker images.

How to change your Elasticsearch schema

1. Before you upgrade your server, you need some files from this 1.11.0 release handy. Also you need direct access to Elasticsearch server. If you can't access it from your local machine, the admin-tools container is a good place to run all the steps. The container has all the needed tools, and you need to copy only these files (with kubectl cp, for example).

2. Create an index template (tune it if you use more replicas or shards):

   $ curl -X PUT "http://elasticsearch:9200/_template/temporal_visibility_v1_template" -H "Content-Type: application/json" --data-binary @index_template_v7.json --write-out "\n"
   {"acknowledged":true}
   

   Use index_template_v6.json if you are still on Elasticsearch 6.

3. Create an index. You can choose any name but it must start with temporal_visibility_v1. The default is temporal_visibility_v1_dev:

   $ curl -X PUT "http://elasticsearch:9200/temporal_visibility_v1_dev" --write-out "\n"
   {"acknowledged":true,"shards_acknowledged":true,"index":"temporal_visibility_v1_dev"}
   

4. Before you start reindexing, you need to block internal visibility queue processor to prevent new documents being created or deleted during reindexing. You can achieve this with a dynamic config setting:

   history.visibilityTaskWorkerCount:
     - value: 0
       constraints: {}
   

   If you added custom search attributes, you need to also add them to your dynamic config:

   frontend.validSearchAttributes:
     - value:
         OrderAmount: "Double"
         ProductId: "Keyword"
   

   You can check your custom search attributes by using the following command:

   $ tctl admin cluster get-search-attributes
   Custom search attributes:
   +-------------+----------+
   |    NAME     |   TYPE   |
   +-------------+----------+
   | OrderAmount | Double   |
   | ProductId   | Keyword  |
   +-------------+----------+
   ...
   

   Please note, full server restart (all nodes) is required for this setting to apply. After this step, visibility updates will stop getting written.

5. Run the reindex.sh script to reindex the old index to the new one. The script has input parameters with reasonable defaults. Please review the Input parameters section in reindex.sh. For example, if you are not using default index names, you need to pass a name. If you are on Elasticsearch 6, pass ES_VERSION=v6.

   If your index has a big number of shards (such as 100) you should specify SLICES_COUNT parameter with some reasonable value (such as 10), especially if you use AWS Elasticsearch to prevent 429 errors during reindexing.

   If you added custom search attributes, you need to pass their names to reindex.sh in a JSON array format. The default is 6 Custom*Field attributes. If you don't use them, pass an empty JSON array ('[]').

   $ CUSTOM_SEARCH_ATTRIBUTES='["ProductId", "OrderAmount"]' ES_SERVER=elasticsearch ES_VIS_INDEX_V0=temporal-visibility-prod ES_VIS_INDEX_V1=temporal_visibility_v1_prod ./reindex.sh
   

   The reindex.sh script has two steps: create custom search attribute fields in the new index, and start the reindexing process itself. Both steps show you the generated JSON and ask for confirmation. These steps are idempotent; if, for some reason, reindexing fails, you can run it again.

   Reindexing is an asynchronous operation that might take a while depending on your index size, index configuration, Elasticsearch load and configuration. Average speed is approximately 500K documents per minute. Expected output should be:

   Started reindex task 8z3GRTmSRyCtGPAKmO2RRA:155347. Check status with:
       curl http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347
   Query subtasks with:
       curl http://127.0.0.1:9200/_tasks?parent_task_id=8z3GRTmSRyCtGPAKmO2RRA:155347
   Cancel with:
       curl -X POST http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347/_cancel
   
   Waiting for reindex to complete (it is safe to Ctrl+C now).
   false
   false
   false
   true
   Reindex complete:
   {
     "total": 1234567,
     "updated": 0,
     "created": 1234567,
     "version_conflicts": 0
   }
   Source index temporal-visibility-dev document count:
   1234567
   Destination index temporal_visibility_v1_dev document count:
   1234567
   

   If, for some reason, you want to start a new reindexing process, you need to cancel the previous task if it is still running. The cancel request command is in a script output. You can read more about Elasticsearch task management here. The version_conflicts value for the second run should be equal to the created value of the first run.

6. When reindexing is done, the number of documents in the source and destination indexes should match. It might be a good idea to review the new index manually:

   $ curl "http://elasticsearch:9200/temporal_visibility_v1_dev/_search?pretty"
   

   A few things to notice:

   • Attr should not appear; instead, all custom search attributes should be at the document level.
   • All *Time fields should be in human-readable time format.
   • ExecutionStatusshould be a string.
   • The new field ExecutionDuration is added.

7. Upgrade Temporal Version to 1.11. As part of this upgrade, change the visibility index in the static config to the new index name. These two changes need to happen together.

   persistence:
     datastores:
       es-visibility:
         elasticsearch:
           indices:
             visibility: temporal_visibility_v1_dev
   

8. Wait for the upgrade to complete and make sure that all nodes run the new version. Remove the history.visibilityTaskWorkerCount setting from dynamic config and restart the server one more time.

9. Verify successful migration. Visibility updates created after Step 4 were buffered and are applied after the visibility queue is unblocked (step 8).

10. It is safe to delete the old index now. (You will need old index only if you decide to roll back.)

Roll back schema changes

If something goes wrong, there is a rollback path.

If the new Temporal Server version (1.11) hasn't written anything to the new index (for example, it crashes during startup), it is safe to just switch to the old version, which will continue to use the old index. (Reindexing doesn't modify the old index.)

If, for some reason, you need to roll back after new records were written to the new index, you need to run steps 4 through 8 using rollback.sh instead of reindex.sh (assuming you have not yet deleted the old index).

Miscellaneous

• Some Elasticsearch-related metrics were renamed to follow the general pattern: https://github.com/temporalio/temporal/pull/1701.
• A new tctl command (tctl admin cluster remove-search-attributes) was added to unregister custom search attributes. It won't modify Elasticsearch schema, though, because that would require reindexing.
• You should never prefix your custom search attributes with Temporal because we might add some system search attributes in the future and would use this prefix.

All changes

2021-07-17 - 1135e135 - Remove docker push step from Buildkite build (#1740)
2021-07-16 - f16c5f5e - Add SLICES_COUNT param and failure report to reindex.sh (#1737)
2021-07-16 - cc47a1d2 - Disable Elasticsearch index auto creation (#1738)
2021-07-16 - 53c33545 - Elasticsearch template tuning (#1732)
2021-07-15 - b36dbeb5 - Ignore caller context cancellation error (#1734)
2021-07-15 - b123ce46 - Remove internode keep alive pings (#1735)
2021-07-14 - 746b75ed - Add rollback.sh for reindex.sh (#1728)
2021-07-13 - fa94abb4 - Add ES_VERSION parameter to reindex.sh (#1725)
2021-07-13 - 21aa499c - Handle mutable state version migration edge case (#1727)
2021-07-09 - ef4ee12f - Split memo size checks from payload; move to history service (#1691)
2021-07-09 - 1a50d49c - Fix cassandra error handling edge case (#1721)
2021-07-09 - 02f7e404 - Increase default bulk actions count from 200 to 500 (#1722)
2021-07-08 - dcc44526 - Wire Cassandra layer using new error extraction util (#1712)
2021-07-08 - a87bf44c - Add support for authorization of read-only frontend calls (#1711)
2021-07-08 - a49fa1c0 - Adds configuration flag to set DisableInitialHostLookup on gocql client driver
2021-07-07 - 9e30a3b8 - Reimplement Cassandra error extraction util (#1710)
2021-07-06 - c4fd77ae - Check shard status before workflow modification (#1707)
2021-07-05 - 28de8447 - Use an SDK client to start workflows (#1694)
2021-07-02 - cd10b355 - Refactor Elasticsearch metrics (#1701)
2021-07-02 - 8913ccad - Use mysql.Config for mysql connection string management (#1705)
2021-07-02 - 3d68d499 - Add hard minimum for retention duration (#1680)
2021-07-02 - 1aa343c7 - Update sdk to 1.8.0 (#1704)
2021-07-01 - 4d64518e - Separate liveness of task queue into a dedicated entity (#1699)
2021-06-30 - 5f1f1c32 - Enforce daemon coroutine at most once start / stop logic within matching service (#1697)
2021-06-30 - 3325b7d5 - Add metrics logging for forward error (#1695)
2021-06-29 - 226d3305 - option to use customized client factory (#1692)
2021-06-25 - 7a390510 - Modify add-search-attributes tctl command to add only missing search attributes (#1676)
2021-06-24 - db8a165b - Fix SDK cache workflow task start to close timeout behavior (#1668)
2021-06-24 - c8efea85 - Rename LastProcessedEvent to LastWorkflowTaskCompleteId (#1677)
2021-06-24 - b8c83dad - Fix to use the configured token. (#1687)
2021-06-24 - 279ba490 - Tune ES bulk processor dynamic config defaults (#1686)
2021-06-24 - 1b482d5b - Fix error prone retry time implementation (#1681)
2021-06-24 - 114c1398 - Lower-case namespace name in default authorizer (#1671)
2021-06-24 - 004cae05 - typo: uknown (#1683)
2021-06-23 - f8cc50ab - Fix enabled flag for TLS-FrontendClientConfig (#1678)
2021-06-23 - df5d49e4 - Add user metric scope (#1649)
2021-06-23 - dd13a0d6 - Refactor AddSearchAttributes to return an empty message (#1674)
2021-06-23 - a89ff408 - Add RemoveSearchAttributes API to the admin service (#1660)
2021-06-23 - a86f44e1 - Fix unable to reset workflow issue (#1673)
2021-06-23 - 09edffc8 - Persistence PayloadSerializer cleanup (#1682)
2021-06-22 - 7f4b2577 - Fix typo in encryption algorithm name ec256 -> es256 (#1664)
2021-06-22 - 56b464eb - Add optional args for DB names to install schema (#1667)
2021-06-22 - 41a953f2 - Add options to not reapply events after workflow reset (#1355)
2021-06-22 - 2661a961 - Update gocql (#1663)
2021-06-21 - 7c89d5f1 - Symlink current Elasticsearch index templates to v1 directory (#1659)
2021-06-21 - 36cd514d - Refactor search attributes GetType method (#1657)
2021-06-20 - 6b2727d4 - nil check reporters when stopping (#1656)
2021-06-17 - fc46c379 - Rename *Timestamp fields to *Time to better reflect its type (#1651)
2021-06-17 - f1352be0 - Sets Reconnection policy on gocql.ClusterObject (#1652)
2021-06-17 - b40d8f41 - Fix search attributes parsing logic (#1647)
2021-06-17 - 6459b904 - Autocreate custom search attributes fields in reindex script (#1653)
2021-06-17 - 5fb34789 - Make dynamic config case insensitive (#1635)
2021-06-17 - 3c427e6a - Add support for passing JWT audience to defaultJWTClaimMapper (#1648)
2021-06-17 - 31de6a46 - Add StateTransitionCount field to Elasticsearch index (#1650)
2021-06-17 - 2666ad92 - Restructure history service mutable state & related logics (#1641)
2021-06-17 - 0087afdf - Adds keep-alive to gRPC connections between the frontend and history/matching services #1654
2021-06-16 - f65e5b2a - Bump Web to v1.10.0 (#1609)
2021-06-16 - 9a5e9069 - Add ExecutionDuration ES field (#1645)
2021-06-16 - 7b6bcf2d - Set CGO_ENABLED to 1 for darwin and windows (#1588)
2021-06-15 - a879f7f6 - Rename Elasticsearch system search attributes (#1639)
2021-06-15 - a2aa44e4 - Refactor Elasticsearch PutMapping API (#1634)
2021-06-15 - 663c3ee4 - Use arbitrary Duration for retention (#1615)
2021-06-15 - 56e4631e - Refactor Queue for namespace replication (#1625)
2021-06-15 - 3bcfe16b - Use scaled_float instead of double for Double search attributes (#1633)
2021-06-15 - 26ac39b0 - Add mocks for data store interfaces (#1640)
2021-06-15 - 0eb0c37f - Convert Elasticsearch ExecutionStatus field to keyword (#1637)
2021-06-14 - fbb2ac2a - Fix auto-setup.sh for PostgreSQL (#1629)
2021-06-14 - f55d0bbc - Convert Elasticsearch time fields to date type (#1624)
2021-06-14 - 9fc20ca9 - Refactor Dockerfiles to better support ARM (#1575)
2021-06-14 - 9f36e826 - hookup custom data store factory (#1630)
2021-06-14 - 8e34c296 - Add minimum date validation (#1631)
2021-06-14 - 3481ed69 - Fix Elasticsearch metrics and memo encoding (#1632)
2021-06-14 - 305c18c9 - Make sync match wait time configurable (#1627)
2021-06-14 - 15666fff - Emit dedicated metrics for context cancelled (#1628)
2021-06-11 - c41ddbdb - Add ExecutionTime to WorkflowExecutionInfo (#1618)
2021-06-11 - 799ee020 - Emits warning/error logs when Temporal tries/fails to establish a new GoCQL session (#1623)
2021-06-10 - c78ea3e4 - Refactor ExecutionStore / ExecutionManager (#1614)
2021-06-10 - 79a537ec - Add logger interceptor to frontend (#1616)
2021-06-10 - 7402bd70 - Enable mutable state DB version CAS by default (#1621)
2021-06-10 - 283bb1e9 - Update SDK to 1.7.0 (#1620)
2021-06-09 - 8043f685 - Use priority mutex for mutable state cache lock (#1612)
2021-06-09 - 202e9811 - Report error for invalid retention days (#1610)
2021-06-08 - e3e7d4a5 - Introduce priority mutex & condition variable (#1611)
2021-06-08 - 80f04b52 - Remove unnecessary metrics on hisory cache (#1606)
2021-06-08 - 3311e673 - Fix error type if query task alrady gone (#1608)
2021-06-07 - ff8b2013 - Allow tctl to start workflows with infinite timeout (#1602)
2021-06-07 - 8f635379 - Remove Attr prefix from Elasticsearch schema (#1586)
2021-06-07 - 804ed608 - Task manager impl continued (#1604)
2021-06-07 - 6b0921be - Adding TaskManagerImpl (#1600)
2021-06-07 - 27456ecb - Fix opentelemetry metrics listen address (#1603)
2021-06-04 - f4bbcbc8 - Upgrade dgrijalva/jwt-go to v4.0.0-preview1 (#1597)
2021-06-04 - ea339fd6 - Remove CLI flags for security tokens (#1599)
2021-06-04 - 026a75e7 - Rename files for history manager and store (#1596)
2021-06-03 - f1c6935d - Refactor ShardManager (#1594)
2021-06-03 - d5f478b7 - Remove spew dependency (#1591)
2021-06-03 - 4951b7ee - Fix DeleteHistoryBranch (#1590)
2021-06-03 - 2eb8fac7 - Include namespace in API error log (#1593)
2021-06-02 - d36837ac - Use time.Time in internal visibility APIs (#1576)
2021-06-02 - 16584ef8 - Refactor persistence for HistoryStore (#1580)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.0)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes:

• Sets Reconnection policy on gocql.ClusterObject (#1652)
• Fix error prone retry time implementation (#1681)

Details about v1.10.0 release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.10.5)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes:

• Fix unable to reset workflow issue (#1673)
• Properly dedup signal request ID (#1558)
• Fix LastFirstEventTxnId not being properly propagated to DB (#1560)

Details about v1.9.0 release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.9.3)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes:

• Fix unable to reset workflow issue (#1673)

Details about v1.10.0 release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.10.4)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes:

• Fix error type if query task already gone (#1608)
• Emits warning/error logs when Temporal tries/fails to establish a new GoCQL session (#1623)
• Make sync match wait time configurable (#1627)
• Emit dedicated metrics for context cancelled (#1628)
• Fix auto-setup.sh for PostgreSQL (#1629)
• Add support for passing JWT audience to defaultJWTClaimMapper (#1648) to fix an issue introduced in 1.10.2 with the upgrade of jwt-go

Details about v1.10.0 release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.10.3)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes:

• Upgrade dgrijalva/jwt-go to v4.0.0-preview1 #1597 that addresses the CVE-2020-26160 vulnerability in the earlier version of jwt-go.

Details about v1.10.0 release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.10.2)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes:

• Include namespace in API error log (#1593)
• Remove CLI flags for security tokens (#1599)
• Allow tctl to start workflows with infinite timeout (#1602)

Details about v1.10.0 release can be found here

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.10.1)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

Schema Changes

This release does not includes any schema changes.

Notable Changes

Advanced Visibility

This release changes how custom search attributes are managed. If you are not using advanced visibility / Elasticsearch, you are not affected and can skip this section.

Before v1.10.0 it wasn't easy to add new custom search attributes to a cluster. It was a two step process:

1. A cluster admin had to first manually change the Elasticsearch schema.
2. Then they had to manually update the dynamic config.

v1.10.0 adds new commands and modifies existing commands to manage and view search attributes:

tctl admin cluster add-search-attributes --name ProductId --type Keyword
tctl admin cluster get-search-attributes
tctl cluster get-search-attributes

Note: Due to Elasticsearch limitations you can only add new custom search attributes but not rename or remove existing ones. This is why we removed 6 predefined Custom*Field search attributes from the default server config.
If you don't use them, you can remove them from your dynamic config before v1.10.0 deployment.

After the upgrade to v1.10.0 , when the Temporal Server starts up, it will automatically read the current search attributes from the dynamic config and save them to the cluster metadata database table. After that, the entire frontend.validSearchAttributes section can be removed from your dynamic config.

To browse existing search attributes, run tctl cluster get-search-attributes command. You will see search attribute output like the following:

Search attributes:
+-----------------------+---------+
|         NAME          |  TYPE   |
+-----------------------+---------+
| BinaryChecksums       | Keyword |
| CloseTime             | Int     |
| CustomNamespace       | Keyword |
| ExecutionStatus       | Int     |
| ExecutionTime         | Int     |
| Operator              | Keyword |
| RunId                 | Keyword |
| StartTime             | Int     |
| TaskQueue             | Keyword |
| TemporalChangeVersion | Keyword |
| WorkflowId            | Keyword |
| WorkflowType          | Keyword |
+-----------------------+---------+

To add new search attributes to your cluster you need to run the tctl admin cluster command:

tctl admin cluster add-search-attributes --name ProductId --type Keyword
You are about to add search attributes [ProductId:Keyword]. Continue? Y/N y

Custom search attributes:
+-----------+---------+
|   NAME    |  TYPE   |
+-----------+---------+
| ProductId | Keyword |
+-----------+---------+
System search attributes:
+-----------------------+---------+
|         NAME          |  TYPE   |
+-----------------------+---------+
| BinaryChecksums       | Keyword |
| CloseTime             | Int     |
| CustomNamespace       | Keyword |
| ExecutionStatus       | Int     |
| ExecutionTime         | Int     |
| Operator              | Keyword |
| RunId                 | Keyword |
| StartTime             | Int     |
| TaskQueue             | Keyword |
| TemporalChangeVersion | Keyword |
| WorkflowId            | Keyword |
| WorkflowType          | Keyword |
+-----------------------+---------+
Storage mappings:
+----------------------------+-------------+
|        COLUMN NAME         | COLUMN TYPE |
+----------------------------+-------------+
| Attr.BinaryChecksums       | keyword     |
| Attr.CustomNamespace       | keyword     |
| Attr.Operator              | keyword     |
| Attr.ProductId             | keyword     |
| Attr.TemporalChangeVersion | keyword     |
| CloseTime                  | long        |
| ExecutionStatus            | long        |
| ExecutionTime              | long        |
| NamespaceId                | keyword     |
| RunId                      | keyword     |
| StartTime                  | long        |
| TaskQueue                  | keyword     |
| WorkflowId                 | keyword     |
| WorkflowType               | keyword     |
+----------------------------+-------------+
Workflow info:
{
  "execution": {
    "workflowId": "temporal-sys-add-search-attributes-workflow",
    "runId": "45386457-3015-47cd-806a-5f40ead7f327"
  },
  "startTime": "2021-06-02T21:11:40.984120413Z",
  "closeTime": "2021-06-02T21:11:41.364028543Z",
  "status": "Completed"
}
Search attributes have been added successfully.

The tctl admin cluster get-search-attributes command will give you the exact same output.
The admin command output has more info than the non-admin tctl cluster get-search-attributes command. It shows custom and system search attributes separately. In addition to this, it shows the actual Elasticsearch schema and state of the system Workflow Execution which is adding the search attributes to the cluster (can be useful for debugging).

Search attributes are cached internally and will be available 1 minute after being added.

Dockerfile changes

Dockerfile changes are mostly related to advanced visibility. Previously, if ENABLE_ES was set to true, the startup script would create the Elasticsearch index for you. Now this happens only with the auto-setup image (used by our docker-compose files) or with helm-chart. If you are using the server image you need to create the index manually.

Also Custom*Field search attributes are added automatically for auto-setup image to support compatibility with samples.

All changes:

2021-04-27 - 08868279a - Add priority throttling (#1500)
2021-04-27 - 836a6387e - Update most of dependencies to the latest versions (#1494)
2021-04-27 - 99a5fb26a - Fix reset workflow pending activity schedule time (#1502)
2021-04-27 - a9fb3e155 - Fix history service metrics double emission issue (#1498)
2021-04-27 - f951d5890 - Update README.md (#1493)
2021-04-28 - 07abf549d - Add metrics propagation across services and no user-induced latency metric (#1476)
2021-04-28 - 1ddd9baa9 - Bump Web to v1.9.0 (#1504)
2021-04-28 - 63f2c831d - Pin go-sql-driver/mysql to v1.5.0 (#1503)
2021-04-29 - 2d7d63de5 - Default to use DB record version for mutable state (#1506)
2021-04-29 - f8e6cad5c - Add SearchAttributesManager (#1489)
2021-04-30 - aeaef6626 - Expose workflow state transition count (#1505)
2021-05-02 - 44cdadf50 - Add check for certificate expiration on start (#1511)
2021-05-03 - 1d020b5a1 - Modify Elasticsearch client for the new search attribute implementation (#1513)
2021-05-03 - 4f9473e70 - Emit shard lock metrics (#1508)
2021-05-03 - 7a69e2b3d - Disable visibility config validation if advanced visibility is configured (#1519)
2021-05-03 - 9d610b4a5 - Do not try set metrics trailer if context is closed (#1516)
2021-05-04 - 3e19d63d3 - Add thread safety to metrics context (#1517)
2021-05-04 - 9dc9ac8f1 - Report correct workflow type name tag (#1509)
2021-05-05 - 4b390d7b7 - Do not persist to DB when resetting sticky task queue (#1520)
2021-05-06 - 22e5045b4 - Align CODEOWNERS with repository access (#1525)
2021-05-06 - 242a0bebe - Fix go test coverage log warrning (#1528)
2021-05-06 - 476a12f69 - Fix readme slack invite link (#1527)
2021-05-06 - 79621111d - Add "addsearchattributes" worker (#1518)
2021-05-06 - a223a1188 - Add fossa scan to buildkite (#1526)
2021-05-07 - 26cbc5f8b - Bump current version to 1.10.0 (#1534)
2021-05-07 - 7606c6a77 - Set branch on fossa analyze (#1529)
2021-05-07 - d2c547711 - Add fossa badge to readme (#1533)
2021-05-07 - dab33d4e1 - Upsert and Delete visibility record in the configured database only (#1532)
2021-05-10 - 76f55e6e1 - Use priority based API throttling (#1512)
2021-05-10 - e7cc7f200 - Add back GlobalNamespaceRPS (#1537)
2021-05-11 - 0f81c5508 - Disabe build fail on fossa (#1541)
2021-05-11 - 3fd1fd800 - Pass logger from server options instead of calling log.NewDefaultLogger() (#1542)
2021-05-11 - 5580e8694 - Disable EnableDBRecordVersion for 1.10 (#1539)
2021-05-12 - 91a6593d2 - Rename log.NewDefaultLogger() to log.NewTestLogger() (#1543)
2021-05-13 - 223ac38ac - Bump Web to v1.9.1 (#1547)
2021-05-13 - 82b05a201 - Feature: data converter websocket (#1437)
2021-05-17 - 36c4bd90d - Use ClusterMetadata based search attributes (#1540)
2021-05-17 - 37583eb41 - Remove CustomField search attributes from index template (#1553)
2021-05-17 - a073b64f3 - Update jmoiron to the latest version 1.3.4 (#1550)
2021-05-18 - 71f073bbe - Allow search attributes when Elasticsearch is not configured (#1556)
2021-05-18 - b3f10a7df - Split integration_tests.go (#1548)
2021-05-19 - 959fb34cc - Properly dedup signal request ID (#1558)
2021-05-19 - eff23dc9a - Refactor Dockerfile and scripts (#1501)
2021-05-20 - 6a0ea05b0 - Set search attribute types for history events attributes (#1551)
2021-05-20 - 8fbc61548 - Fix LastFirstEventTxnId not being properly propagated (#1560)
2021-05-21 - 03225b694 - Add ES_USER and ES_PWD to auto-setup.sh (#1563)
2021-05-21 - 4f54be5aa - Add shellcheck to the base builder image (#1524)
2021-05-21 - fb25ef331 - Remove dynamic config UpdateValue API (#1557)
2021-05-22 - c09cd4523 - Remove Custom*Field search attributes (#1562)
2021-05-22 - e7276006f - Use shellcheck in ci-build target (#1565)
2021-05-22 - f04753f81 - Use ONBUILD in Dockerfile for develop image (#1564)
2021-05-24 - 1867fa71e - Improve namespace ack level calculation (#1568)
2021-05-24 - 23ced89a4 - Fix issue that namespace replication name is not set in req (#1569)
2021-05-24 - e15368448 - Add option to specify custom metrics reporter (#1561)
2021-05-24 - f355adc95 - Remove dry run flag and test run from documentation (#1567)
2021-05-25 - 103ba39d8 - Merge serialization methods into new Serializer (#1570)
2021-05-25 - 1688801e9 - Use 10s timeout instead of long poll (2m) timeout as default for visibility commands in tctl (#1571)
2021-05-25 - a8c960600 - Fix tctl visibility timeout to 10s (#1572)
2021-05-26 - 2e3332f29 - Bump Web to v1.9.2 (#1574)
2021-05-27 - a69bc4262 - Header providers plugin support (#1488)
2021-05-27 - b04065349 - Show connection error details for PostgreSQL layer (#1581)
2021-06-01 - 790af7c30 - Include namespace tag to API error metrics (#1582)
2021-06-02 - 44131b52c - Add Attr properties to Elasticsearch GetMapping API (#1587)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.10.0)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes:

• Upsert and Delete visibility record in the configured database only (#1532).

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.9.2)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

UPDATE: Please use v1.9.2 instead of this release.

Release Highlights

This release includes:

• Add check for certificate expiration on start (#1511).
• Disable visibility config validation if advanced visibility is configured (#1519).

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.9.1)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

Breaking Changes

Beginning from 1.9.x, tctl / temporal-cassandra-tool / temporal-sql-tool, will by default perform host name verification if TLS is enabled. Previous versions (1.0.x to 1.8.x) will not perform host name verification by default if TLS is enabled.

As the result of the above changes:

• tls_enable_host_verification flag is removed from tctl
• tls-enable-host-verification flag is removed from temporal-cassandra-tool and temporal-sql-tool
• tls_disable_host_verification flag is introduced for tctl to opt out of host name verification
• tls-disable-host-verification flag is introduced for temporal-cassandra-tool and temporal-sql-tool to opt out of host name verification

Schema Changes

This release includes schema changes.

Before upgrading your Temporal Server cluster, run the schema upgrade tool to support version 1.5.

Schema update tool instructions are here:
https://docs.temporal.io/docs/server-versions-and-dependencies/#upgrade-your-version-of-temporal.

Notable Changes

This release includes enhancements to the following areas:

• Visibility DB config is not required if Elasticsearch is used for advanced visibility(#1497)
• Periodic refreshing of TLS certificates from files (#1415)
• Ability to inject TLS certificate provider (#1391)

Persistence Layer Changes

• Begin the migration of executions table operation using atomic counter (#1420)
• Add prev transaction ID attribute to history node table for events chaining & trimming (#1435, #1441, #1449, #1450)

Performance Improvements

• Perf optimization for NDC replication reducing number of DB queries if cluster is idle (#1392)
• Perf optimization reducing unnecessary update query to DB (#1463)

All changes:

2021-03-22 - 698d8a7ff - Adding image for debugging and traffic shaping (#1394)
2021-03-23 - 7a6539cfa - Rename dc.NopClient to dc.NoopClient (#1396)
2021-03-23 - e2e260045 - Refactor cert provider interfaces and make cert provider injectable (#1391)
2021-03-24 - 9fcf5e415 - Update server and cli versions for 1.8.0 release (#1400)
2021-03-24 - b2fe4a6f0 - Remove RINGPOP_SEEDS env (#1402)
2021-03-24 - bbeab1c3b - Performance optimization for start child workflow task (#1397)
2021-03-24 - c91146e2c - Include mac arm64 for binary release (#1401)
2021-03-26 - 1ff79f26f - dynamicconfig: Make Keys public for use in extension clients (#1411)
2021-03-26 - 285514c48 - Remove not used config for public client (#1410)
2021-03-26 - 32419f66b - Report GetWFExecutionHistory long poll under custom operation tag (#1404)
2021-03-26 - 46920a1f1 - Fix reserved search attribute name validation logic (#1398)
2021-03-26 - 5ab080a69 - Add missing retention days check on update namespace API (#1407)
2021-03-26 - 917664781 - Reduce unnecessary data type conversion between mutable state & persistence (#1409)
2021-03-26 - c34d9bb7a - Improve compatibility with Aiven DB (#1413)
2021-03-26 - eed91445e - Perf optimize NDC RPC replication (#1392)
2021-03-29 - 3d1012381 - Fix metrics namespace tagging issue for matching & history service (#1414)
2021-03-29 - 7c1c89baa - Fix broken links in docs (#1417)
2021-03-29 - c2d3f6f99 - Fix NDC events reapplication potential deadlock issue (#1418)
2021-03-29 - cbece1efe - Add documentation for running dependencies on the host (#1416)
2021-03-30 - df0c86904 - Add basic connection management and keep alive parameters (#1334)
2021-03-31 - 2947301d4 - Port history builder into dedicated package (#1421)
2021-03-31 - a04944a0a - Refactor cluster metadata for search attributes (#1422)
2021-04-01 - 1e062a832 - Add support for periodic refreshing of TLS certs loaded from files (#1415)
2021-04-01 - 9d06352fc - Cleanup new history builder (#1425)
2021-04-01 - a3571d1ff - Remove params field from service and handler objects (#1424)
2021-04-02 - 30c89b9f9 - Add an ExtraData field to Claims struct (#1429)
2021-04-02 - f4fffdd1c - Bump Web from 1.7.1 to 1.8.0 (#1430)
2021-04-05 - 576724433 - Begin the migration of executions table operation using atomic counter (#1420)
2021-04-05 - ea173e065 - Port mutable state events buffering functionality to new history builder (#1431)
2021-04-06 - 394542b00 - Add prev transaction ID attribute to history node table for events chaining (#1435)
2021-04-06 - 3fc9b169b - Bump Web to v1.8.1 (#1438)
2021-04-06 - b536952d3 - Use new history builder (#1434)
2021-04-07 - 0ff2418a3 - Add feature flag to disable cross namespace commands (#1432)
2021-04-07 - 6d179156d - Fail workflow if search attributes size exceed limit (#1439)
2021-04-07 - a1a4de231 - Remove unnecessary Cassandra iter nil check (#1440)
2021-04-09 - 6b0d5ecc3 - Update Go SDK to v1.6.0 (#1447)
2021-04-09 - ca6febf83 - Adds additional logging around connection validation to temporal-cassandra-tool
2021-04-09 - d3178aa25 - Remove unused variable in CLI (#1436)
2021-04-10 - b75a7cfe2 - Don't send a name and an ID to DescribeNamespace (#1446)
2021-04-13 - 1b1aef9b6 - Implement history node deletion logic (#1443)
2021-04-13 - 6005b2b3e - Remove visibility related transfer task types (#1458)
2021-04-13 - bd7df352e - Remove TTL from open_executions Cassandra visibility table (#1456)
2021-04-13 - fd744a87a - Unify ReadHistoryBranch logic (#1441)
2021-04-14 - 5d924f47b - Implement history node validation & trimming logic (#1449)
2021-04-15 - 57d98512a - Add additional service protection against data inconsistency (#1450)
2021-04-15 - 5d17b1e0d - Introduce Number util for dynamic config (#1464)
2021-04-15 - 682b1f288 - Perf optimize mutable state DB operation (#1463)
2021-04-15 - b4c4b84ab - Relax the constraint on which event ID is allow for ResetWorkflow API (#1465)
2021-04-16 - 10822d684 - Add XDC integration test for ResetWorkflow API (#1469)
2021-04-16 - 7398beb5c - Fix periodic checks for expiring certificates (#1466)
2021-04-16 - a02eecf74 - Add more tests for history node util (#1470)
2021-04-16 - b06a313aa - Implement metrics reporting via opentelemetry (#1445)
2021-04-16 - bd21495f6 - Update gocql to latest master (87a5d7a) (#1467)
2021-04-16 - cb02a52fb - Add missing error check for schema version (#1471)
2021-04-19 - 8ed4f1728 - Introduce SQL_TLS_ENABLED env var to allow enabling TLS for SQL datastores (#1473)
2021-04-20 - 132c8c72d - Change CLI default TLS host name verification behavior (#1478)
2021-04-20 - 29e9280a8 - Export encryption.NewLocalStoreCertProvider(), so that it can be used outside of NewTLSConfigProviderFromConfig() (#1475)
2021-04-20 - 4aa0c5141 - Add TLS host name override config for Cassandra / SQL tool (#1480)
2021-04-20 - 4bd74c543 - Added TLS config for only server name if it's only option provided. (#1479)
2021-04-21 - 49d6bce9f - Add support for IPv6 (#1472)
2021-04-21 - b891cf46e - Adding an optional Reason to authorization.Result (#1462)
2021-04-23 - 0416d3173 - Add additional service protection against reordering (#1490)
2021-04-23 - 87f30afa5 - Set Temporal internode comm max recv payload size to 128MB (#1491)
2021-04-23 - f0429156f - Add logging for attempted and successful TLS connections to server (#1486)
2021-04-26 - 6c23f0249 - Skip visibility database validation if advanced visibility is enabled (#1497)
2021-04-26 - e67e845e4 - Improve integration tests stability (#1495)
2021-04-26 - fee1c4382 - Update README.md (#1487)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.9.0)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes a patch which fix potential data loss of visibility data when using Cassandra visibility persistence and infinity workflow timeout (#1456).
Also it includes #1432 which adds system.enableCrossNamespaceCommands dynamic config flag (default true) and Request field to CallTarget struct for custom authorizer.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.8.2)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes a patch which fix potential data loss of visibility data when using Cassandra visibility persistence and infinity workflow timeout (#1456).

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.7.3)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes a patch which fix potential data loss of visibility data when using Cassandra visibility persistence and infinity workflow timeout (#1456).

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.6.7)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools

Release Highlights

This release includes a few patches improving the overall stability of the Temporal service

• Fix issue that blob is mistakenly used as encoding type for DescribeMutableState API
  This bug affects the tctl admin workflow describe command, as well as making Temporal worker service to emit error logs

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.7.2)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools