Client

• Tested versions of Ruby for 8.13.0: Ruby (MRI) 3.0, 3.1, 3.2 and 3.3. JRuby 9.3 and JRuby 9.4.

Experimental ES|QL Helper

This version provides a new experimental Helper for the ES|QL query API. The helper returns an array of hashes with the columns as keys and the respective values instead of the default JSON value. Additionally, you can iterate through the response values and transform the data in by passing in a Hash of column => Proc values. You could use this for example to convert a @timestamp column value into a DateTime object. Please check out the documentation and open an issue if you encounter any problems or have any feedback.

API

API Changes:

• async_search.status - adds Time :keep_alive parameter: Specify the time interval in which the results (partial or final) for this search will be available.
• bulk - adds boolean :require_data_stream parameter: When true, requires the destination to be a data stream (existing or to-be-created). Default is false.
• connector.list - Adds the following parameters:
  • :index_name (List): A comma-separated list of connector index names to fetch connector documents for.
  • :connector_name (List): A comma-separated list of connector names to fetch connector documents for.
  • :service_type (List): A comma-separated list of connector service types to fetch connector documents for.
  • :query (String): A search string for querying connectors, filtering results by matching against connector names, descriptions, and index names.
• esql.query - adds boolean :drop_null_columns parameter: Should entirely null columns be removed from the results? Their name and type will be returning in a new all_columns section.
• field_caps - Adds :include_empty_fields boolean parameter: Include empty fields in result.
• index - adds boolean :require_data_stream parameter: When true, requires the destination to be a data stream (existing or to-be-created). Default is false.
• indices.rollover - adds boolean :lazy parameter: If set to true, the rollover action will only mark a data stream to signal that it needs to be rolled over at the next write. Only allowed on data streams.
• connector_sync_job.list - adds List :job_type parameter: A comma-separated list of job types.
• inference.delete_model, inference.get_model, inference.inference, inference.put_model: renames :model_id parameter to :inference_id.
• termvector will show a warning since it's been deprecated. Please use the plural version, termvectors.

New APIs:

• indices.resolve_cluster - Resolves the specified index expressions to return information about each cluster, including the local cluster, if included.
• profiling.flamegraph - Extracts a UI-optimized structure to render flamegraphs from Universal Profiling.
• profiling.stacktraces - Extracts raw stacktrace information from Universal Profiling.
• security.query_user - Retrieves information for Users using a subset of query DSL
• text_structure.test_grok_pattern - Tests a Grok pattern on some text.

APIs Migrated from experimental to stable:

• synonyms.delete_synonym
• synonyms.delete_synonym_rule
• synonyms.get_synonym
• synonyms.get_synonym_rule
• synonyms.get_synonyms_sets
• synonyms.put_synonym
• synonyms.put_synonym_rule

New Experimental APIs:

• connector.update_api_key_id - Updates the API key id and/or API key secret id fields in the connector document.
• connector.update_index_name - Updates the index name of the connector.
• connector.update_native - Updates the is_native flag of the connector.
• connector.update_service_type - Updates the service type of the connector.
• connector.update_status - Updates the status of the connector.
• esql.async_query - Executes an ESQL request asynchronously
• esql.async_query_get - Retrieves the results of a previously submitted async query request given its ID.

New Experimental namespace connector_secret:

• connector_secret.delete - Deletes a connector secret.
• connector_secret.get - Retrieves a secret stored by Connectors.
• connector_secret.post - Creates a secret for a Connector.
• connector_secret.put - Creates or updates a secret for a Connector.

Development

• Migrated from byebug to debug.
• Added extra testing for OpenTelemetry.

Drops runtime dependency on base64. Thanks Earlopain! Pull Request: #2295.

Drops runtime dependency on base64. Thanks Earlopain! Pull Request: #2295.

Adds base64 dependency: base64 was added to the gemspec, since starting in Ruby 3.4.0, base64 will no longer be part of the default gems and will no longer be in the standard library.

Base64 is used for API key and Cloud ID. The dependency used to be declared in transport, but it's not needed there since the implementation using it is in this codebase. It was removed from transport in the latest patch releases: 8.1.3, 8.2.5 and 8.3.1.

Adds base64 dependency: base64 was added to the gemspec, since starting in Ruby 3.4.0, base64 will no longer be part of the default gems and will no longer be in the standard library.

Base64 is used for API key and Cloud ID. The dependency used to be declared in transport, but it's not needed there since the implementation using it is in this codebase. It was removed from transport in the latest patch releases: 8.1.3, 8.2.5 and 8.3.1.

Client

• Tested versions of Ruby for 8.12.0: Ruby (MRI) 3.0, 3.1, 3.2 and 3.3. JRuby 9.3 and JRuby 9.4.

API

API Changes:

• bulk - Adds boolean :list_executed_pipelines parameter: Sets list_executed_pipelines for all incoming documents. Defaults to unset (false).
• indices.put_settings - Adds boolean :reopen parameter: Whether to close and reopen the index to apply non-dynamic settings. If set to true the indices to which the settings are being applied will be closed temporarily and then reopened in order to apply the changes. The default is false.
• open_point_in_time - Adds Hash :body parameter: an index_filter specified with the Query DSL.
• security.get_api_key - Adds boolean :active_only parameter: flag to limit response to only active (not invalidated or expired) API keys.

New APIs

New API for Universal profiling:

• profiling.status - Returns basic information about the status of Universal Profiling.

New experimental API:

• simulate.ingest - Simulates running ingest with example documents. See: https://www.elastic.co/guide/en/elasticsearch/reference/8.12/simulate-ingest-api.html

Connectors API

Version 8.12 introduces the experimental Connectors API. Use the following APIs to manage connectors:

• connector.post - Creates a connector. See documentation
• connector.put - Creates or updates a connector. See documentation
• connector.delete - Deletes a connector. See documentation
• connector.get - Returns the details about a connector. See documentation
• connector.list - Lists all connectors. See documentation
• connector.check_in - Updates the last_seen timestamp in the connector document. See documentation
• connector.update_configuration - Updates the connector configuration. See documentation
• connector.update_error - Updates the error field in the connector document. See documentation
• connector.update_filtering - Updates the filtering field in the connector document. See documentation
• connector.last_sync - Updates the stats of last sync in the connector document. See documentation
• connector.update_name - Updates the name and/or description fields in the connector document. See documentation
• connector.update_pipeline - Updates the pipeline field in the connector document. See documentation
• connector.update_scheduling - Updates the scheduling field in the connector document. See documentation

Use the following APIs to manage sync jobs:

• connector_sync_job.cancel - Cancels a connector sync job. See documentation
• connector_sync_job.check_in - Checks in a connector sync job (refreshes 'last_seen'). See documentation
• connector_sync_job.delete - Deletes a connector sync job. See documentation
• connector_sync_job.error - Sets an error for a connector sync job. See documentation
• connector_sync_job.get - Returns the details about a connector sync job. See documentation
• connector_sync_job.list - Lists all connector sync jobs. See documentation
• connector_sync_job.post - Creates a connector sync job. See documentation
• connector_sync_job.update_stats - Updates the stats fields in the connector sync job document. See documentation

Client

• Tested versions of Ruby for 8.11.0: Ruby (MRI) 3.0, 3.1 and 3.2. JRuby 9.3 and JRuby 9.4.
• Adds native support for Open Telemetry. See Open Telemetry for documentation.
• Improved documentation, now you can find more examples in Ruby in the REST API reference.

API

New Experimental APIs:

• esql.query - Executes an ESQL request.
• inference.delete_model - Delete model in the Inference API.
• inference.get_model - Get a model in the Inference API.
• inference.inference - Perform inference on a model.
• inference.put_model - Configure a model for use in the Inference API.

Backports support for Faraday 2 from elastic-transport. ¡Gracias santiagorodriguez96!

This version of the gem now supports Faraday v2. If you don't have a locked version of Faraday in your project, when you upgrade your gems, Faraday v2 will be installed. The main change on dependencies when using Faraday v2 is all adapters, except for the default net_http one, have been moved out of Faraday into separate gems. This means if you're not using the default adapter and you migrate to Faraday v2, you'll need to add the adapter gems to your Gemfile.

These are the gems required for the different adapters with Faraday 2, instead of the libraries on which they were based:

# HTTPCLient
gem 'faraday-httpclient'

# NetHTTPPersistent
gem 'faraday-net_http_persistent'

# Patron
gem 'faraday-patron'

# Typhoeus
gem 'faraday-typhoeus'

Things should work fine if you migrate to Faraday 2 as long as you include the adapter (unless you're using the default one net-http), but worst case scenario, you can always lock the version of Faraday in your project to 1.x:
gem 'faraday', '~> 1'

Be aware if migrating to Faraday v2 that it requires at least Ruby 2.6, unlike Faraday v1 which requires 2.4.

Troubleshooting

If you see a message like:
:adapter is not registered on Faraday::Adapter (Faraday::Error)
Then you probably need to include the adapter library in your gemfile and require it.

Please submit an issue if you encounter any problems.

• Backports fix from elastic-transport: #66 - Manticore transport unable to send custom headers with perform_request Pull Request.

Client

• Tested versions of Ruby for 8.10.0: Ruby (MRI) 3.0, 3.1 and 3.2. JRuby 9.3 and JRuby 9.4.

API

New Experimental APIs, for internal use:

• fleet.delete_secret
• fleet.get_secret
• fleet.post_secret

New stable APIs:

• security.get_settings - Retrieve settings for the security system indices
• security.update_settings - Update settings for the security system indices

New Experimental API:

• query_ruleset.list List query rulesets.

API Changes:

• indices.reload_search_analyzers - Adds parameter resource changed resource to reload analyzers from if applicable

Promoted from Experimental to Beta:

• security.create_cross_cluster_api_key
• security.update_cross_cluster_api_key

Synonyms namespace update:

All synonym related APIs have been moved to the synonyms namespace and some of the endpoints have been renamed, as well as their parameters:

• synonyms.delete => synonyms.delete_synonym - requires id, the id of the synonyms set to be deleted.
• synonyms.get => synonyms.get_synonym - requires id, the name of the synonyms set to be retrieved.
• synonyms_set.get_synonyms_sets => synonyms.get_synonyms_sets
• synonyms.put => synonyms.put_synonym - requires id of the synonyms set to be created or updated.
• synonym_rule.put => synonyms.put_synonym_rule - Parameters changed to set_id (the id of the synonym set to be updated with the synonym rule) and rule_id (the id of the synonym rule to be updated or created).
• New Experimental API synonyms.delete_synonym_rule - Deletes a synonym rule in a synonym set
• New Experimental API synonyms.get_synonym_rule - Retrieves a synonym rule from a synonym set

• Patch releases back to being detached from Elastic stack releases.
• Tested compatibility with Elasticsearch v7.17 APIs.
• Tested versions of Ruby for 7.17.8: Ruby (MRI) 2.7, 3.0, 3.1, 3.2, JRuby 9.3, JRuby 9.4.
• Bugfix in elasticsearch-transport: Fixes enforcing UTF-8 in Response body, causing an error when the string is frozen, particularly when using webmock: issue #63.

Client

• Tested versions of Ruby for 8.9.0: Ruby (MRI) 3.0, 3.1 and 3.2. JRuby 9.3 and JRuby 9.4.
• Updated product validation. The code for the product validation was refactored in a few ways:
  • Just check header, does not check the version of the server.
  • Warns only once when there's a general server error.
  • Removes the call to '/' (client.info) when doing the first request, checking on the first actual request from the client.
• Fixes User-Agent code. In the migration to 8.x, the user agent code was extracted into transport, since we're now using that library in other projects. So for the Elasticsearch Client, the user-agent would be reported as the one defined in elastic-transport. This release fixes the issue and brings back the user agent in the format that was being used in 7.x

Helpers

This release introduces two new Helpers in the client:

• BulkHelper - This helper provides a better developer experience when using the Bulk API. At its simplest, you can send it a collection of hashes in an array, and it will bulk ingest them into {es}.
• ScrollHelper - This helper provides an easy way to get results from a Scroll.

See Helpers to read more about them.

API

New APIs

• cluster.info - Returns different information about the cluster.

New Experimental APIs and namespaces:

This functionality is Experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

• New namespace: query_ruleset
  • query_ruleset.delete - Deletes a query ruleset.
  • query_ruleset.get - Returns the details about a query ruleset.
  • query_ruleset.put - Creates or updates a query ruleset.
• New API: search_application.render_query Renders a query for given search application search parameters.
• New API: security.create_cross_cluster_api_key - Creates a cross-cluster API key for API key based remote cluster access.
• New API: security.upate_cross_cluster_api_key - Updates attributes of an existing cross-cluster API key.
• New namespace: synonyms
  • synonyms.delete- Deletes a synonym set
  • synonyms.get - Retrieves a synonym set
  • synonyms.put - Creates or updates a synonyms set
• New namespace: synonym_rule
  • synonym_rule.put - Creates or updates a synonym rule in a synonym set
• New namespace: synonyms
  • synonyms_set.get - Retrieves a summary of all defined synonym sets

• Tested versions of Ruby for 8.8.0: Ruby (MRI) 3.0, 3.1 and 3.2. JRuby 9.3 and JRuby 9.4.

API

• Updates development dependency minitest-reporters to >= 1.6 to include showing failures at the end of the test run.

New APIs

• watcher.get_settings - Retrieve settings for the watcher system index.
• watcher.update_settings - Update settings for the watcher system index.

New Experimental APIs

• indices.delete_data_lifecycle- Deletes the data lifecycle of the selected data streams
• indices.explain_data_lifecycle - Retrieves information about the index's current DLM lifecycle, such as any potential encountered error, time since creation etc.
• indices.get_data_lifecycle - Returns the data lifecycle of the selected data streams.
• indices.put_data_lifecycle - Updates the data lifecycle of the selected data streams.
• search_application.delete - Deletes a search application.
• search_application.delete_behavioral_analytics - Delete a behavioral analytics collection.
• search_application.get - Returns the details about a search application.
• search_application.get_behavioral_analytics - Returns the existing behavioral analytics collections.
• search_application.list - Returns the existing search applications.
• search_application.post_behavioral_analytics_event - Creates a behavioral analytics event for existing collection.
• search_application.put - Creates or updates a search application.
• search_application.put_behavioral_analytics - Creates a behavioral analytics collection.
• search_application.search - Perform a search against a search application.

API Changes

• clear_scroll now works with the argument ignore: 404. Issue on GitHub.
• The code generator was updated to fix a bug for ignore: 404. APIs that were supposed to support this wouldn't parse the parameters correctly. The support it now: security.get_role, watcher.delete_watch
• cluster.get_component_template, indices.get_data_stream, indices.get_index_template, indices.simulate_index_template, indices.simulate_template - Add include_defaults (Boolean) parameter: Return all default configurations for the component template (default: false).
• machine_learning.put_trained_model - Adds wait_for_completion (Boolean) parameter: Whether to wait for all child operations(e.g. model download) to complete, before returning or not (default: false).
• machine_learning.start_trained_model_deployment - Adds deployiment_id (String) parameter: The Id of the new deployment. Defaults to the model_id if not set.
• search - Adds include_named_queries_score (Boolean) parameter: Indicates whether hit.matched_queries should be rendered as a map that includes the name of the matched query associated with its score (true) or as an array containing the name of the matched queries (false).
• transform.delete_transform - Adds delete_dest_index (Boolean) parameter: When true, the destination index is deleted together with the transform. The default value is false, meaning that the destination index will not be deleted.

API Bugfix

• Updates logstash.get_pipeline, fixed in the specification id is not a required parameter, so removes raising ArgumentError when id is not present.

• Tested versions of Ruby for 8.6.0: Ruby (MRI) 2.7, 3.0, 3.1 and 3.2. JRuby 9.3 and JRuby 9.4. Ruby 2.7's end of life is coming in a few days, so this will probably be the last release to test for Ruby 2.7.

New APIs

• health_report - Returns the health of the cluster.
• transform.schedule_now_transform - Schedules now a transform.

API Changes

• transform.get_transform_stats - Adds timeout (Time) parameter. Controls the time to wait for the stats.
• transform.start_transform - Adds from (String) parameter. Restricts the set of transformed entities to those changed after this time.
• ml.delete_job, ml.reset_job - Add delete_user_annotations (Boolean) parameter. Should annotations added by the user be deleted.
• ml.clear_trained_model_deployment_cache, ml.infer_trained_model, ml.put_trained_model_definition_part, ml.put_trained_model_vocabulary, ml.start_trained_model_deployment, ml.stop_trained_model_deployment - These APIs are no longer in Beta.

• Tested versions of Ruby for 8.6.0: Ruby (MRI) 2.7, 3.0, 3.1 and 3.2. JRuby 9.3 and JRuby 9.4.

New APIs

• update_trained_model_deployment - Updates certain properties of trained model deployment (This functionality is in Beta and is subject to change).

API Changes

• cluster.reroute - :metric parameter adds none as an option.
• ml.start_trained_model_deployment - New parameter :priority (String), the deployment priority

API Bugfix

Fixes security.create_service_token API, uses POST when token name isn't present.
Thanks @carlosdelest for reporting in #1961.

Bugfix

Fixes bug when instantiating client with api_key: When passing in api_key and transport_options that don't include headers to the client, the api_key code would overwrite the arguments passed in for transport_options. This was fixed in this Pull Request.
Thanks svdasein for reporting in #1940.

• Tested versions of Ruby for 8.5.0: Ruby (MRI) 2.7, 3.0 and 3.1, JRuby 9.3.

Client

With the latest release of elastic-transport - v8.1.0 - this gem now supports Faraday v2. Elasticsearch Ruby has an open dependency on elastic-transport ('elastic-transport', '~> 8'), so when you upgrade your gems, 8.1.0 will be installed. This supports both Faraday v1 and Faraday v2. The main change on dependencies when using Faraday v2 is all adapters, except for the default net_http one, have been moved out of Faraday into separate gems. This means if you're not using the default adapter and you migrate to Faraday v2, you'll need to add the adapter gems to your Gemfile.

These are the gems required for the different adapters with Faraday 2, instead of the libraries on which they were based:

# HTTPCLient
gem 'faraday-httpclient'
# NetHTTPPersistent
gem 'faraday-net_http_persistent'
# Patron
gem 'faraday-patron'
# Typhoeus
gem 'faraday-typhoeus'

Things should work fine if you migrate to Faraday 2 as long as you include the adapter (unless you're using the default one net-http), but worst case scenario, you can always lock the version of Faraday in your project to 1.x:
gem 'faraday', '~> 1'

Be aware if migrating to Faraday v2 that it requires at least Ruby 2.6, unlike Faraday v1 which requires 2.4.

Troubleshooting

If you see a message like:
:adapter is not registered on Faraday::Adapter (Faraday::Error)
Then you probably need to include the adapter library in your gemfile and require it.

Please submit an issue if you encounter any problems.

API

New APIs

• machine_learning.clear_trained_model_deployment_cache - Clear the cached results from a trained model deployment (Beta).
• security.bulk_update_api_keys - Updates the attributes of multiple existing API keys.

API Changes

• rollup.rollup renamed to indices.downsample. The method now receives the index to downsample (Required) and instead of rollup_index, use target_index as the index to store downsampled data.

• security.get_api_key and security.query_api_keys add :with_limited_by flag to show the limited-by role descriptors of API Keys.

• security.get_user adds :with_profile_uid flag to retrieve profile uid (if exists) associated to the user.

• security.get_user_profile now retrieves user profiles for given unique ID(s). :uid is now a list of comma-separated list of unique identifier for user profiles.

• text_structure.find_structure adds :ecs_compatibility, optional parameter to specify the compatibility mode with ECS Grok patterns - may be either 'v1' or 'disabled'.

Machine learning APIs promoted from Experimental to Beta:

• machine_learning.clear_trained_model_deployment_cache.rb
• machine_learning.infer_trained_model.rb
• machine_learning.put_trained_model_definition_part.rb
• machine_learning.put_trained_model_vocabulary.rb
• machine_learning.start_trained_model_deployment.rb
• machine_learning.stop_trained_model_deployment.rb

Security usef profile APIs promoted from Experimental to Stable:

• security/activate_user_profile
• security/disable_user_profile
• security/enable_user_profile
• security/get_user_profile
• security/has_privileges_user_profile
• security/suggest_user_profile
• security/update_user_profile_data

• Compatibility with Elasticsearch v7.17.7 APIs.
• Tested versions of Ruby for 7.17.7: Ruby (MRI) 2.6, 2.7, 3.0 and 3.1, JRuby 9.3.