elasticsearch-ruby

• Tested versions of Ruby for 8.4.0: Ruby (MRI) 2.7, 3.0 and 3.1, JRuby 9.3.

API

New APIs

• security.update_api_key - Updates attributes of an existing API key. Documentation.

API Changes

• get - Adds new parameter force_synthetic_source (Boolean) Should this request force synthetic _source? Use this to test if the mapping supports synthetic _source and to get a sense of the worst case performance. Fetches with this enabled will be slower the enabling synthetic source natively in the index.
• machine_learning.start_trained_model_deployment - Adds new parameter cache_size (String) A byte-size value for configuring the inference cache size. For example, 20mb.
• mget - Adds new parameter force_synthetic_source (Boolean) Should this request force synthetic _source? Use this to test if the mapping supports synthetic _source and to get a sense of the worst case performance. Fetches with this enabled will be slower the enabling synthetic source natively in the index.
• search - Adds new parameter force_synthetic_source (Boolean) Should this request force synthetic _source? Use this to test if the mapping supports synthetic _source and to get a sense of the worst case performance. Fetches with this enabled will be slower the enabling synthetic source natively in the index.
• snapshot.get - Adds new parameters:
  • sort (String) Allows setting a sort order for the result. Defaults to start_time (options: start_time, duration, name, repository, index_count, shard_count, failed_shard_count).
  • size (Integer) Maximum number of snapshots to return. Defaults to 0 which means return all that match without limit.
  • order (String) Sort order (options: asc, desc).
  • from_sort_value (String) Value of the current sort column at which to start retrieval.
  • after (String) Offset identifier to start pagination from as returned by the 'next' field in the response body.
  • offset (Integer) Numeric offset to start pagination based on the snapshots matching the request. Defaults to 0.
  • slm_policy_filter (String) Filter snapshots by a comma-separated list of SLM policy names that snapshots belong to. Accepts wildcards. Use the special pattern '_none' to match snapshots without an SLM policy.

• Tested versions of Ruby for 8.3.0: Ruby (MRI) 2.7, 3.0 and 3.1, JRuby 9.3.

API

• Added build hash to auto generated code. The code generator obtains the git hash from the Elasticsearch specification and adds it as a comment in the code. This allows us to track the version for each generated class.
• Updated for compatibility with Elasticsearch 8.3's API.

API Changes

• cluster.delete_voting_config_exclusions, cluster.post_voting_config_exclusions - Add new parameter master_timeout (Time) Timeout for submitting request to master.
• machine_learning.infer_trained_model_deployment is renamed to machine_learning.infer_trained_model. The url /_ml/trained_models/{model_id}/deployment/_infer is deprecated since 8.3, use /_ml/trained_models/{model_id}/_infer instead.
• machine_learning.preview_datafeed - Adds new parameters:
  • start (String) The start time from where the datafeed preview should begin
  • end (String) The end time when the datafeed preview should stop
• machine_learning.start_trained_model_deployment - Adds new parameters:
  • number_of_allocations (Integer) The number of model allocations on each node where the model is deployed.
  • threads_per_allocation (Integer) The number of threads used by each model allocation during inference.
  • queue_capacity (Integer) Controls how many inference requests are allowed in the queue at a time.
• search_mvt - Adds new parameter: with_labels (Boolean) If true, the hits and aggs layers will contain additional point features with suggested label positions for the original features
• snapshot.get - Adds new parameter: index_names (Boolean) Whether to include the name of each index in the snapshot. Defaults to true.

New Experimental APIs

• security.has_privileges_user_profile Determines whether the users associated with the specified profile IDs have all the requested privileges

• Updates dependency on elastic-transport to ~> 8.0

Tested versions of Ruby for 8.2.0: Ruby (MRI) 2.7, 3.0 and 3.1, JRuby 9.3.

API

Updated for compatibility with Elasticsearch 8.2's API.

New parameters:

• field_caps
  • filters An optional set of filters: can include +metadata,-metadata,-nested,-multifield,-parent
  • types Only return results for fields that have one of the types in the list

New APIs:

• cat.component_templates - Returns information about existing component_templates templates.
• ml.get_memory_stats - Returns information on how ML is using memory.

New Experimental APIs:

• security.activate_user_profile - Creates or updates the user profile on behalf of another user.
• security.disable_user_profile - Disables a user profile so it's not visible in user profile searches.
• security.enable_user_profile - Enables a user profile so it's visible in user profile searches.
• security.get_user_profile - Retrieves a user profile for the given unique ID.
• security.suggest_user_profiles - Get suggestions for user profiles that match specified search criteria.
• security.update_user_profile_data - Update application specific data for the user profile of the given unique ID.

API

• Fixes an issue with the generated API code. When updating the code generator for 8.x, the order of arguments.clone in the generated code was changed. This would make it so that we would modify the parameters passed in before cloning them, which is undesired. Issue: #1727.

API

• Fixes an issue with the generated API code. When updating the code generator for 8.x, the order of arguments.clone in the generated code was changed. This would make it so that we would modify the parameters passed in before cloning them, which is undesired. Issue: #1727.

• Tested versions of Ruby for 8.1.0: Ruby (MRI) 2.6, 2.7, 3.0 and 3.1, JRuby 9.3.

API

Updated for compatibility with Elasticsearch 8.1's API.

New parameters:

• indices.forcemerge - wait_for_completion Should the request wait until the force merge is completed.
• indices.get - features Return only information on specified index features (options: aliases, mappings, settings).
• ingest.put_pipeline if_version (Integer), required version for optimistic concurrency control for pipeline updates.
• ml.delete_trained_model - timeout controls the amount of time to wait for the model to be deleted. force (Boolean) true if the model should be forcefully deleted.
• ml.stop_trained_model_deployment - allow_no_match whether to ignore if a wildcard expression matches no deployments. (This includes _all string or when no deployments have been specified). force true if the deployment should be forcefully stopped. Adds body parameter, the stop deployment parameters.
• nodes.hot_threads - sort the sort order for 'cpu' type (default: total) (options: cpu, total)

Updated parameters:

• indices.get_index_template - name is now a String, a pattern that returned template names must match.
• knn_search - index removes option to use empty string to perform the operation on all indices.
• ml.close_job, ml.get_job_stats, ml.get_jobs, ml.get_overall_buckets - Remove allow_no_jobs parameter.
• ml.get_datafeed_stats, ml.get_datafeeds - Remove allow_no_datafeeds parameter.
• nodes.hot_threads - type parameter adds mem option.
• nodes.info - metric updated to use _all to retrieve all metrics and _none to retrieve the node identity without any additional metrics. (options: settings, os, process, jvm, thread_pool, transport, http, plugins, ingest, indices, aggregations, _all, _none). index_metric option shards changes to shard_stats.
• open_point_in_time - keep_alive is now a required parameter.
• search_mvt - grid_type parameter adds centroid option in addition to grid and point.

New APIs

• OpenID Connect Authentication: security.oidc_authenticate, security.oidc_logout, security.oidc_prepare_authentication.
• transform.reset_transform.
• New experimental APIs, designed for internal use by the fleet server project: fleet.search, fleet.msearch.

• Improves handling of YAML parsing, uses safe_load instead of load when doing the product verification (should only affect Ruby < 3.0).
• Updates headers setup when using the Manticore adapter. This fixes an issue where the user-agent header was being foverridden even when it was being set on initialization via the transport options. Pull Request, issue.

8.0.0

First release for the 8.x branch with a few major changes.

• Tested versions of Ruby for 8.0.0: Ruby (MRI) 2.6, 2.7, 3.0 and 3.1, JRuby 9.3.

Client

Elastic Transport

The code for the dependency elasticsearch-transport has been promoted to its own repository and the project and gem have been renamed to elastic-transport. This gem now powers elasticsearch and elastic-enterprise-search. The elasticsearch-transport gem won't be maintained after the last release in the 7.x branch, in favour of elastic-transport.

This will allow us to better address maintainance in both clients and the library itself.

API

The elasticsearch-api library has been generated based on the Elasticsearch 8.0.0 REST specification.

X-Pack Deprecation

X-Pack has been deprecated. The elasticsearch-xpack gem will no longer be maintained after the last release in the 7.x branch. The "X-Pack" integration library codebase was merged into elasticsearch-api. All the functionality is available from elasticsearch-api. The xpack namespace was removed for accessing any APIs other than _xpack (client.xpack.info) and _xpack/usage (client.xpack.usage). But APIs which were previously available through the xpack namespace e.g.: client.xpack.machine_learning are now only available directly: client.machine_learning.

Parameter checking was removed

The code in elasticsearch-api will no longer validate all the parameters sent. It will only validate the required parameters such as those needed to build the path for the request. But other API parameters are going to be validated by Elasticsearch. This provides better forwards and backwards compatibility in the client.

Response object

In previous versions of the client, calling an API endpoint would return the JSON body of the response. With 8.0, we are returning a new Response object Elasticsearch::API::Response. It still behaves like a Hash to maintain backwards compatibility, but adds the status and headers methods from the Elastic::Transport:Transport::Response object:

elastic_ruby(main)> response = client.info
=> #<Elasticsearch::API::Response:0x000055752b0c50a8
 @response=
  #<Elastic::Transport::Transport::Response:0x000055752b0c50f8
   @body=
    {"name"=>"instance",
     "cluster_name"=>"elasticsearch-8-0-0-SNAPSHOT-rest-test",
     "cluster_uuid"=>"oIfRARuYRGuVYybjxQJ87w",
     "version"=>
      {"number"=>"8.0.0-SNAPSHOT",
       "build_flavor"=>"default",
       "build_type"=>"docker",
       "build_hash"=>"7e23c54eb31cc101d1a4811b9ab9c4fd33ed6a8d",
       "build_date"=>"2021-11-04T00:21:32.464485627Z",
       "build_snapshot"=>true,
       "lucene_version"=>"9.0.0",
       "minimum_wire_compatibility_version"=>"7.16.0",
       "minimum_index_compatibility_version"=>"7.0.0"},
     "tagline"=>"You Know, for Search"},
   @headers={"X-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"567"},
   @status=200>>
elastic_ruby(main)> response.status
=> 200
elastic_ruby(main)> response.headers
=> {"X-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"567"}
elastic_ruby(main)> response['name']
=> "instance"
elastic_ruby(main)> response['tagline']
=> "You Know, for Search"

Please let us know if you find any issues.

• Drops Ruby 2.5 from the test matrix. Support for Ruby 2.5 was dropped March 2021.
• Updates the product verification when the response is a 413 error.

API

Bugfix for #1475, an issue where if you indexed a document with an id such as an id, it would get escaped to an+id instead of an%20id when using index or create. This would result in the document id being an+id instead of the intended value an id.

Patch release corresponding with Elastic Stack version 7.16.1 that addresses the Apache Log4j2 vulnerability, more information.

Client

The only changes in the client since 7.16.0 are a few minor updates for the Compatibility mode with 8.0. We added the compatibility header in 7.13.0, but now we have integration tests and compatibility tests for version 7.x of the client with Elasticsearch 8.0.

7.16.0

Client

• Adds the delay_on_retry parameter, a value in milliseconds to wait between each failed connection, thanks DinoPullerUqido! Pull Request and backport.
• Adds CA fingerprinting. You can configure the client to only trust certificates that are signed by a specific CA certificate (CA certificate pinning) by providing a ca_fingerprint option. This will verify that the fingerprint of the CA certificate that has signed the certificate of the server matches the supplied value:

ca_fingerprint = '64F2593F...'
client = Elasticsearch::Client.new(
  host: 'https://elastic:changeme@localhost:9200',
  transport_options: { ssl: { verify: false } },
  ca_fingerprint: ca_fingerprint
)

The verification will be run once per connection.

• Fixes compression. When compression is set to true, the client will now gzip the request body properly and use the appropiate headers. Thanks johnnyshields! Pull Request and backport.

API

Updates

• Cleaned up some deprecated code.
• count - The API is documented as using GET, but it supports both GET and POST on the Elasticsearch side. So it was updated to only use POST when there's a body present, or else use GET. Elasticsearch would still accept a body with GET, but to be more semantically correct in the clients we use POST when there's a body.
• delete_index_template was updated to support the ignore_404 parameter to ignore 404 errors when attempting to delete a non-existing template.
• ingest.put_pipeline adds new parameter if_version: Required version for optimistic concurrency control for pipeline updates.
• ml.put_trained_model: adds new parameter defer_definition_decompression: If set to true and a compressed_definition is provided, the request defers definition decompression and skips relevant validations.
• nodes.hot_threads adds new parameter sort: The sort order for 'cpu' type (default: total) (options: cpu, total).
• open_point_in_time: keep_alive is now a required parameter.
• search_mvt: adds new parameter track_total_hits: Indicate if the number of documents that match the query should be tracked. A number can also be specified, to accurately track the total hit count up to the number.
• transform.preview_transform: adds new parameter transform_id. Body is now optional and the API will use GET or POST depending on the presence of a body.

APIs promoted from experimental to stable since last version:

• fleet.global_checkpoints
• get_script_context
• get_script_language
• indices.resolve_index
• monitoring.bulk
• rank_eval
• searchable_snapshots.mount
• searchable_snapshots.stats
• security.clear_cached_service_tokens
• security.create_service_token
• security.delete_service_token
• security.get_service_accounts
• security.get_service_credentials
• shutdown.delete_node
• shutdown.get_node
• shutdown.put_node
• terms_enum

New APIs

• fleet.mseach
• fleet.search
• indices.modify_data_stream
• ml.infer_trained_model_deployment
• ml.start_trained_model_deployment
• ml.stop_trained_model_deployment
• migration.get_feature_upgrade_status
• migration.post_feature_upgrade_status
• security.enroll_kibana
• security.enroll_node
• transform.updgrade_transforms

First pre-release of elasticsearch 8.x 🥳

Client

Elastic Transport

The code for the dependency elasticsearch-transport has been promoted to its own repository and the project and gem have been renamed to elastic-transport. This gem now powers elasticsearch and elastic-enterprise-search. The elasticsearch-transport gem won't be maintained after the last release in the 7.x branch, in favour of elastic-transport.

API

X-Pack Deprecation

X-Pack has been deprecated. The elasticsearch-xpack gem will no longer be maintained after the last release in the 7.x branch. The "X-Pack" integration library codebase was merged into elasticsearch-api. All the functionality is available from elasticsearch-api. The xpack namespace was removed for accessing any APIs other than _xpack (client.xpack.info) and _xpack/usage (client.xpack.usage). But APIs which were previously available through the xpack namespace e.g.: client.xpack.machine_learning are now only available directly: client.machine_learning.

Parameter checking was removed

The code in elasticsearch-api will no longer validate all the parameters sent. It will only validate the required parameters such as those needed to build the path for the request. But other API parameters are going to be validated by Elasticsearch. This provides better forwards and backwards compatibility in the client.

Response object

In previous versions of the client, calling an API endpoint would return the JSON body of the response. With 8.0, we are returning a new Response object Elasticsearch::API::Response. It still behaves like a Hash to maintain backwards compatibility, but adds the status and headers methods from the Elastic::Transport:Transport::Response object:

elastic_ruby(main)> response = client.info
=> #<Elasticsearch::API::Response:0x000055752b0c50a8
 @response=
  #<Elastic::Transport::Transport::Response:0x000055752b0c50f8
   @body=
    {"name"=>"instance",
     "cluster_name"=>"elasticsearch-8-0-0-SNAPSHOT-rest-test",
     "cluster_uuid"=>"oIfRARuYRGuVYybjxQJ87w",
     "version"=>
      {"number"=>"8.0.0-SNAPSHOT",
       "build_flavor"=>"default",
       "build_type"=>"docker",
       "build_hash"=>"7e23c54eb31cc101d1a4811b9ab9c4fd33ed6a8d",
       "build_date"=>"2021-11-04T00:21:32.464485627Z",
       "build_snapshot"=>true,
       "lucene_version"=>"9.0.0",
       "minimum_wire_compatibility_version"=>"7.16.0",
       "minimum_index_compatibility_version"=>"7.0.0"},
     "tagline"=>"You Know, for Search"},
   @headers={"X-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"567"},
   @status=200>>
elastic_ruby(main)> response.status
=> 200
elastic_ruby(main)> response.headers
=> {"X-elastic-product"=>"Elasticsearch", "content-type"=>"application/json", "content-length"=>"567"}
elastic_ruby(main)> response['name']
=> "instance"
elastic_ruby(main)> response['tagline']
=> "You Know, for Search"

7.15.0

Client

• Compatibility with Elasticsearch v7.15.0 APIs.
• We've tested and added documentation on best practices for leveraging the client in a Function-as-a-Service (FaaS) environment to the official docs.

API

• New experimental endpoints: indices.disk_usage. indices.field_usage_stats, nodes.clear_repositories_metering_archive, get_repositories_metering_info, search_mvt
• The index parameter is now required for open_point_in_time.
• The index_metric parameter in nodes.stats adds the shards option.

X-Pack

• New parameters for ml.put_job: ignore_unavailable, allow_no_indices, ignore_throttled, expand_wildcards.
• New endpoint: security.query_api_keys.

Client

• Fixes for Manticore Implementation: Addresses custom headers on initialization (https://github.com/elastic/elasticsearch-ruby/commit/3732dd4f6de75365460fa99c1cd89668b107ef1c) and fixes tracing (https://github.com/elastic/elasticsearch-ruby/commit/3c48ebd9a783988d1f71bfb9940459832ccd63e4). Related to #1426 and #1428.

This is a pre-release of 7.14.1.

Client

• Updates Manticore implementation to keep headers from initialization, fixes a bug introduced in 7.2. Commit

Client

Added check that client is connected to an Elasticsearch cluster. If the client isn't connected to a supported Elasticsearch cluster the UnsupportedProductError exception will be raised.

This release changes the way in which the transport layer and the client interact. Previously, when using elasticsearch-transport, Elasticsearch::Transport::Client had a convenient wrapper, so it could be used as Elasticsearch::Client. Now, we are decoupling the transport layer from the Elasticsearch client. If you're using the elasticsearch gem, not much will change. It will instantiate a new Elasticsearch::Transport::Client when you instantiate Elasticsearch::Client and the endpoints from elasticsearch-api will be available.

Elasticsearch::Client has an attr_accessor for the transport instance:

> client = Elasticsearch::Client.new
> client.transport.class
=> Elasticsearch::Transport::Client
> client.transport.transport.class
=> Elasticsearch::Transport::Transport::HTTP::Faraday

The interaction with elasticsearch-api remains unchanged. You can use the API endpoints just like before:

> client.info
=> {"name"=>"instance",
 "cluster_name"=>"elasticsearch",
 "cluster_uuid"=>"id",
 "version"=>
  {"number"=>"7.14.0",
  ...
},
 "tagline"=>"You Know, for Search"}

Or perform request directly from the client which will return an Elasticsearch::Transport::Response object:

> client.perform_request('GET', '/')
# This is the same as doing client.transport.perform_request('GET', '/')
=> #<Elasticsearch::Transport::Transport::Response:0x000055c80bf94bc8
 @body=
  {"name"=>"instance",
   "cluster_name"=>"elasticsearch",
   "cluster_uuid"=>"id",
   "version"=>
    {"number"=>"7.14.0-SNAPSHOT",
    ...
    },
   "tagline"=>"You Know, for Search"},
 @headers=
  {"content-type"=>"application/json; charset=UTF-8",
   "content-length"=>"571",
   ...
   },
 @status=200>

If you have any problems, please report them in this issue.

API

Code is now generated from Elastic artifacts instead of checked out code of Elasticsearch. See the Generator README for more info.

• Endpoints msearch, msearch_template and search_template remove query_and_fetch and dfs_query_and_fetch options from the search_type parameter.
• New parameter include_repository in snapshot.get: (boolean) Whether to include the repository name in the snapshot info. Defaults to true.

X-Pack

X-Pack is being deprecated. The first time using xpack on the client, a warning will be triggered. Please check this issue for more information.

• New endpoints: index_lifecycle_management.migrate_to_data_tiers, machine_learning.reset_job, security.saml_authenticate, security.saml_complete_logout, security.saml_invalidate, security.saml_logout, security.saml_prepare_authentication, security.saml_service_provider_metadata, sql.delete_async, sql.get_async, sql.get_async_status, terms_enum.
• New experimental endpoints: machine_learning.infer_trained_model_deployment, machine_learning.start_trained_model_deployment, machine_learning.stop_trained_model_deployment.
• Deprecation: indices.freeze and indices.unfreeze: Frozen indices are deprecated because they provide no benefit given improvements in heap memory utilization. They will be removed in a future release.

This is a pre-release, but there are some important changes coming in version 7.14.0 of the client:

Client

This release changes the way in which the transport layer and the client interact. Previously, when using elasticsearch-transport, Elasticsearch::Transport::Client had a convenient wrapper, so it could be used as Elasticsearch::Client. Now, we are decoupling the transport layer from the Elasticsearch client. If you're using the elasticsearch gem, not much will change. It will instantiate a new Elasticsearch::Transport::Client when you instantiate Elasticsearch::Client and the endpoints from elasticsearch-api will be available.

Elasticsearch::Client has an attr_accessor for the transport instance:

> client = Elasticsearch::Client.new
> client.transport.class
=> Elasticsearch::Transport::Client
> client.transport.transport.class
=> Elasticsearch::Transport::Transport::HTTP::Faraday

The interaction with elasticsearch-api remains unchanged. You can use the API endpoints just like before:

> client.info
=> {"name"=>"instance",
 "cluster_name"=>"elasticsearch",
 "cluster_uuid"=>"id",
 "version"=>
  {"number"=>"7.14.0",
  ...
},
 "tagline"=>"You Know, for Search"}

Or perform request directly from the client which will return an Elasticsearch::Transport::Response object:

> client.perform_request('GET', '/')
# This is the same as doing client.transport.perform_request('GET', '/')
=> #<Elasticsearch::Transport::Transport::Response:0x000055c80bf94bc8
 @body=
  {"name"=>"instance",
   "cluster_name"=>"elasticsearch",
   "cluster_uuid"=>"id",
   "version"=>
    {"number"=>"7.14.0-SNAPSHOT",
    ...
    },
   "tagline"=>"You Know, for Search"},
 @headers=
  {"content-type"=>"application/json; charset=UTF-8",
   "content-length"=>"571",
   ...
   },
 @status=200>

If you have any problems, please report them in this issue.

• API Support for Elasticsearch version 7.13.3