Typed API

Update APIs to latest elasticsearch-specification 8.13

Fixes

This patch release brings a fix to the initialisation of the Request in endpoints which would prevent using the shortcuts for fields.
Canonical.Request() method was unaffected.

• Autoscaling.PutAutoscalingPolicy
• Indices.Downsample
• Indices.PutSettings
• Indices.SimulateTemplate
• Inference.PutModel
• Logstash.PutPipeline
• Ml.ValidateDetector
• SearchApplication.Put

API

New APIS:

• ConnectorSecretGet

• ConnectorSecretPost

• ConnectorSecretPut

• ConnectorSecretDelete

• ConnectorUpdateIndexName

• ConnectorUpdateNative

• ConnectorUpdateStatus

• ConnectorUpdateAPIKeyDocumentID

• ConnectorUpdateServiceDocumentType

• EsqlAsyncQuery Documentation

• EsqlAsyncQueryGet Documentation

• ProfilingFlamegraph Documentation

• ProfilingStacktraces Documentation

• TextStructureTestGrokPattern Documentation

• Indices.ResolveCluster Documentation

• Security.QueryUser Documentation

Typed API

• indices.ResolveCluster Documentation
• textstructure.TestGrokPattern Documentation

Thanks to @pakio, transport now has an optional pool based compression option. https://github.com/elastic/elastic-transport-go/pull/19
And to @tblyler for fixing a very subtle memory leak in the BulkIndexer. #797

• Fix: ticker memory leak in bulk indexer due to internal flush call resetting the ticker. #797
• Fix: Scroll now uses the body to pass the scroll_id. #785
• Add: generated UnmarshalJSON for Requests to allow injecting payloads using aliases.

Many thanks to @tblyler, @frkntplglu and @HaraldNordgren for their contribution!

Client

Golang version

The client now requires Golang version 1.20

OpenTelemetry

The client now provides OpenTelemetry integration. This integration can be enabled in the config using the elasticsearch.NewOpenTelemetryInstrumentation.
Once set up, the provided context will be used to record spans with useful information about the request being made to the server.

More about what you can expect in the Semantic Conventions for Elasticsearch.

BulkIndexer

if_seq_no & if_primary_term are now supported thanks to @benjyiw #783

API

• SimulateIngest
• ConnectorCheckIn
• ConnectorDelete
• ConnectorGet
• ConnectorLastSync
• ConnectorList
• ConnectorPost
• ConnectorPut
• ConnectorSyncJobCancel
• ConnectorSyncJobCheckIn
• ConnectorSyncJobDelete
• ConnectorSyncJobError
• ConnectorSyncJobGet
• ConnectorSyncJobList
• ConnectorSyncJobPost
• ConnectorSyncJobUpdateStats
• ConnectorUpdateConfiguration
• ConnectorUpdateError
• ConnectorUpdateFiltering
• ConnectorUpdateName
• ConnectorUpdatePipeline
• ConnectorUpdateScheduling

Typed API

• Esql.Query documentation
• Fleet.PostSecret
• Inference documentation
  • DeleteModel documentation
  • GetModel documentation
  • Inference documentation
  • PutModel documentation
• SearchApplication
  • GetSettings documentation
  • UpdateSettings documentation

Typed API

• Fix https://github.com/elastic/go-elasticsearch/issues/756 preventing from settings indices in indices.PutSettings

API

Experimental APIs

• EsqlQuery documentation
• InferenceDeleteModel documentation
• InferenceGetModel documentation
• InferenceInference documentation
• InferencePutModel documentation

Typed API

• Mandatory URL parameters are not exposed as functions anymore as they already exist in the constructor.

New Compatibility Policy

Starting from version 8.12.0, this library follow the Go language policy. Each major Go release is supported until there are two newer major releases. For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release.

If you have any questions or concerns, please do not hesitate to reach out to us.

Typed API

Update APIs to latest elasticsearch-specification 8.10

API

Experimental APIs for internal use

• FleetDeleteSecret
• FleetGetSecret
• FleetPostSecret

Exprimental APIs

QueryRulesetList

Stable APIs

Security.GetSettings
Security.UpdateSettings

Typed API

Exprimental APIs

QueryRuleset.List

Technical Preview

• QueryRuleSet

Beta

• Synonyms

API

New API

• Cluster.Info Documentation

Experimental APIs

• QueryRulesetGet Documentation
• QueryRulesetDelete Documentation
• QueryRulesetPut Documentation
• SearchApplicationRenderQuery Documentation
• Security.CreateCrossClusterAPIKey Documentation
• Security.UpdateCrossClusterAPIKey Documentation

Typed API

• Propagated request fields towards the endpoint for ease of access, taking priority over same-name query string fields.
• Added a stub for Do methods on endpoints that only support a boolean response such as core.exists.
• NDJSON endpoints support with custom serialization like core.bulk.
• Link to endpoints documentation in API index to better display and ease of use.

fixes

• Fixed a deserialization issue for Property & Analyzer #696

Typed API

• Fixed deserialization for Suggest in search responses.
• Fixed double-quoted strings in deserialization for unions normalized as string. #684
• Fixed handling of core.Get response when the index did not exist. #678

API

New APIs

• Watcher.GetSettings Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/watcher-api-get-settings.html
• Watcher.UpdateSettings Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/watcher-api-update-settings.html

Experimental APIs

• ML.DeleteDataLifecycle Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/dlm-delete-lifecycle.html

• ML.ExplainDataLifecycle Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/dlm-explain-lifecycle.html

• ML.GetDataLifecycle Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/dlm-get-lifecycle.html

• ML.PutDataLifecycle Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/dlm-put-lifecycle.html

• SearchApplications https://www.elastic.co/guide/en/elasticsearch/reference/8.8/search-application-apis.html

Transport

• Fixed an issue with subdirectory in url during retry. (#657)
• Fixed a deadlock in transport (#603 & #640)

Typed API

• This release include fixes for responses deserialization. #654 #655

API

• ML.DeleteJob: Added WithDeleteUserAnnotations. Should annotations added by the user be deleted.
• ML.ResetJob: Added WithDeleteUserAnnotations. Should annotations added by the user be deleted.
• ML.StartTrainedModelDeployment: Added WithPriority. The deployment priority.
• TransformGetTransformStats: Added WithTimeout. Controls the time to wait for the stats.
• TransformStartTransform: Added WithFrom. Restricts the set of transformed entities to those changed after this time.

New APIs

TransformScheduleNowTransform documentation.
HealthReport documentation.

Typed API

• Inclusion of responses structures.

Changes

• Do method on endpoints now return a typed response, one per endpoint.
• Perform method added on endpoints, returns http.Response as did Do.
• Elasticsearch exceptions are now handled as types.ElasticsearchError with .As and .Is methods.
• .Raw now takes a reader as input.
• User defined values such as _source in Hits are now json.RawMessage to highlight they later deserializable nature.

AdditionalProperties, like the ones found in multi-bucket aggregations, are not yet supported.

API

• ML.StartTrainedModelDeployment: Added WithPriority

New APIs

• ML.UpdateTrainedModelDeployment: Updates certain properties of trained model deployment.

Client

BulkIndexer

Improvements were made to the BulkIndexer memory usage to allow better handling under burst use cases. Thanks to @christos68k and @rockdaboot !

API

• ML.StartTrainedModelDeployment: Description of NumberOfAllocations has been changed in "The total number of allocations this model is assigned across machine learning nodes".
• Security.GetAPIKey: Added WithLimitedBy boolean parameter. Flag to show the limited-by role descriptors of API Keys.
• Security.GetUser: Added WithProfileUID boolean parameter. Flag to retrieve profile uid (if exists) associated to the user.
• Security.GetUserProfile: Changed the description of uid parameter, a comma-separated list of unique identifier for user profiles.
• Security.QueryAPIKeys: Added WithLimitedBy boolean parameter. Flag to show the limited-by role descriptors of API Keys.
• TextStructureFindStructure: Added EcsCompatibility string parameter. Optional parameter to specify the compatibility mode with ECS Grok patterns - may be either 'v1' or 'disabled'.

Promoted to stable

• ML.InferTrainedModel
• ML.PutTrainedModelDefinitionPart
• ML.PutTrainedModelVocabulary
• ML.StartTrainedModelDeployment
• ML.StopTrainedModelDeployment
• Security.activateUserProfile
• Security.DisableUserProfile
• Security.EnableUserProfile
• Security.GetUserProfile
• Security.HasPrivilegesUserProfile
• Security.SuggestUserProfiles
• Security.UpdateUserProfileData

New APIs

• ML.ClearTrainedModelDeploymentCache documentation.
• Security.BulkUpdateAPIKeys documentation.
• Indices.Downsample (Experimental API) documentation

Typed API

Following multiple feedbacks we decided to remove the builder API for the type tree.

In its place, work has started to further simplify the type tree by removing redundant type aliases. The API also now comes with a helper package named some that allows to call for inline pointers on primitive types.

In addition, a bug was fixed preventing the use of wildcards in index names, and enums are now extensible by default.

The Typed API remains in alpha stage while its development continues.

Client

• Added the option to override the Content-Type to enable usage of alternative serialization. (#542)
• API is compatible with Elasticsearch 7.17.7

API

• get, mget and search added force_synthetic_source: Should this request force synthetic _source? Use this to test if the mapping supports synthetic _source and to get a sense of the worst case performance. Fetches with this enabled will be slower the enabling synthetic source natively in the index.
• ML.StartTrainedModelDeployment added cache_size: A byte-size value for configuring the inference cache size. For example, 20mb.
• Snapshot.Get added sort, size, order, from_sort_value, after, offset and slm_policy_filter. More on these in the documentation.

New API

• Security.UpdateAPIKey documentation.

Typed API

As highlighted in the release not for the 8.4.0-alpha.1, this release marks the beginning of the newly arrived TypedClient.

This new API is still in alpha stage and will be release alongside the existing esapi.

A few examples of standard use-cases can be found in the TypedAPI section of the documentation.

This second prerelease of the 8.4.0 updates the API for the client and fixes the serialization for types using additional properties.

This prerelease introduces a new typed API generated from the elasticsearch-specification. This generation from the common specification allows us to provide a complete API which uses an exhaustive hierarchy of types reflecting the possibilities given by Elasticsearch.

This new API is the next iteration of the Go client for Elasticsearch, it now lives alongside the existing API, it is in alpha state and will gain features over time and releases.

What's new

The TypedClient is built around a fluent builder for easier request creation and a collection of structures and helpers that mimics as closely as possible the Elasticsearch JSON API.

As a first example, here is a search request:

cfg := elasticsearch.Config{
	// Define your configuration
}
es, _ := elasticsearch.NewTypedClient(cfg)
res, err := es.Search().
    Index("index_name").
    Request(&search.Request{
        Query: &types.QueryContainer{
            Match: map[types.Field]types.MatchQuery{
                "name": {Query: "Foo"},
            },
        },
    },
    ).Do(context.Background())

The Request uses the structures found in the typedapi/types package which will lead you along the possibilities. A builder for each structure that allows easier access and declaration is also provided.

More on the specifics and a few examples of standard use-cases can be found in the TypedAPI section of the documentation.

Limitations

While most of the endpoints are covered, a few points are still being worked on and will be part of future releases:

• NDJSON endpoints: bulk, msearch, msearch_template, ML.post_data, find_structure, to name a few.
• Response and Errors structures with deserialization.

Transport & config

While being different, the new API uses all the existing layers that were built so far, elastic-transport-go remains the preferred transport and all your configuration and credentials applies, same as before.

Feedback

Feedback is very welcome, play with it, use it, let us know what you think!