The purpose of this repo is to:
provision_elasticsearch
is set to false
)cd infra
terraform init
project_id
, k8s cluster_name
, k8s nodes machine_type
and the cluster region
in terraform.tfvars
file.
project_id
, region
and machine_type
defaults can be used. cluster_name
has to be unique.terraform.tfvars
file. There are two options available:
In case user wants a new elasticsearch cluster to be provisioned using ECK then provision_elasticsearch
must be set to true
.
In that case variables es_password
and es_host
can be left empty. es_user
should keep the default value and imageTag
should be set to the version required.
In case user already has an elasticsearch cluster deployed and reachable by gcp (elastic cloud) then provision_elasticsearch = false
must be set as well as the right values to variables
es_host, es_user, es_password, imageTag
gke_num_nodes
and gke_max_num_nodes
inside terraform.tfvars
file.gke_num_nodes
* (number of zones in region)).gke_max_num_nodes
enables cluster autoscaling in case there is a need for more resources.deployBeat
, deployAgent
.terraform apply
kubectl
by running gcloud container clusters get-credentials <cluster-name> --zone europe-west1 --project elastic-obs-integrations-dev
kubectl get node
, kubectl get pod -A
project_id = "elastic-obs-integrations-dev"
region = "europe-west1"
cluster_name = "test-k8s-cluster-simple"
machine_type = "e2-standard-4"
gke_num_nodes = 1
gke_max_num_nodes = 1
provision_elasticsearch = false
es_password = ""
es_user = "elastic"
es_host = ""
deployBeat = false
deployAgent = false
imageTag = ""
namespace = "kube-system"
project_id = "elastic-obs-integrations-dev"
region = "europe-west1"
cluster_name = "test-k8s-cluster-autoscaling-beats"
machine_type = "e2-standard-4"
gke_num_nodes = 1
gke_max_num_nodes = 6
provision_elasticsearch = false
es_password = "mypassword"
es_user = "elastic"
es_host = "https://bxxxxxed.europe-west1.gcp.cloud.es.io:9243"
deployBeat = true
deployAgent = false
imageTag = "8.3.0"
namespace = "kube-system"
project_id = "elastic-obs-integrations-dev"
region = "europe-west1"
cluster_name = "test-k8s-cluster-autoscaling-elasticsearch-agent"
machine_type = "e2-standard-4"
gke_num_nodes = 1
gke_max_num_nodes = 6
provision_elasticsearch = true
es_password = ""
es_user = "elastic"
es_host = ""
deployBeat = false
deployAgent = true
imageTag = "8.3.0"
namespace = "kube-system"
NOTE. The command may end with an error like this but everything should be successfully deployed:
Error: Kubernetes cluster unreachable: Get "https://35.239.222.162/version?timeout=32s": dial tcp 35.239.222.162:443: connect: connection refused
cd scripts
go build
./stress_test_k8s --kubeconfig=/Users/<username>/.kube/config --deployments=20 --namespaces=10 --podlabels=4 --podannotations=4
The above command will create 10 namespaces and deploy one demo nginx deployment in each one with
as many 20 replicas as indicated in the deployments
flag. Each pod will have 4 labels and 4 annotations.
By default, no logs are being produced. If you want your pods to create logs run stress_test tool with --logs argument:
./stress_test_k8s --kubeconfig=/Users/andreasgkizas/.kube/config --deployments=1 --namespaces=2 --podlabels=2 --podannotations=2 --logs --periodoflogs 2
--Periodoflogs (in sec): Default value is 1 sec
####Prerequisite: Existence of 2 Elasticsearch Clusters. One with metricbeat index TSDB enabled and one without.
In order to get a quick estimation of the status of the 2 Elasticsearch indices(one simple and one TSDB enabled) one can execute scripts/es_bench
. By now the script can only be executed manually. More specifically the script provides the following information about the cluster:
pri.store.size
docs.count
_cat/indices?v=true&s=index
API.Execution example: TSDB_ES_URL="https://35.157.42.42:9200/" TSDB_ES_PASS="passpasstsdb" SIMPLE_ES_URL="https://104.199.42.42:9200/" SIMPLE_ES_PASS="passpasssimple" TSDB_INDEX=".ds-metricbeat-tsdb-8.3.0-2022.05.24-000001" SIMPLE_INDEX=".ds-metricbeat-8.3.0-2022.05.24-000001" go run main.go Example output:
Executing against new ES cluster
Client: 8.2.0 Server: 8.3.0-SNAPSHOT
index name: .ds-metricbeat-tsdb-8.3.0-2022.05.24-000001
pri.store.size: 5.8gb
docs.count: 25635493
median query time is: 2ms
Executing against new ES cluster
Client: 8.2.0 Server: 8.3.0-SNAPSHOT
index name: .ds-metricbeat-8.3.0-2022.05.24-000001
pri.store.size: 23gb
docs.count: 39051417
median query time is: 333ms