{"id":3360,"date":"2024-03-26T23:19:15","date_gmt":"2024-03-26T15:19:15","guid":{"rendered":"https:\/\/chow.karmeng.my\/?p=3360"},"modified":"2024-03-26T23:19:15","modified_gmt":"2024-03-26T15:19:15","slug":"elasticsearch-8-12-kibana-cluster-using-vagrant-and-docker-compose","status":"publish","type":"post","link":"https:\/\/chow.karmeng.my\/?p=3360","title":{"rendered":"ElasticSearch 8.12 kibana cluster using vagrant and docker compose"},"content":{"rendered":"\n

Pre-requisite:<\/strong>
– VirtualBox 7.0.14
– Vagrant 2.4.1
– Windows 10 or better OS
– 16GB RAM (10GB RAM are required for creating ElasticSearch with kibana; 1GB and 2 x ElasticSearch node; 4GB each, rest of the RAM for VM host OS)<\/p>\n\n\n\n

Overview:<\/strong>
There are 2 layers of virtualization, first the Virtual Box, then the docker engine running in the Virtual Box VM running on Ubuntu 20.04 focal.<\/p>\n\n\n\n

Orchestration used in the host OS level; Windows 10 are the hashicorp vagrant<\/a>. The vagrant is used to configure the VM Ubuntu OS to be configured to run properly configured docker and Ubuntu 20.04.<\/p>\n\n\n\n

Then docker compose v2 are used to create the ElasticSearch 8.12 cluster or stack.<\/p>\n\n\n\n

The downside of this example, vagrant up needs to be run initially to configure the VM Ubuntu 20.04 OS. I have yet to discover if Vagrant has the ability to bootstrap grub and configuring the sysctl to allow the docker engine to run properly with the ElasticSearch 8.12 stack.<\/p>\n\n\n\n\n\n\n\n

Here are the bitbucket repository<\/a> that contains the codes that is mentioned in this post.<\/p>\n\n\n\n

Overview of this repo is as follows.<\/p>\n\n\n\n

\"\"
File structure of the ElasticSearch 8.12 running on docker in Ubuntu 20.04 VM<\/figcaption><\/figure>\n\n\n\n

Steps:<\/strong>
1. Run the vagrant up in the project localdev<\/p>\n\n\n\n

\"\"
Vagrant up to start creating the VBox VM that will be running ElasticSearch 8.12 in docker container<\/figcaption><\/figure>\n\n\n\n

2. Wait for the vagrant up to complete<\/p>\n\n\n\n

\"\"
Vagrant up completed, the first time running will force provision shell in Vagrantfile to run and configure the VM host Ubuntu OS to works fine with<\/figcaption><\/figure>\n\n\n\n

3. Confirm the provision script complete by running “vagrant status”<\/p>\n\n\n\n

\"\"
Confirm VM are powered off using “vagrant status”<\/figcaption><\/figure>\n\n\n\n

4. If the VM is powered off, rerun “vagrant up”, then run “vagrant ssh” to login into the VirtualBox VM.<\/p>\n\n\n\n

\"\"
Windows command prompt indicating that the vagrant ssh are successfully logon into VM shell<\/figcaption><\/figure>\n\n\n\n

5. Start up the ElasticSearch 8.12 with 2 ElasticSearch nodes in docker container, cd into \/data\/docker, then run “sudo docker compose -f docker-compose-duo.yml up -d”<\/p>\n\n\n\n

\"\"
Starting up ElasticSearch 8.12 in the VirtualBox VM<\/figcaption><\/figure>\n\n\n\n
\"\"
If everything is fine, and the images repo from docker hub will be downloaded for the first time, extracted before running as container in the VM.<\/figcaption><\/figure>\n\n\n\n
\"\"
the docker compose for ElasticSearch 8.12 completed without issues<\/figcaption><\/figure>\n\n\n\n

6. Confirm the status of the containers, run in the terminal “sudo docker ps”<\/p>\n\n\n\n

\"\"
Proof of containers of ElasticSearch and Kibana are running<\/figcaption><\/figure>\n\n\n\n

This however does not proof that the stacks are completed correctly.<\/p>\n\n\n\n

Use the “docker logs” to check all the containers.<\/p>\n\n\n\n

\"\"
Check last 5 line of logs from container ElasticSearch node, es01<\/figcaption><\/figure>\n\n\n\n
\"\"
Check last 5 line of logs from container ElasticSearch node, es02<\/figcaption><\/figure>\n\n\n\n
\"\"
Check last 13 line of logs from container Kibana<\/figcaption><\/figure>\n\n\n\n
\"\"
The official kibana template provides a setup container that are used to create certificate and assigning passwords into nodes. <\/figcaption><\/figure>\n\n\n\n

Other than the checking all the ElasticSearch containers, and kibana, the setup container will be stop running and provide the logs as seen to indicate the configuration of the ElasticSearch is completed.<\/p>\n\n\n\n

Finally, to put all into a complete conformation, use browser to visit the kibana and ElasticSearch interface.<\/p>\n\n\n\n

\"\"
Kibana is running on http:\/\/127.0.0.1:5601<\/figcaption><\/figure>\n\n\n\n
\"\"
Successful login, first time into Kibana, now just call it as Elastic<\/figcaption><\/figure>\n\n\n\n
\"\"
The problem of self-signed certificate, just click continue<\/figcaption><\/figure>\n\n\n\n
\"\"
Use the default elastic username and password to logon into the ElasticSearch node at https:\/\/127.0.0.1:9200<\/figcaption><\/figure>\n\n\n\n
\"\"
After signing on successfully into https:\/\/127.0.0.1:9200<\/figcaption><\/figure>\n\n\n\n
\"\"
Proof that kibana (now known as Elastic) and ElasticSearch cluster are interconnected. <\/figcaption><\/figure>\n\n\n\n

To truly ensure that the Elastic (kibana) are working well, go into the Dev Tools -> Console. Type “GET _cluster\/health”, run the console command, on the right panel of the console, you should receive similar result.<\/p>\n\n\n\n

Will add more post on details how to make the example from ElasticSearch 8.13 works. At the time of the post, the example provided by ElasticSearch does not work as expected.<\/p>\n\n\n\n

The steps that I was doing to get this ElasticSearch running is akin to Hal trying to fix the light bulb.<\/p>\n\n\n\n

\n