Replace Data Volumes of Elasticsearch Cluster with Encrypted Volumes
How To Remove Node From Elasticsearch Cluster Without Affecting Cluster Or Without Downtime
Encrypt Data Volumes Of Elasticsearch Node Without Any Downtime
Replace Existing Node Of Elasticsearch Cluster Without Any Downtime
You may encounter a situation when you need to remove/replace a node from an Elasticsearch cluster. let’s consider a use-case where we need to encrypt all the data volumes (Volumes containing Elasticsearch data) needs to be encrypted.
Following process will be followed to achieve the same:
- Select a Node, say N2, take AMI of the root volume.
- Launch an instance from the AMI taken, with the number of volumes to be used to store data. Check the option for “encrypted“.
- After launching it, Format and mount the volumes at the required mount-point.
- Change the node.name and network.host parameters in /etc/elasticsearch.yml file
- Since we have taken the image of root volume, the elasticsearch configuration should already be having the master node IP configuration. But now, since the IP has changed we simply need to update the node.name and network.host in elasticsearch.yml file.
- Start the Elasticsearch process. It will automatically join the cluster and the cluster will start relocating the data to this node.
- Now we have N+1 nodes in the cluster, we can decommission the existing node N2 on which we have unencrypted volume using a command.
- After decommissioning the instance, wait for all the data to be relocated from N2 to other nodes.
- When no data is left on the N2, stop the Elasticsearch process on it.
Let’s go deeper and see how to do this:
Considerations:
Consider a cluster having two instances N1 and N2.
Master node: N1
Data Node: N1, N2
Volumes attached to N1: V1N1 with mount-point /vol/es1 and V2N1 with mount-point /vol/es2
Volumes attached to N2: V1N2 with mount-point /vol/es1 and V2N2 with mount-point /vol/es2
All volumes are unencrypted.
Take AMI of root volume of N2 node, with the option “No Reboot” to make sure that current instance should not go down.
Launch a new instance from the AMI by adding two additional drive of required size and the check the encryption check-box.
Now, we have the new instance with two volumes which is going to serve the purpose of data volumes for Elasticsearch. These volumes are not yet mounted, so go ahead and mount the volumes.
Check the volume first with the command
fdisk -l
The available volumes may be /dev/xvdb and /dev/xvdc.
Now format the volumes with the required file system
mkfs.xfs /dev/xvdb
mkfs.xfs /dev/xvdc
Now mount the volumes:
mount /dev/xvdb /vol/es1
mount /dev/xvdc /vol/es2
As we have taken the AMI of the existing node, the mount point must be present with the required permission, if not, than do create them and give required permission.
Edit the /etc/elasticsearch/elasticsearch.yml file and change the values of following option
node.name give_unique_name_of_node
network.host private_ip_address_of_node
We do not need to configure Master Node IP address since we have take the AMI of existing instance, its already configured. If have created a new instance than do configure it.
Now, we are done with the required configuration. Simply start the elasticsearch process and observe the logs.
Logs after adding the new elasticsearch instance.
[2016-09-20 13:10:04,718][INFO ][cluster.service ] [X.X.X.X] detected_master {ip-X-X-
X-X.ec2.internal}{UfGNZVZ0SayYjZW50NO_FA}{X.X.X.X}{X.X.X.X:9300}{data=1, master=1},
added {{ip-X-X-X-X.ec2.internal}{UTZRgnw4TfiswzBTV2gEXw}{X.X.X.X}{X.X.X.X:9300}
{data=1, master=0},{ip-X-X-X-X.ec2.internal}{UfGNZVZ0SayYjZW50NO_FA}{X.X.X.X}
{X.X.X.X:9300}{data=1, master=1},}, reason: zen-disco-receive(from master [{ip-X-X-X-
X.ec2.internal}{UfGNZVZ0SayYjZW50NO_FA}{X.X.X.X}{X.X.X.X:9300}{data=1, master=1}])
Here X.X.X.X are the IPs. The important thing here to note is the line detected_master and added {ip-X-X-X-X.ec2.internal} which signify that our new instance has detected the master node and it has been added to the cluster.
Run the command to check if data relocation (Transfer of data to newly added node) has started or not.
curl P.P.P.P:9200/_cat/shards?v
where P.P.P.P is the private IP of the node. Output must be as follows:
collection_name 1 p RELOCATING 1392274 1.2gb X.X.X.X ip-X-X-X-X.ec2.internal -> Y.Y.Y.Y Mvv7ZxnCQqCapOhALzOAgA Y.Y.Y.Y
You may observe several lines like this which signifies that the data relocation is going on.
Data would get distributed evenly on the nodes. After this we will deregister the N2 node from the cluster. To deregister, run the following command on master node N1.
curl -XPUT M.M.M.M:9200/_cluster/settings -d '{"transient" :
{"cluster.routing.allocation.exclude._ip" :"P.P.P.P"}}';echo{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":
{"allocation":{"exclude":{"_ip":"P.P.P.P"}}}}}}After deregistering the N1, the data should start moving from N1 to other nodes which can be seen by running the following command:The output should be as follows:shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
0 0b 64.4mb 199.8gb 199.9gb 0 P1.P1.P1.P1 P1.P1.P1.P1 ip-p1-p2-p3-p4.ec2.internal
41 50.1gb 51.2gb 148.6gb 199.9gb 25 P2.P2.P2.P2 P2.P.2.P2.P2 P2.P.2.P2.P2
41 47.4gb 47.9gb 151.9gb 199.9gb 23 P3.P3.P3.P3 P3.P3.P3.P3 ip-P3-P3-P3-P3.ec2.internal
When the shards and disk.indices on the node N2 gets 0 that means all data is being transferred to other nodes and the N2 is safe to remove from the cluster.
Now Just stop the Elasticsearch process and you may terminate the instance N2.
Replacing the Master Node
We are done replacing the data-node. Now we will replace the Master node. In our case the master is also a data node which have two unencrypted volume. So we will be following the same process:
- Create a new node from the AMI of root volume of the node N1 and adding two volumes with encrypted option checked.
- Format and mount newly added volumes.
- Change the node.name, network.host paramters in /etc/elasticsearch/elasticsearch.yml file.
- Make sure to have node.master set to 1 in /etc/elasticsearch/elasticsearch.yml file.
This setting will make this node master enabled. so if we remove the current master node, this node will automatically become the master.
When this instance is ready, just start the elasticsearch process and it will automatically join the cluster and data relocation will start automatically which can be seen by the commands mentioned above for data node.
To check the current master node and the new master enabled node, run the following command:http://M.M.M.M:9200/_cat/nodes
This will show all the nodes in the cluster. An "*" sign in front of a node shows that its a master node, "m" signifies a master enabled node which we created just now and "-" shows a data node.P1.P1.P1.P1 P1.P1.P1.P1 41 99 0.02 d - P1.P1.P1.P1
P2.P2.P2.P2 P2.P2.P2.P2 25 99 0.30 d * ip-P2-P2-P2-P2.ec2.internal
P3.P3.P3.P3 P3.P3.P3.P3 0 19 0.34 d m P3.P3.P3.P3
We can now deregister the N1 node with the command which we previously used to deregister the data-node. Just put the IP of N1 to deregister it. Also, after deregistering it, check if the acknowledgement is true in the output and data relocation has started or not.
Once, when the data relocation is done and no shards is left on N1, stop elastsicsearch process and now we can terminate/stop the N1 instance.
After stopping elasticsearch process, the master enabled node will automatically become the master and there would be only two nodes left in the cluster.