We were fortunate enough to get an opportunity to do an Elasticsearch cluster snapshot and restore on a production highly active cluster. The indices we needed to restore were around 2 – 3 TB in size.
Our task was to take a snapshot from an old cluster (v 6.4.2) which had several huge indices and restore a few of them to a new cluster (v7.9.2). This endeavour was supposed to bring the load down from the old cluster.
Elasticsearch is a scalable, full-text search and analytics engine based on Apache Lucene. It is Java-based and allows you to store, search, and analyze big volumes of data quickly in near real-time. It can search and index the document files in diverse formats. Lucene is the underlying technology that Elasticsearch uses for extremely fast data retrieval.
As flexible, scalable, and useful as Elasticsearch is, monitoring your cluster can help you ensure that the cluster is appropriately sized and handles all operations efficiently.
Have you noticed an unexpected unallocation of Shards happening at a duration of 1 hour resulting in Cluster state switching from Green > Yellow > Red > Yellow > Green?. During this transition, ES becomes unreachable and the API calls start responding with non 200 code.
Right off the bat, I want to say that, this blog does not cover installing and configuring ElastAlert in the usual sense, i.e. working with pre-existing rules. It helps, I hope, in understanding the requirements for adding one’s own rule. Continue reading “Make Your Own Rules, ElastAlert Style”