Start United States dc3 United States software in english pdf How to Debug an Unresponsive Elasticsearch Cluster

How to Debug an Unresponsive Elasticsearch Cluster

401
0
TEILEN

While highly scalable, Elastisearch is complex to set up. Read on for a cheat sheet for common integration issues, what they mean, and how to solve them.
Join the DZone community and get the full member experience.
Elasticsearch is an open-source search engine and analytics store used by a variety of applications from search in e-commerce stores, to internal log management tools using the ELK stack (short for “Elasticsearch, Logstash, Kibana”). As a distributed database, your data is partitioned into “shards” which are then allocated to one or more servers.
Because of this sharding, a read or write request to an Elasticsearch cluster requires coordinating between multiple nodes as there is no “global view” of your data on a single server. While this makes Elasticsearch highly scalable, it also makes it much more complex to setup and tune than other popular databases like MongoDB or PostgresSQL, which can run on a single server.
When reliability issues come up, firefighting can be stressful if your Elasticsearch setup is buggy or unstable. Your incident could be impacting customers which could negatively impact revenue and your business reputation. Fast remediation steps are important, yet spending a large amount of time researching solutions online during an incident or outage is not a luxury most engineers have. This guide is intended to be a cheat sheet for common issues that engineers running that can cause issues with Elasticsearch and what to look for.
As a general purpose tool, Elasticsearch has thousands of different configurations which enables it to fit a variety of different workloads. Even if published online, a data model or configuration that worked for one company may not be appropriate for yours. There is no magic bullet getting Elasticsearch to scale, and requires diligent performance testing and trial/error.
Cluster stability issues are some of the hardest to debug, especially if nothing changes with your data volume or code base.
Download the cluster state using the below command and look at the size of the JSON returned.
To fix, flatten your data into something that is Elasticsearch-friendly:
It’s ok for certain long running tasks like moving an index to take a long time. However, normal query and index operations should be quick.
Run the command and ensure none or few tasks in-flight.
If only a small set of nodes have high heap usage, you may need to better balance your customer. For example, if your shards vary in size drastically or have different query/index bandwidths, you may have allocated too many hot shards to the same set of nodes. To move a shard, use the reroute API. Just adjust the shard awareness sensitivity to ensure it doesn’t get moved back.
Look at an index or all indices fielddata size like so:
Enable Trace logging to review discovery related activities.
See reason behind why allocation has stopped.
Get a list of red indices to understand which indices are contributing to red state. The cluster state will be in the red state as long as at least one index is red.
For more detail on a single index, you can see recovery status for the offending index.
If you see a timeout from max_retries (maybe the cluster was busy during allocation), you can temporarily increase the circuit breaker threshold (Default is 5). Once the number is above the circuit breaker, Elasticsearch will start to initialize the unassigned shards.
Review logs for nodes for a message like below:
Once the flood stage reached, you’ll see logs like so:
To confirm, see which indices have read_only_allow_delete set to true.
To remove this block of read-only, make the command:
Troubleshooting stability and performance issues can be challenging. The best way to find the root cause is by using the scientific method of hypothesis and proving it correct or incorrect. Using these tools and the Elasticsearch management API, you can gain a lot of insights into how Elasticsearch is performing and where issues may be.
Published at DZone with permission of Derric Gilling.
See the original article here.

Opinions expressed by DZone contributors are their own.