Elasticsearch

Elasticsearch is a distributed full-text search and analytics engine built on the Apache Lucene library and exposed through a simple JSON-based API. Its job is to search enormous datasets — text, logs, metrics — in a fraction of a second, taking relevance, typos and word variants into account. Together with Logstash and Kibana it forms the popular analytics toolkit known as the ELK stack.

How Elasticsearch works

At its heart lies an inverted index: instead of scanning documents to find a word, Elasticsearch maintains a map that, for each term, points to every document containing it. This makes searching across millions of records almost instantaneous. The key concepts are:

• Document — a single record in JSON format, the basic unit of data.
• Index — a logical collection of similarly structured documents, the equivalent of a table.
• Shard — an index is split into shards spread across many nodes, enabling horizontal scaling.
• Replica — a copy of a shard that boosts availability and read performance.

Elasticsearch runs as a cluster of many nodes that automatically distribute data and take over each other's work if one fails, making it resilient and easy to scale. It is a NoSQL store, holding data without a rigid relational schema.

Elasticsearch in practice

The most common use cases are product search in e-commerce stores, searching content across portals and — above all — centralised collection and analysis of server logs and application metrics in big data environments. DevOps teams rely on it to monitor and quickly diagnose issues in distributed systems, visualising the gathered data on Kibana dashboards.