Story of ELK (Elasticsearch, Logstash, and Kibana)

I would like to thank my readers for their time in visiting this blog. Its been a good month where traffic is high and new followers are joining in. So welcome and I hope you enjoy the blog.

In the past few days I read about ELK. It’s not a new thing I have read about it but I never tried the three of them all together. Before, I only consider elasticsearch as the star of the show because of its search ccapabilities. But in the past few days I really learned the all three and found that they could be useful together.

So I will describe each of them separately first. Starting with elasticsearch. I have mentioned this thing in my several posts. It is Lucene-based search platform. It has many similarities with its twin brother, Apache Solr. Personally, I like elasticsearch better than Solr because of its simplicity in setting up the cluster. Apache Solr requires Zookeeper while Elasticsearch doesn’t. But other than that I don’t see significant differences in term of search functionality.

Many of the company uses elasticsearch. One example that for it seesm cool is One Million Tweet Map. It really shows how good elasticsearch is in handling indexing and geolocation. It also it shows my country, Indonesia, as one of the loudest country in Twitter 🙂

So we know elasticsearch is scalable, easy to setup and can retain huge amount of data. Having these advantages, why don’t we just use elasticsearch as databases. There are several reasons you can google them. But for me there is one reason in particular. Elasticsearch is a search platform. Thus, it doesn’t have a good authentications and authorization functions as good as databases for storage. Another reason would be the write speed to save data is not as good as other databases like Cassandra. Some guys from elasticsearch may argue this but this is what I found out. But other than those two, it is a great search platform.

Now its Logstash’s turn. Logstash is actually kind of ingestion engine. Maybe I also put it on m previous blog about big data ingestion engine. But I never heard Logstash being used other than with Elasticsearch. So Logstash is similar to Apache Flume when it comes to mechanism in data ingestion. Logstash reads data from file or databases with JDBC support, do some process and then push the data to Elasticsearch.

What makes Logstash different than other big data ingestion engine is its capability to do some small process before pushing the data to elasticsearch. By process I mean is data enrichment. We can add some additional data before we push to elasticsearch. For example if the data contains IP address, Logstach can enrich the data by adding the GeoIP data such as country and city based on the IP address. In other ingestion engine like Kafka or Flume, the data enrichment needs to be carriedout after the message/data pass through. For example by Apache Storm or Spark Streaming.

Now the last of three musketeer is Kibana. Kibana basically is a visualization tool. It is similar to Tableau but it is specially tailored for elasticsearch data. In fact, I never read Kibana used by other than Elasticsearch, same as Logstash. PLease do let me know when you found one example.

With Kibana we can choose various charts to visualize our data. Those basic charts like bar charts, pie can be used easily to visualize the data. Map is also in Kibana’s arsenal in displaying the data.

All of those three are available on Elasticsearch website. You can download and read some docs over there.

I found that many companies are using ELK for vaious purposes. For example The Guardian uses Elasticsearch to gauge their articles performance. For exxample an article is intended for British readers then they can see on Kibana how many British read views the article. And the good thing is they can see this in realtime. Another company use it for security monitoring purpose.

For me, what in mind ofr ELK is it can be the best Minimum VIable Product (MVP) platform for data-based application. Because it is easiness of setup and can deliver results. It give the fast picture of an idea. Later some data analytics can be applied for example using Apache Spark, etc. If you have some opinioins or experiences in using ELLK, feel free to share in the comments.

Hope this helps 🙂