DevOps Adventures…Log parser - Genesis
Lately I begin to wonder, what kind of tools we have to aggregate and collect and view logs...out of most prominent we have Zabbix, Greylog, or ELK (ElasticSearch, Logstash, Kibana).
On the first hand we have Zabbix, which is available via Open Source license, have great capabilities, but requires some knowledge and know-how, and if custom needs will arise, it might become more troublesome to configure. For instance, log aggregation is fairly simple tasks, however if we want to make custom responses or alerts based on gathered logs then it might be far more difficult to set up properly. In most such cases, we have our knowledge and friendly internet forums in hand.
On the other hand we have Greylog, which is created strictly for log aggregation, so its capabilities are smaller when compared to Zabbix. I don’t use it currently, so users of both of the above can say which solution works better for it and how to best use it.
Finally, we have combination of tools – ELK – which seems to be most popular solution for log aggregation, filtering, monitoring and alerting. It is worth to mention, that it can be used not only for log monitoring, but also for system monitoring. This combo comes with plenty plugins and useful features which overall helps a lot in tailoring for custom needs. It supports Kubernetes as well, which popularity grew significantly in last several years, due to its flexibility. However, to use ELK with k8s we need to deploy Filebeat to kubernetes cluster directly, which will aggregate logs and push them to Logstash. It seems that idea behind creators is good, and in fact it really is – Filebeat collects logs, then send them to Logstash, then Logstash sends logs to ElasticSearch and finally they can be viewed in Kibana, where we can monitor, view, or even send alerts based on ruleset. In short – we can do whatever we want, so our monitoring will be as best as it can be. Of course, all features are not available in Open Source version, so for alerting we will have to pay extra for the license. When it comes to technology, it works fairly well – Filebeat is written in Go, Logstash and ElasticSearch in Java, which is not so great. Unfortunately, system requirements for ElasticSearch itself are quite big – 8GB Ram and 4 CPU to run stable is bare minimum. We can set up ElasticSearch in cluster mode, which will slightly decrease chance of unexpected crash, but that will costs us more. One piece of advice, I strongly suggest to start Logstash on different node than ElasticSearch, due to possibility of running out of Ram, unless you have large servers which can carry the system load, which might become futile, but on this subject I’ll write later.
Great as it looks, this stack has some disadvantages, which in DevOps world might become hard to accept. For instance, if we want to aggregate different kind of logs, then we might find ourselves in the “rabbit hole” of Logstash filter configuration (no ordinary logs, like quasi Jsons or PlainText with special characters or both at the same time), price of use is fairly high (VM costs or additional license if we want to expand), or amount of resources to get it started. It has some key advantages like: documentation is prepared fairly good for typical solutions, thats a major plus, which shortens time of final configurations (again, for typical use case). Another advantage is encryption between all tools used in this stack – Filebeat – Logstash – ElasticSearch – Kibana. All of it seems that entire product seems very attractive, but not for all. Small companies, which do not require such big product could use something “smaller”, but the main problem is that there is no such things as similar alternatives which is quite as functional as ELK.
Hence I’ve started thinking about some actual alternative, which on first sight might seems unnecessary. Maybe I could write a parser, with dedicated frontend, where we could view aggregated logs ? Why not !
