DevOps Adventures…Log parser - Idea

DevOps Adventures…Log parser – Idea

For starters, let’s take a look at the idea. How such parser would work ? In which language it should be written ? What functionalities it should have ? How the architecture suppose to look like ? What tools should be used ? These questions came to my mind at the very beginning, when I first started to think about it seriously. After brief analysis I came to conclusion, that basic schema of it should like like this :

“Service” → Logs → BE → Database ← BE ← FE

Some explenation to the above – same BE will download/parse logs and provide them to the FE, so they will be accessible. When above basics become clear, time has come to choose tools.

“Service” – means any kind of service available in kubernetes eco system.

Logs – stream of information what is actually happening in the service. In this place we have to differentiate between all kinds of logs format, from plain text, json to custom made. Not all languages gives any kind of logs format, then it will have to be done via BE.

BE – the core of entire system, it downloads logs and parse them, then insert into DB and provide access to them after they are inserted into DB.

Database – place where logs will be stored

FE – Interface where logs will be easily accessible.

Writing own log parser will require several skills from different “disciplines”:

1. Development – writing BE and FE will require programming skills

2. Database Administration – will be required to create proper data structure

3. DevOps – network architecture understanding will be required to access kubernetes API in proper and safe way.

4. QA – will be required for testing.

5. UX/UI – will be required for nice design, so logs will be accessible without hassle.

When it comes to actual tools, I’ve chosen the below:

1. Development – Rust – it works really fast and use minimum system requirements. For FE Rust also will be used (vertigo)

2. Database – PostgreSQL – provide good performance and it’s supported by Rust Diesel library.

3. DevOps – Beside kubernetes cluster and services deployed within, we will need to set up proper safety access, so logs will be available to view only for BE. GitLab CI is left future development at the moment.

4. QA – Postman and cURL – will be needed for advances BE checks.

5. UI/UX – Improvements will be applied in the meantime, as design is a second priority right now.

As I wrote at the beginning, this is just the idea, because behind every aspect of it, there are plenty little tasks which needs to be thought through and implemented. It was helpful as much, as it helps avoid problems in the future. Not every tasks can be constructed as I first thought it would, and some things are better to be corrected in the beginning phase, so we can avoid problems in the future.