Instan(t)a-neous Monitoring AND ClickHouse Database applied

We’ll take you on a modern environment journey by explaining how to apply continuous monitoring and column-oriented DBMS in your daily work, what’s in it for you and what (business) benefits are accomplished. You can use this knowledge instantly.

Agenda:

17.30 Opening with drinks and food

18.00 Instan(t)a-neous Monitoring

19.00 Break 19.15 ClickHouse Database

20.30 Closing with drinks

20.45 Semi-Final Champions League on a big screen

Why real-time Continuous Monitoring?

The Problems Facing the Container and Microservices Crowd: When everything is working as expected and users are not complaining, life is good. As soon as you run into your first major customer impacting incident the typical response is to gather up just about anyone you can via Slack, HipChat, conference call, etc. and ask, “What do we have for monitoring?” This scene plays itself out across the entire globe daily so don’t feel too bad if you resemble the description.

Monitoring these modern environments is challenging and here are 5 major reasons why:

1. Rate of Deployment –Modern APM MUST not require manual configuration of monitoring or alerting.

2. Rate of Change –Modern APM MUST automatically adapt to dynamic environments and must report data within a few seconds in high granularity (1 second).

3. Number of Technologies – Modern APM must be capable of quickly supporting, and then automatically monitoring new technologies shortly after release and treating those technologies as first-class entities.

4. Size of Environment – Modern APM MUST be able to ingest and process the data from 10’s and 100’s of thousands of components in real-time.

5. The Complexity of Orchestration - Modern APM MUST understand the orchestration tool and let you know exactly what is malfunctioning at any given time. It must also accurately depict the end result of the orchestration engine deploying containers into the runtime. This is critical for ensuring that orchestration is properly configured and not running away with out of control container growth (a very expensive byproduct of improper configuration).

The solution: Instan(t)a-neous Monintoring

Continuous Integration, Continuous Delivery, Continuous Monitoring! These days CI and CD are commonly used mechanics to achieve fast turn-around times for high-demand applications. Microservices architectures and highly dynamic environments (based on Kubernetes, Docker, …), however, come with a whole different set of problems. Systems, that not only appear and disappear dynamically (e.g. autoscaling), but most commonly tend to be written using multiple different programming languages, are hard to monitor from the point of view that matters: User Requests and User Experience. but the answer is simple; Continuous Monitoring (CM). Let's build a polyglot microservices infrastructure. A way to monitor and trace multi-service requests will be demonstrated using Instana’s automatic discovery system.

Why Columns oriented DBMS?

Businesses and environments are increasing fast. So, you need to analyse terabytes of data, with analytical queries that span thousands of rows. There’s no one fits all DBMS solution for modern environments. Column-oriented databases can provide a 100x speedup. A column-oriented DBMS is a database management system (DBMS) that stores data tables by column rather than by row. Practical use of a column store versus a row store differs little in the relational DBMS world. Both columnar and row databases can use traditional database query languages like SQL to load data and perform queries. Both row and columnar databases can become the backbone in a system to serve data for common extract, transform, load (ETL) and data visualization tools. However, by storing data in columns rather than rows, the database can more precisely access the data it needs to answer a query rather than scanning and discarding unwanted data in rows. Query performance is increased for certain workloads.

Lessons learned using ClickHouse Database:

ClickHouse; Or how Instana is using a column-oriented DBMS for sub-second analytics. Instana monitors and captures every request in their customers environments. But just capturing all those terabytes of information is worthless without the ability to use it. Therefore at Instana ClickHouse is selected, a column-oriented DBMS, for near real-time analytics to give our customers the best possible insights. We’ll present the benefits of Clickhouse and how we solved the challenges and limitations ClickHouse has.

Speakers:

Christoph Engelbert is a passionate Java geek with a deep commitment for Open Source software. He is mostly interested in Performance Optimizations and understanding the internals of the JVM and the Garbage Collector. He loves to bring software to its limits by looking into profilers and finding problems inside of the codebase. In addition, he is highly interested in new ideas, technologies and new ways of solving problems. He has a deep understanding of IP based technologies like Protocol Stacks, TCP, UDP and asynchronous service implementations and fast serialization solutions. Normally he doesn't like to reinvent the wheel but if there is a reason and a chance to make it faster or eas…

Miel Donkers is a senior software craftsman. His focus is mostly on back-end (Java) development but also touches the front-end or some infrastructure on a regular basis. His focus is always on delivering customer value, from reliably delivering new features to providing assistance on support tickets.