TechPrufer
Sign inGet started

Blogs›IBM›Log Template Mining for Network Outage Detection

Log Template Mining for Network Outage Detection

Log Template Mining for Network Outage Detection

1
posts
2020

This post introduces the Drain3 log-template mining project, an open-source tool for extracting structured information from unstructured log files. It details the refactoring of the original Drain tool into Python 3, enhancements for streaming support and resiliency (using Kafka for state persistence), and the addition of masking for improved accuracy. The post explains how log templates can be used to create time-series data for anomaly detection, enabling early identification of network outages in large-scale data centers like IBM Cloud. Future posts will cover the analytics pipeline for correlating anomalies.

2020

Use open source Drain3 log-template mining project to monitor for network outages

5/12/2020

Introduced Drain3, an open-source log-template mining project, as a solution for monitoring network outages in IBM Cloud data centers. Detailed the refactoring of Drain to Python 3, added streaming support with a new `add_log_message()` function, implemented resilient state persistence using Kafka, and incorporated masking prior to de-templating. Explained how log templates can be transformed into time-series data for anomaly detection to identify incidents.