autonomous monitoring tools






In the following sections, we will dig a bit deeper into the first step, i.e extracting log templates from raw logs (yellow elements in the diagram above). We will first motivate this step, discuss state of the art tools and techniques, then present “the PacketAI way”.


Current observability practices don’t fully exploit logs





Log2template: A universal log parser


Introducing Log2template


High level design




Offline stage (training phase)


  • historical logs data set is split into training and test
  • a word embedding neural network model is trained on the training set then test set
  • test results are evaluated against a template extraction heuristic, amplified with expert feedback. Model weights are adjusted accordingly
  • when a desirable result quality is achieved, the weights representing the chosen model, are stored into a weights database
  • and corresponding resulted templates are stored into a log template database


Online stage (inference phase)


  • real-time stream of logs is ingested and the saved word embedding model is applied
  • the output embedding vectors are compared to saved templates, if there is a match then the log line is assigned to the corresponding template and the tuple is output
  • if an Out Of Vocabulary (OOV) vector is found (vector that does not correspond to any template), then further heuristic analysis is done for confirmation
  • if the OOV is confirmed, then the corresponding template is saved into the templates database





  • manual creation and maintainability of regex
  • lack of semantic representations
  • high computation cost of regex-based systems


Related Post

Subscribe to our newsletter