Event Log Sampling for Predictive Monitoring

Abstract

Predictive process monitoring is a subfield of process mining that aims to estimate case or event features for running process instances. Such predictions are of significant interest to the process stakeholders. However, state-of-the-art methods for predictive monitoring require the training of complex machine learning models, which is often inefficient. This paper proposes an instance selection procedure that allows sampling training process instances for prediction models. We show that our sampling method allows for a significant increase of training speed for next activity prediction methods while maintaining reliable levels of prediction accuracy.

Publication
International Workshop on Leveraging Machine Learning in Process Mining (ML4PM)

Practical problem

Analysts who forecast the next step of running cases train deep neural networks on event logs, and training can take hours to weeks. They must repeat it many times to tune parameters and again whenever the process changes, so the long training times limit how often models can be refreshed and on what hardware. Generic data-reduction techniques from machine learning pick a subset of training examples, but they assume examples are independent, whereas events in a log belong to cases and are strongly correlated. A way to shrink an event log for training that respects its case structure is missing.

Research questions

  1. Can training predictive monitoring models on a sampled event log speed up training while keeping accuracy close to that of the full log?
  2. Is there a relationship between event log characteristics and the suitable sampling strategy and size?

Contributions

  • An event log sampling procedure for predictive monitoring that traverses variants, computes attribute distributions per variant, sorts cases within each variant and returns a subset of cases.
  • Three variant-aware selection rules: unique (one case per variant), logarithmic (log base k of the variant frequency) and division (frequency divided by k), the last guaranteed to preserve all variants.
  • Formalizes variant-preserving sampling and the evaluation measures reduction rate, relative accuracy and training and feature-extraction speed-up.
  • An implementation of the sampling methods in ProM, with feature extraction and prediction code on GitHub.
  • An evaluation protocol on two real logs (Road Traffic Fine Management and BPI Challenge 2012 W) with LSTM and XGBoost next-activity predictors, 5-fold cross-validation repeated five times.

Key findings

  • Division sampling preserves accuracy: on the Road log, d10 shrinks the log 9.75 times and speeds up LSTM training 9.0 times and XGBoost training 84.1 times while relative accuracy stays at 0.990 and 1.000; d3 even raises LSTM accuracy by 0.4 percent.
  • Aggressive sampling trades accuracy for speed: on Road, log10 gives a 572.29-fold reduction and 355.0 times faster XGBoost training but relative accuracy drops to 0.772 (XGBoost) and 0.767 (LSTM); unique sampling falls to 0.582 and 0.631.
  • Feature-extraction time shrinks more than proportionally to log size: the 572-fold log10 reduction on Road yields a 74,912-fold feature-extraction speed-up.
  • The choice of distribution matters more than k and depends on the log: BPIC-2012-W, with many unique variants, barely shrinks under division (d2 reduction 1.22) but tolerates unique sampling (LSTM relative accuracy 0.996).
  • Limitation: very high reduction makes accuracy unacceptable, so sample size should be adjusted gradually in hyperparameter optimization, and the evidence rests on only two logs.