# Performance-preserving event log sampling for predictive monitoring

- Type: Journal article
- URL: https://www.gyunam.com/publication/2023-sampling-jiis/
- Date: 2023-03-06
- Authors: Mohammadreza Fani Sani, Mozhgan Vazifehdoostirani, Gyunam Park, Marco Pegoraro, Sebastiaan J van Zelst, Wil MP van der Aalst
- Venue: Journal of Intelligent Information Systems
- DOI: https://doi.org/10.1007/s10844-022-00775-9
- PDF: https://www.gyunam.com/publication/2023-sampling-jiis/paper.pdf
- Research lines: [Prescriptive process analytics and digital twins](https://www.gyunam.com/research-line/prescriptive-process-analytics/index.md)

## Abstract

Predictive process monitoring is a subfield of process mining that aims to estimate case or event features for running process instances. Such predictions are of significant interest to the process stakeholders. However, most of the state-of-the-art methods for predictive monitoring require the training of complex machine learning models, which is often inefficient. Moreover, most of these methods require a hyper-parameter optimization that requires several repetitions of the training process which is not feasible in many real-life applications. In this paper, we propose an instance selection procedure that allows sampling training process instances for prediction models. We show that our instance selection procedure allows for a significant increase of training speed for next activity and remaining time prediction methods while maintaining reliable levels of prediction accuracy.

## Practical problem

Analysts who want to predict the next activity or the remaining time of running cases increasingly rely on machine learning models such as LSTM (Long Short-Term Memory) networks and XGBoost (eXtreme Gradient Boosting). Training these models on a full event log is slow, and getting acceptable accuracy requires repeated training runs for hyperparameter tuning, which is not feasible when models must be refreshed frequently to follow a changing process or when hardware is limited. Generic instance selection techniques from machine learning could shrink the training data, but they assume independent instances, whereas the prefixes extracted from one case are strongly correlated. Practitioners therefore lack a way to cut training time without silently degrading prediction quality.

## Research questions

1. Can prediction models trained on a sampled training log reach similar accuracy to models trained on the whole log while training substantially faster?
2. Which sampling distribution (unique, logarithmic or division) and which log characteristics determine whether accuracy is preserved?
3. Does the sensitivity to sampling differ between next activity prediction and remaining time prediction?

## Contributions

- A four-step instance selection procedure for event logs: traverse variants, compute attribute distributions per variant, sort the traces of each variant, and return a sample of the training log only.
- Three selection strategies, unique, logarithmic (log base k) and division (ceiling of frequency over k), with a formal notion of variant-preserving sampling.
- Relative evaluation metrics for size reduction, accuracy, F1-score, MAE (mean absolute error), RMSE (root mean squared error), feature extraction time and training time against the full-log baseline.
- An implementation on both Java and Python platforms, with code and feature generation published at `github.com/gyunamister/pm-prediction`.
- An evaluation protocol on three real-life logs (BPIC2012-W, Road Traffic Fine Management, Sepsis) with LSTM and XGBoost, 5-fold cross-validation repeated 5 times on one CPU thread.

## Key findings

- Logarithmic sampling with base 10 gives the largest reduction; the Road Traffic Fine Management training log shrinks by more than 1000 times, while division sampling barely reduces Sepsis because most of its variants are unique.
- For next activity prediction, division sampling keeps accuracy and F1-score close to the full-log model (relative accuracy near 1) for both LSTM and XGBoost; division by 2 on Sepsis with LSTM even raises accuracy.
- Unique and logarithmic sampling distort the variant frequency distribution of Road Traffic Fine Management, which has a few highly frequent variants, and lower accuracy and F1-score; base-10 logarithmic sampling there counts as oversampling.
- Training-time speed-up grows with the size reduction and is mostly larger for XGBoost than for LSTM; the choice of distribution matters more than the value of k.
- Remaining time prediction is more sensitive to sampling: XGBoost preserves MAE and RMSE only without oversampling, and the LSTM remaining time model is inaccurate even on the full log (high bias), so sampling cannot be judged from it.