# Toward Automated Ticket Escalation in Customer Support Processes: From Emails to Explainable Predictions

- Type: Conference paper
- URL: https://www.gyunam.com/publication/2026-bpm-ticketescalation/
- Date: 2026-09-27
- Authors: Furkan Kocak, Gyunam Park
- Venue: Business Process Management
- DOI: https://doi.org/10.1007/978-3-032-37777-7_27
- PDF: https://www.gyunam.com/publication/2026-bpm-ticketescalation/paper.pdf
- Research lines: [Prescriptive process analytics and digital twins](https://www.gyunam.com/research-line/prescriptive-process-analytics/index.md)

## Abstract

Business processes in customer support often remain undocumented, with critical decisions such as ticket escalation relying on tacit knowledge rather than explicit process models. Automating such process decisions is challenging because the underlying data is unstructured email text, not structured event logs assumed by existing process mining and predictive monitoring approaches. We present a novel pipeline artifact that addresses this gap: a semi-automated four-stage pipeline that extracts structured features from customer support emails using large language models (LLMs) together with lightweight domain-expert validation, and employs explainable machine learning to predict the optimal timing of ticket creation, a key decision point in the support process. Our pipeline generates an initial feature schema from sample threads, consolidates semantically similar features through embedding-based clustering under domain-expert review, extracts cumulative feature values at each email point, and constructs a structured dataset ready for machine learning. We introduce a SHAP-based active feature learning strategy that identifies which missing features, when acquired, would most improve prediction confidence, enabling principled decisions about when to request additional information versus making immediate predictions. We evaluate our approach in two stages: first, on a reproducible synthetic dataset where comparison against six baselines demonstrates that LLM-extracted features provide complementary semantic information, with combined features achieving the best overall performance (ROC-AUC 0.90) while end-to-end LLM classification performs worst (ROC-AUC 0.46), validating our design of using LLMs as feature extractors rather than decision makers; second, on 1,500 email instances from 800 real-world email threads from Nosco, a commercial CRM platform, where we demonstrate effective feature extraction, predictive performance, and active feature learning for practical deployment.

## Practical problem

Customer support teams, especially in small and medium-sized businesses, decide when an email conversation should become a formal support ticket, and that judgment lives in the heads of experienced agents rather than in documented rules. The raw material is unstructured email text, so predictive and prescriptive monitoring techniques for business processes, which expect event logs with case identifiers, activities and timestamps, cannot be applied directly. Earlier ticket-escalation predictors needed analysts to hand-engineer features from interviews, an effort such businesses cannot afford, while letting a large language model make the decision itself raises trust, auditability and privacy concerns. Support organizations need a way to turn email threads into structured, explainable predictions and to know when more information should be requested before deciding.

## Research questions

1. How can unstructured customer support emails be transformed, with bounded domain-expert effort, into a structured dataset for predicting when a ticket should be created?
2. Do features extracted by large language models improve escalation-timing prediction compared with lexical, structural, rule-based and end-to-end language-model baselines?
3. Can SHAP-based reasoning about missing features tell the system when to withhold a prediction and request additional information instead?

## Contributions

- A semi-automated four-stage pipeline (feature identification, semantic clustering and consolidation, feature extraction, dataset construction) that converts email threads into cumulative per-email feature vectors using large language models (LLMs).
- An embedding-based consolidation step (mxbai-embed-large-v1 embeddings, HDBSCAN density-based clustering) in which domain experts pick representative features, merge correlated ones and bound categorical value sets.
- An explainable gradient-boosted classifier with TreeSHAP (SHapley Additive exPlanations) attributions that predicts ticket-creation timing at each email point, including attributions for missing features.
- A SHAP-based active feature learning algorithm that withholds a prediction when missing features carry high impact or make the prediction unstable, and names the feature to request.
- An evaluation on a public synthetic dataset of 350 threads and on 1,500 email instances from 800 threads of the Nosco customer relationship management platform, with code released on Zenodo.

## Key findings

- On the synthetic dataset, combining LLM-extracted, metadata and term-frequency features with CATBoost reaches an area under the receiver operating characteristic curve (ROC-AUC) of 0.90 ± 0.05 and F1 0.85 ± 0.02, adding 0.13 AUC over metadata-only features.
- End-to-end LLM classification performs worst (ROC-AUC 0.46 zero-shot, 0.48 few-shot), supporting the design of using LLMs as feature extractors rather than decision makers.
- On Nosco data, consolidation reduces 785 LLM-generated features to 35 (95.5% reduction) in about four hours of expert effort; CATBoost achieves ROC-AUC 0.73 ± 0.04 and F1 0.67 ± 0.04, the best of six classifiers.
- Active feature learning on 150 test instances raises precision from 0.67 to 0.74 and ROC-AUC from 0.73 to 0.75 while predicting on 50.7% of instances; recall drops from 0.67 to 0.61.
- Limitations: the real-world set contains only threads that ended in a ticket, extraction varies across models and prompts, threads are treated independently, and the Nosco data cannot be released.