Business processes in customer support often remain undocumented, with critical decisions such as ticket escalation relying on tacit knowledge rather than explicit process models. Automating such process decisions is challenging because the underlying data is unstructured email text, not structured event logs assumed by existing process mining and predictive monitoring approaches. We present a novel pipeline artifact that addresses this gap: a semi-automated four-stage pipeline that extracts structured features from customer support emails using large language models (LLMs) together with lightweight domain-expert validation, and employs explainable machine learning to predict the optimal timing of ticket creation, a key decision point in the support process. Our pipeline generates an initial feature schema from sample threads, consolidates semantically similar features through embedding-based clustering under domain-expert review, extracts cumulative feature values at each email point, and constructs a structured dataset ready for machine learning. We introduce a SHAP-based active feature learning strategy that identifies which missing features, when acquired, would most improve prediction confidence, enabling principled decisions about when to request additional information versus making immediate predictions. We evaluate our approach in two stages: first, on a reproducible synthetic dataset where comparison against six baselines demonstrates that LLM-extracted features provide complementary semantic information, with combined features achieving the best overall performance (ROC-AUC 0.90) while end-to-end LLM classification performs worst (ROC-AUC 0.46), validating our design of using LLMs as feature extractors rather than decision makers; second, on 1,500 email instances from 800 real-world email threads from Nosco, a commercial CRM platform, where we demonstrate effective feature extraction, predictive performance, and active feature learning for practical deployment.
Customer support teams, especially in small and medium-sized businesses, decide when an email conversation should become a formal support ticket, and that judgment lives in the heads of experienced agents rather than in documented rules. The raw material is unstructured email text, so predictive and prescriptive monitoring techniques for business processes, which expect event logs with case identifiers, activities and timestamps, cannot be applied directly. Earlier ticket-escalation predictors needed analysts to hand-engineer features from interviews, an effort such businesses cannot afford, while letting a large language model make the decision itself raises trust, auditability and privacy concerns. Support organizations need a way to turn email threads into structured, explainable predictions and to know when more information should be requested before deciding.