Configuring Large Reasoning Models Using Process Mining: A Benchmark and a Case Study

Alessandro Berti, Humam Kourani, Gyunam Park, Wil M. P. Van Der Aalst
April, 2026
PDF Cite DOI

Abstract

Large Reasoning Models (LRMs), a subset of Large Language Models (LLMs) trained to articulate their chain-of-thought, have shown promise in tackling complex scientific tasks. However, evaluating and configuring their reasoning processes remains underexplored. This paper leverages a process mining-specific LLM evaluation framework to propose a methodology for analyzing and configuring LRMs. We introduce an approach to extract and classify reasoning steps by type (e.g., Deductive Reasoning, or Hypothesis Generation) and effect (Positive, Indifferent, Negative) on the overall reasoning, enabling a detailed assessment of reasoning quality. From this, we derive a new benchmark, PMLRM-Bench, which evaluates not only the correctness of outputs but also the robustness of the reasoning process. A case study on the QwQ-32B LLM demonstrates how targeted adjustments to reasoning type frequencies can boost task-specific performance. Our results reveal distinct reasoning patterns across models and provide actionable insights for LRM configuration. This work bridges process mining and LLM evaluation, offering a scalable framework for reasoning analysis.

Type
Conference paper
Publication
Business Process Management Workshops

Practical problem

Analysts increasingly ask large language models to read event logs, spot bottlenecks, check conformance or draft process models, and a newer class of these models, large reasoning models, writes out its chain of thought before answering. Existing benchmarks only grade the final answer, so a model that reaches the right conclusion through flawed or speculative steps scores the same as one that reasons soundly, and nobody can tell which reasoning habits a model should do more or less of. Teams who want to tune such a model for their domain therefore have no systematic way to diagnose its reasoning or to measure whether a configuration change helped.

Research questions

  1. How can the reasoning traces of large reasoning models be extracted and classified by reasoning type and by their effect on correctness so that reasoning quality can be measured, not just answer accuracy?
  2. Do targeted adjustments to the frequency of specific reasoning types, applied through system prompts, change a model’s performance on process mining tasks?

Contributions

  • A reasoning analysis pipeline that parses chain-of-thought traces into steps and classifies each by one of ten reasoning types and by positive, indifferent or negative effect, using a judge LLM.
  • A composite score S that rewards correct conclusions and positive-effect steps while penalizing partially correct and wrong conclusions and indifferent or negative steps.
  • PMLRM-Bench, an extension of PM-LLM-Benchmark v2.0 covering 17 large reasoning models, released on GitHub (fit-alessandro-berti/pmllmbench-lrms-reasoning-analysis).
  • A case study protocol that varies the frequency of hypothesis generation, ethical reasoning and validation/backtracking in QwQ-32B via system prompts and re-evaluates the variants.
  • A validation of the judge’s classifications with a second judge LLM.

Key findings

  • Composite scores range from 14,459 (Grok-3-thinking) to -7,724 (R1-Distill-Qwen-1.5B), and they correlate strongly with PM-LLM-Benchmark scores (Pearson’s r = 0.89), so robust reasoning tracks task performance.
  • Across models, deductive reasoning (25 to 40% of steps) and hypothesis generation dominate; weaker models produce far more negative-effect steps (438 for the 1.5B model versus 10 for the top model) and lean on speculative reasoning without validation.
  • A second judge (ChatGPT-4o) fully agrees with the Gemini-2.5-Pro classifications in 82.40% of steps, partially in 13.01%, and disagrees in 4.59%.
  • For QwQ-32B, prompting for more hypothesis generation raises the PM-LLM-Benchmark score from 36.9 to 37.1, while suppressing it lowers the score to 36.3 and the Hypotheses Generation category from 4.9 to 4.3.
  • Adjusting ethical reasoning does not help: less of it drops Fairness from 5.7 to 5.1, and more of it also lowers Fairness to 5.3 despite the highest composite score (14,218); results depend on the judge LLM’s classification accuracy.