Screening Fast and Slow

Introduction

Process-level scoring presumes that intermediate reasoning steps can be objectively verified as correct or incorrect (Lightman et al., 2023). While deterministic domains like formal logic, code execution, and mathematics support objective verification, open-ended professional workflows governed by tacit knowledge resist strict binary classification, requiring that intermediate step labels be assigned by proxy. The dominant proxy, conformity to a single recorded trajectory, is theoretically sound only if the recorded move was the singular defensible action at each node. Wherever a node branches into multiple valid moves, this assumption degrades. Consequently, exact-match evaluation risks penalizing defensible divergence, measuring conformity to a specific expert rather than underlying analytical quality.

To bypass the bottleneck of single-path trajectory matching, the field has increasingly relied on automated judges to evaluate reasoning at scale. However, this introduces a secondary vulnerability: algorithmic verbosity bias. In automated evaluation, verbosity bias is the well-documented tendency of LLM-as-a-judge pipelines to conflate response length and structural exhaustiveness with analytical quality (Zheng et al., 2023; Wang et al., 2023; Dubois et al., 2024). Consequently, these judges structurally reward exhaustive generation—a reasoning approach where a model surfaces and verifies peripheral variables regardless of their operational time cost, rather than filtering for high-yield signals. In bounded professional environments, expert performance relies on sequential triage and precise stopping rules (Gigerenzer & Todd, 1999; Klein, 1998). Therefore, an automated judge risks systematically penalizing human efficiency, misinterpreting an expert’s tactical restraint as a lack of thoroughness.

With these constraints in mind, this study introduces a multi-rater instrument for step-level evaluation, with two objectives:

1. Investigate temporal prioritization. By enforcing a strict operational budget, we evaluate whether frontier LLMs can selectively prioritize information, rather than defaulting to unconstrained search.
2. Model multi-path acceptability. Rather than assuming a single correct trajectory, we test whether multiple valid paths exist by measuring how often a blinded panel of senior practitioners approves a model's divergent proposals.

The reasoning traces collected in this study, and the multi-rater instrument built upon them, offer direct methodological applications for both AI development and deployment:

• Frontier capability researchAssessing human grader preference on frontier model generations under constraints provides pairs for fine-tuning, a proxy of a given specialized domain’s “open-endedness”, and calibration sets for LLM-as-judge qualification.
• Enterprise copilot deploymentPresenting models with an analyst’s working context at each step of the task provides a measure of copilot utility: the degree to which proposed next actions are executable within real-world temporal constraints, and whether the assistance augments the workflow (suggesting directions the expert did not otherwise explore), scaffolds it (proposing courses of action they would have taken anyway), or disrupts it (through the introduction of inaccurate or less useful recommendations).

Background

To construct the evaluation instrument, we captured the multimodal reasoning traces of senior finance practitioners across three distinct workflows: investment analysis, credit agreement analysis, and thesis construction. For this study, we isolated the investment analysis task because it uniquely demanded deterministic, verifiable computation alongside practitioner judgement and heuristic-driven pacing.

The evaluation is anchored on a screening session by an investment professional, who was tasked with reviewing an investment target within a 30-minute time budget.

• EnvironmentMaterials are delivered as a virtual data room, including a confidential investment memorandum and financial pack. Participants were instructed to solve the task as they would in a real work setting, including the usage of apps such as Excel, Word, and the internet – only direct use of LLMs (e.g., ChatGPT, Claude, Gemini, and other reasoning tools) was forbidden.
• Data collectionParticipants were requested to reason aloud as they went through the problem (Ericsson & Simon, 1993; Fox et al., 2011), while computer-use and audio was recorded over Zoom.
• ConfidentialityPII was redacted during the data processing phase.
• Reasoning trace productionCorroborating stated reasoning and intent against observed in-context behaviour led to the development of high-fidelity reasoning traces for each of the task types. For the investment judgement task, we isolated n = 19 decision points: moments where the practitioner reasoned through and then selected among competing alternatives. The median gap between consecutive decision points was 98 seconds (range 20s to 376s).

Objectives

Our objective was to use the resulting reasoning trace to evaluate two dimensions of open-ended reasoning:

Experiment 1. Investigating Temporal Prioritization and Epistemic Triage.When given step-by-step context about a long-horizon reasoning task with n steps, how accurately do models recommend a course of action for step n+1?In bounded, high-context environments, reasoning quality is inextricably linked to pacing. However, unconstrained LLMs naturally bypass step-level prioritization, exhibiting scope inflation by proposing exhaustive, parallelized analytical agendas (Valmeekam et al., 2023). Because models do not experience the temporal friction of sequential execution, they often respond to single decision nodes by generating comprehensive checklists rather than a singular, prioritized next action, artificially inflating output length.

By enforcing strict operational bounds, we aim to measure two facets of strategic pacing. First, we evaluate whether frontier models can exercise epistemic triage—deliberately delaying deep computation to build global context, as human experts do—or whether they default to premature verification by abandoning the text to verify isolated figures. Second, we test whether these pacing deficits are simple missing-context errors (fixable via prompt-based temporal warnings) or ingrained behavioral defaults.

Experiment 2. Multi-Path Acceptability.To what extent do models produce valid divergences from the expert's recorded trajectory on long-horizon tasks?Single-reference evaluation pipelines treat divergence from a recorded expert trajectory as error. That penalty is sound only if the recorded move was the sole defensible action at that node. The space of defensible moves in a heuristic workflow is intractable to enumerate, so we do not attempt to map it; we test the narrower, falsifiable claim that the recorded move is unique among defensible options.

We formalize this as the Conformity False-Rejection Rate (CFRR): the share of divergent model moves, sampled under the bounded condition, that a blinded practitioner panel rates Acceptable or Strongest. Each such rating is a counterexample to uniqueness; the rate is the proportion of valid strategic deviations that single-path grading penalizes.

Exp 1: Investigating Temporal Prioritization and Epistemic Triage

To evaluate strategic executive function, we constrained four frontier models—opus-5, gpt-5, gemini-3.1-pro, and deepseek-v4—alongside a weak control (llama-3.1-8b) to the temporal budgets available to the human expert at each node. At each of the 19 decision nodes, models received the active screen state and a chronological transcript of all preceding actions.

To identify the impacts of various conditions on pacing, we evaluated recommended model actions at each step of the task across three distinct conditions:

• Condition 1 (Unbounded Baseline)Models were prompted to generate their next step with no length constraints. This condition serves as an empirical baseline to measure default verbosity bias and observe unconstrained information foraging.
• Condition 2 (Operational Bound)To prevent models from bypassing step-level prioritization via exhaustive generation, models were restricted to proposing only their immediate next analytical action (operationalized as a 300-character, 60-second slice of the overarching budget). This bound does not penalize models for their native computational speed; rather, it serves as a forcing function requiring the model to reveal its rank-ordered strategic priorities and demonstrating whether it sequences qualitative context-building before quantitative verification.
• Condition 3 (The Cost-Aware Warning)To determine whether the premature verification observed in Condition 2 stems from a simple lack of real-world context or a deeply ingrained behavioral default, models faced the exact bounds of Condition 2, plus an explicit temporal penalty warning. Models received an appended statement that the two documents are separate files and that moving between them to find and check a figure "takes several minutes of the remaining budget"—the budget whose elapsed and remaining time the prompt had just stated. This tests whether models recalculate their strategy when informed of the cost, or if the behavior is insensitive to instruction.

Enforcing a strict operational bound (Condition 2) compressed frontier model output length by approximately 50% compared to the unbounded baseline (Condition 1), filtering structural verbosity while isolating the strategic intent useful for problem-solving.

Discussion: The unbounded baseline (Condition 1) confirmed the expected scope inflation behaviour: without the friction of sequential execution, models responded to single decision nodes by generating comprehensive theoretical checklists instead of isolating a prioritized next action.

Enforcing the 60-second operational bound in Condition 2 required the model to drop secondary checklist items and reveal its rank-ordered priority. This structurally halved model output length (e.g., Opus-5 compressed from a median of 380 to 187 characters; GPT-5 from 322 to 169) while leaving the pooled strategy mix essentially unchanged (frontier document-switches: 32 of 76 unbounded, 30 of 76 bounded), though individual models shifted (e.g., GPT-5 dropped from 7 to 3 switches, while Gemini rose from 8 to 11, and the Llama control rose from 2 to 7).

This indicates that a significant portion of open-ended generation in professional workflows is structural verbosity, and that tight operational constraints can act as an effective semantic filter. This finding can be extended to the development of copilot tools that propose prioritized next-action recommendations during complex, open-ended tasks, enabling human-in-the-loop monitoring and alignment over long horizons.

Frontier models consistently bypassed qualitative text in favor of immediate quantitative calculation. Explicitly prompting models that these calculations "take several minutes of the remaining budget" (Condition 3) did not reduce the frequency of this behavior.

Discussion: While imposing Condition 2 revealed each model’s priorities, we wanted to understand why those priorities were disproportionately weighted towards quantitative verification over global context-building. The pilot data shows a clear behavioural divergence: the practitioner stayed on the primary narrative text, switching to verify an external figure just 1 time out of 19 within the limited time allocated. GPT-5 mirrored this tendency (switching 3 of 19 times), while Opus-5 and Gemini-3.1-Pro switched to external figures in 11 of 19 nodes each. This tendency extended to proposal content: the models that diverged on switching behaviour also diverged on the number of quantitative figures cited per action, with Opus-5 proposing a mean of 1.63 against the practitioner’s 0.05.

Condition 3 asked whether models verify early because they are unaware of the time cost, or because the behaviour is an ingrained default. Models were subjected to the exact bounds of Condition 2, but received the appended warning that switching documents "takes several minutes of the remaining budget."

The intervention did not alter the macro-strategy. The four-branch interpretation rule was registered before generation; the result fell in the pre-specified branch. Across 76 total frontier model encounters (19 nodes × 4 models), the baseline bounded models (Condition 2) switched documents 30 times. When explicitly warned about the time cost (Condition 3), switches rose slightly to 34. The warning changed which items models verified but left their overall rate of verification unchanged, suggesting the switching behaviour is an ingrained default rather than the result of missing information. For deployment, the implications are that prompting a model about the time cost of its actions does not reduce the interruption or refocus the scope of next-action recommendations. Mitigating such behaviour would likely require intervention at the model training level.

Information Foraging Theory (Pirolli & Card, 1999) offers one account of the divergence: an effective forager weighs the cost of leaving its current information patch against the expected yield of exploring another, and the practitioner, who pays that cost in real time, exploits the primary text before switching. Models never bear the temporal cost of the switches they propose, and the pilot suggests the resulting habit persists even when the cost is stated.

Exp 2: Multi-path acceptability

To evaluate the models' divergent paths, we will assemble a blinded panel of senior finance practitioners. Raters will be presented with the state context and a randomized, blinded list of proposed next steps—including both the human reference move and the models' generated alternatives.

Raters will evaluate each move using a forced-choice scale (Strongest, Acceptable, Weak), assessing the strategic viability of executing the proposed action in the subsequent 60 seconds. Any model-generated alternative that the panel labels Acceptable or Strongest will be logged as a valid divergence.

Aggregating these valid divergences will yield the CFRR. This metric will provide an asymmetric, one-directional evidence base for interpreting the evaluation:

• High CFRR (e.g., >35%)Will conclusively falsify the single-path premise, rendering conformity an invalid proxy for reasoning quality.
• Low CFRR (e.g., <15%)Will imply that exact-match grading is a practical, cheap approximation for the domain. However, a low CFRR will leave the underlying topology ambiguous: it cannot distinguish between a genuinely narrow state-space (where few valid alternatives exist) and incapable generators that simply lack the tacit knowledge to discover them.

See Exp 1 findings