Binary classification benchmark for rule-based reasoning.
Source: LegalBench (Hearsay)
This task evaluates a model's ability to identify inadmissible hearsay under the Federal Rules of Evidence.
The baseline prompt asks the model to determine whether a specific piece of evidence meets the criteria for hearsay given a particular legal issue:
Does the provided evidence constitute hearsay in the context of the issue being litigated?
To ensure logical consistency and formal verification, the task is decomposed into entity identification and predicate extraction, governed by the following rule as defined in Sadowski & Chudziak (2025):
Where:
-
Terms (Entities):
-
$s$ : A communicative act (oral, written, or non-verbal assertive conduct). -
$I$ : The specific legal issue or disputed matter being litigated. -
$a$ : The specific factual claim or proposition conveyed by the statement.
-
-
Predicates:
-
$\text{IsStatement}(s)$ : The act$s$ is assertive conduct intended to communicate information. -
$\text{IsOutOfCourt}(s)$ : The statement was made outside of the current trial or hearing. -
$\text{HasAssertion}(s, a)$ : The statement$s$ contains a specific factual claim$a$ . -
$\text{IntroducedForLegalIssue}(s, I)$ : The statement is offered specifically to resolve the disputed matter$I$ . -
$\text{ProvesTruthOfAssertion}(s, I)$ : The statement's probative value depends on establishing the truth of the assertion.
-
The binary label reflects whether the deterministic chain of legal inference is satisfied.
- YES (Positive Class):
- Logic: The logical expression evaluates to
True. - Interpretation: The evidence is an out-of-court statement being used specifically to prove what the speaker originally asserted.
- Example: "To prove that the insured under a life policy is dead, his wife offers a death certificate."
- Logic: The logical expression evaluates to
- NO (Negative Class):
- Logic: The logical expression evaluates to
False. - Interpretation: The evidence fails at least one hearsay criteria, such as being a statement made in court, non-assertive conduct, or introduced for purposes other than proving its truth.
- Example: "On the issue of whether Will knew that the company intended to announce its drug trials had been cancelled, the fact that he told the jury that 'he didn't know the first thing about how medicines worked.'"
- Logic: The logical expression evaluates to
Source: huggingface.co (DDXPlus)
This task evaluates a model's ability to perform differential diagnosis using the principle of Diagnosis of Exclusion. The model must determine if a patient's presentation justifies a general diagnosis of Upper Respiratory Tract Infection (URTI) or if the clinical evidence fulfills the criteria for a more specific respiratory pathology.
The baseline prompt asks the model to evaluate symptoms against specific exclusion criteria:
Is URTI (Upper Respiratory Tract Infection) the correct diagnosis, or does a more specific respiratory condition apply?
To ensure clinical safety and explainability, the diagnosis is treated as a logical exclusion. URTI is valid if and only if no specific respiratory condition (
Where:
-
$S$ (Specific Conditions): {Pneumonia, Bronchitis, Influenza, Tuberculosis, Pharyngitis, Sinusitis, Acute Laryngitis, Bronchiectasis, Viral Pharyngitis, Rhinosinusitis}. -
Predicates:
-
$\text{HasCriteria}(p, c)$ : Patient$p$ exhibits hallmark indicators (e.g., high fever, purulent sputum, chronic duration) that define condition$c$ .
-
The binary label reflects whether the "Exclusion" rule holds based on the presence or absence of "red flag" indicators.
-
YES (Positive Class):
-
Logic: The logical expression evaluates to
True. - Interpretation: Symptoms are mild and non-specific (nasal congestion, mild sore throat). Crucially, the presentation lacks features of more severe or specific conditions.
- Example: "Patient presents with a runny nose and a mild cough for 2 days. No fever or chest pain."
-
Logic: The logical expression evaluates to
-
NO (Negative Class):
-
Logic: The logical expression evaluates to
False. - Interpretation: The patient presentation includes at least one specific indicator (Hard Negatives) that triggers a more precise diagnosis.
-
Example: "Patient has a runny nose but also reports a high fever (39°C) and severe muscle aches." (Indicator for Influenza
$\to$ NOT URTI).
-
Logic: The logical expression evaluates to
The benchmark utilizes a "Least Confusing" selection strategy from the DDXPlus dataset to challenge the model's boundary-case reasoning:
| Case Type | Selection Criteria |
|---|---|
| Positive | Ground truth is URTI. Differential includes at least one specific respiratory disease with |
| Negative | Ground truth is a specific respiratory disease. URTI was considered in the differential with |
Specific conditions included in
Source: huggingface.co (DDXPlus) — pathology label Allergic sinusitis.
This task evaluates a model's ability to distinguish allergic upper-airway inflammation from clinically confusable infectious conditions. Unlike the URTI task — which is governed by an exclusion rule — Allergic Rhinitis is governed by an inclusion + bounded exclusion rule. The contrast in logical shape is the point: it tests whether rule-based evaluation generalises across rule structures within the same data source.
Does the patient's presentation justify a diagnosis of Allergic Rhinitis, or does an infectious respiratory condition apply?
Where:
-
Predicates:
-
$\text{HasAtopicFeatures}(p)$ : Patient$p$ exhibits hallmark allergic indicators — sneezing, itchy nose / eyes / palate, watery rhinorrhoea, wheeze without fever, history of asthma / eczema / prior allergy. -
$\text{AllergenTrigger}(p, t)$ : Patient$p$ has identifiable exposure to allergen$t$ — pollen / seasonal, dust, animals, occupational, food. -
$\text{HasInfectiousFeatures}(p)$ : Patient$p$ shows signs of infection — fever, purulent sputum, productive cough with infection, sick contacts, severe systemic illness.
-
-
YES (Positive Class):
-
Logic: The logical expression evaluates to
True. - Interpretation: The patient has atopic features tied to an identifiable trigger, without infectious red flags.
- Example: "Patient reports recurrent sneezing fits and itchy eyes when around the family cat; no fever, no purulent discharge."
-
Logic: The logical expression evaluates to
-
NO (Negative Class):
-
Logic: The logical expression evaluates to
False. - Interpretation: The presentation includes infectious features (or lacks atopic features / a trigger) such that an infectious condition is the correct diagnosis.
-
Example: "Patient has nasal congestion and a productive cough with green sputum and a fever of 38.5°C." (Indicator for Bronchitis
$\to$ NOT Allergic Rhinitis).
-
Logic: The logical expression evaluates to
| Task | Rule shape | Reasoning pattern |
|---|---|---|
| URTI | Pure exclusion: rule out every specific condition | |
| Allergic Rhinitis | Inclusion (positive criteria + trigger) with bounded exclusion of one feature class |
The benchmark uses the same "Least Confusing" selection strategy as URTI:
| Case Type | Selection Criteria |
|---|---|
| Positive | Ground truth is Allergic sinusitis. Differential includes at least one confusable infection with |
| Negative | Ground truth is a confusable infection. Allergic sinusitis was considered in the differential with |
Confusable infections included: Bronchitis, Pneumonia, Influenza, Croup.
The set above is empirically derived from DDXPlus's differential structure, not from clinical priors. A clinician would expect the natural confusables for Allergic Rhinitis to be other upper-airway conditions — URTI, acute / chronic rhinosinusitis, viral pharyngitis. In DDXPlus, however, those conditions almost never carry Allergic sinusitis in their differentials: when ground truth is URTI or rhinosinusitis, the AR probability is
We follow the data, not the clinical intuition. This is intentional: rule-based evaluation should be testable against whichever confusion structure the data exposes, and the resulting rule (atopic features + trigger, absent infection) cleanly discriminates the actual decision boundary the dataset presents.
Viral pharyngitis was further excluded from the confusable set because its negative-pool size differs by three orders of magnitude between train (1,058) and test (1), which would create an unreliable train/test distribution.
Source: huggingface.co
This task evaluates a model's ability to determine whether a scientific abstract describes a specific computational Method being applied to a specific Task. It is derived from the SciERC dataset and formulated as a binary classification problem.
The baseline prompt asks the model to determine if a functional "used-for" relationship exists between identified entities:
Does the text describe a scientific method, model, or tool being explicitly used for, applied to, or designed for a specific task or application?
To ensure explainability, the reasoning process is decomposed into entity identification and predicate verification. The final determination is governed by the following logical rule:
Where:
-
Terms (Entities):
-
$m$ (Method): Algorithms, models, tools, or system components (e.g., "LSTM", "kernel method"). -
$t$ (Task): Applications, problems to solve, or systems to construct (e.g., "information extraction", "image segmentation").
-
-
Predicates:
-
$\text{UsedFor}(m, t)$ : A directed relationship indicating that method$m$ is used to solve task$t$ , or that$m$ is trained on/exploits$t$ .
-
The binary label reflects whether the specific UsedFor condition is met, distinguishing it from other valid scientific relationships found in the dataset.
-
YES (Positive Class):
- Logic: The logical expression evaluates to
True. - Interpretation: The text explicitly links a method and a task via application.
- Example: "The TISPER system has been designed to enable text applications." (Relation:
Used-For).
- Logic: The logical expression evaluates to
-
NO (Negative Class):
- Logic: The logical expression evaluates to
False. - Interpretation: While the text may contain scientific terms or other relationships, it does not describe a method application. This includes "Hard Negatives" such as taxonomic definitions or part-whole relationships.
- Example: "TUIT is a software library." (Relation:
Hyponym-Of, notUsed-For).
- Logic: The logical expression evaluates to
Source: huggingface.co (NLI4CT)
This task evaluates a model's ability to determine whether a natural language statement about clinical trial eligibility logically follows from (entails) or contradicts the stated inclusion/exclusion criteria. It requires interpreting medical terminology, applying logical inference, and recognizing synonyms.
The baseline prompt asks the model to verify logical consistency:
Does the statement accurately follow from the eligibility criteria?
Where:
-
Terms:
-
$s$ (Statement): A claim about clinical trial eligibility (e.g., "Patients with stage II T2 N2 carcinoma are not eligible"). -
$p$ (Premise): The inclusion and/or exclusion criteria from a clinical trial.
-
-
Predicates:
-
$\text{StatementFollowsFromPremise}(s, p)$ : The statement's claim logically follows from the criteria, accounting for synonyms (e.g., "claustrophobia" = "fear of confined spaces") and logical implications (e.g., if inclusion requires N0 or N1, then N2 is excluded).
-
-
YES (Entailment):
- Logic: The statement logically follows from the premise.
- Example: Premise: "Exclusion: Known claustrophobia" → Statement: "Patients with irrational fear of confined spaces are not eligible."
-
NO (Contradiction):
- Logic: The statement conflicts with or cannot be derived from the premise.
- Example: Premise: "Inclusion: Age ≥ 18" → Statement: "Pediatric patients are eligible for the trial."
Filtered from NLI4CT to eligibility-related instances:
| Filter | Criteria |
|---|---|
| Section | sentence1 contains "Eligibility", "Inclusion", "Exclusion", or "DISEASE CHARACTERISTICS" |
| Balance | Stratified sampling to balance Entailment/Contradiction labels |