Datasets Strategy¶

Episteme employs a strategic dataset portfolio designed to balance rigorous empirical evaluation against standard natural language processing benchmarks with deep theoretical and argumentative validation on German and English scholarly literature.

Dataset Selection Principles¶

Our dataset strategy adheres to four core principles:

1. Scholarly Depth & Domain Relevance: Prioritize academic philosophy and science literature exhibiting rich theoretical frameworks, complex argument structures, and explicit conceptual hierarchies.
2. Multi-Task Coverage: Ensure empirical validation across the entire construction lifecycle: Named Entity Recognition (NER), Entity Linking (EL), Relation Extraction (RE), and Argument Mining (ADU, ACC, ARC).
3. Multilingualism (DE/EN): Support both German and English corpora to evaluate cross-lingual concept alignment and domain-specific terminological transfer.
4. Reproducibility & Open Licensing: Favor permissive licenses (CC-BY, MIT, BSD) or clear research instructions with automated adapter mappings.

Dataset Portfolio Matrix¶

The table below summarizes the complete evaluation portfolio used across pipeline benchmarking, domain review, and stress testing:

Dataset Roles & Stratification¶

In accordance with docs/research/evaluation_methodology.md, corpora are partitioned into four operational roles:

graph TD
    subgraph Stratification ["Dataset Stratification"]
        GOLD["Gold Standards (Curated reference data for NER, RE, and ARCs)"]
        SILVER["Silver Corpora (Weakly-supervised or larger-scale benchmarking)"]
        REVIEW["Domain Review Sets (Expert-reviewed edge cases & ambiguous entities)"]
        STRESS["Stress-Test Collections (Dense citations, cross-lingual terminology)"]
    end

• Gold Datasets: High-quality ground truth used for quantitative scoring with GM-GBS, OEP, and Information Retrieval metrics:
  • Layer 1 & 2 (Extraction & Relations): SciERC, SciFact.
  • Layer 3 (Argument Components): Arg-Microtexts.
  • Layer 4 (TheoryNet & Architectonic Structure): Structuralist Theory-Net Benchmark (STNB). Evaluates formal model classes (\(\mathcal{M}_p, \mathcal{M}, \mathcal{M}_{pp}\)), constraints (\(GC\)), and specialization trees (\(\alpha\)). Ground truth pairs human-authored reconstructions (Balzer et al.) with a Model-Assisted Human Verification (HITL) protocol where candidate text anchors are model-proposed, double-blind audited, and adjudicated by domain scholars (\(\alpha \ge 0.80\)).
• Silver Datasets: Larger corpora used for triage, ablation studies, and broad sensitivity sweeps.
• Review Datasets: Targeted passages where domain experts execute structured rubric assessments (EvaluationRubric).
• Stress-Test Collections: Passages specifically engineered to expose failure modes:
  • Dense References: Texts with extreme citation density and nested theoretical attributions.
  • Ambiguous Naming: Polysemous concepts across different historical traditions (e.g., Idealismus in Kant vs. Hegel).
  • Cross-Lingual Shifting: Mixed German/English technical terminology.

Adapter Architecture¶

To prevent coupling between external dataset schemas and the internal Episteme graph model, all datasets interface through dedicated adapters in packages/episteme-pipeline/evaluation/adapters/.

Adapter Responsibilities¶

Each adapter performs three normalization steps:

1. Label Mapping: Normalizes dataset-specific entity and relation labels into the canonical taxonomy defined in pipeline/schema/.
2. Text & Span Alignment: Normalizes character offsets and section boundaries to generate standard L1Chunk and L1Document inputs.
3. Gold Standard Serialization: Converts ground-truth annotations into nx.DiGraph instances or SearchResult reference sets for immediate consumption by GM-GBS, OEP, and retrieval scorers.

Active Adapter Implementations¶

• Layer 1 & 2 (Factual Extraction): adapters/scierc.py and adapters/scifact.py normalize entity and factual relation benchmarks.
• Layer 3 (Argumentation): adapters/arg_microtexts.py maps argument component and relation classification tasks.
• Layer 4 (Theory-Nets): adapters/structuralist.py normalizes JSON-LD structuralist graph records (TheoryNet, TheoryElement, ActualModel, Constraint) into canonical (list[L1Chunk], nx.DiGraph) representations with preserved Episteme:textAnchor coordinates.

Adapter Implementation Pattern¶

"""Typical Adapter Structure (e.g. evaluation/adapters/scierc.py or structuralist.py)"""

from pathlib import Path
import json
import networkx as nx
from pipeline.contracts.domain import L1Chunk


def load_dataset(data_path: Path) -> list[dict]:
    """Ingests raw dataset serialization."""
    ...


def adapt_to_evaluation_format(raw_data: list[dict]) -> tuple[list[L1Chunk], nx.DiGraph]:
    """
    Transforms raw annotations into:
    1. Pipeline inputs (chunks and text).
    2. Gold standard DiGraph for intrinsic scoring.
    """
    ...

Contamination Prevention & Reproducibility¶

Contamination Guardrails¶

To ensure that reported evaluation metrics reflect genuine generalization:

• Closed-Book Evaluation: Standard benchmark runs prohibit dynamic web search or external retrieval beyond the declared corpus scope.
• Split Isolation: Strict physical separation is maintained between prompt few-shot development examples and evaluation test splits.
• Model Provenance: All runs record the exact model version, API provider, and decoding parameters in the run manifest.
• Model-Assisted Annotation Isolation: In human-in-the-loop curated benchmarks (such as STNB), the LLM model family used to propose candidate text spans during curation is permanently recorded in run provenance manifests and strictly partitioned from the models evaluated on the benchmark to prevent circular self-evaluation and data leakage.

Environmental Locking¶

Every evaluation execution is cryptographically reproducible:

• Dataset files are checksummed and referenced by persistent paths.
• Execution manifests capture random seeds, prompt template checksums, and git commit SHAs.
• Artifact lineage links all evaluation metrics directly to the intermediate artifacts generated during the run.

Related Documentation¶

• Structuralist Theory-Net Benchmark (STNB): STNB Specification & Tasks
• Evaluation Methodology & Scaffolding: Evaluation Methodology
• Evaluation Harness Architecture: Evaluation Harness
• Empirical Construction Metrics: Empirical Metrics & Validation
• Run Manifest Specification: run_manifest.schema.yaml