Artifact and Run Model¶

This document defines the architecture of the Artifact and Run Model in Episteme.

The pipeline executes entirely on durable, run-scoped research artifacts and first-class run manifests, rather than ephemeral inter-phase execution values or graph-as-run-state.

1. Purpose & Motivation¶

In earlier iterations of the pipeline, intermediate outputs were transient Python objects passed directly in memory, and the Neo4j database was overloaded as both execution state tracker and target knowledge graph.

The artifact and run model solves this by establishing two decoupled architectural foundations:

1. Run Model — Tracks what execution took place, with which configuration snapshot, input fingerprints, method fingerprints, and phase records.
2. Artifact Model — Tracks what was produced by each phase, with full provenance, stable identity keys, and content-addressed dependency fingerprints.

Key Architectural Benefits¶

• Graph-Independent Execution: Intermediate outputs are evaluated, replayed, and inspected without querying Neo4j. Graph persistence is isolated to an explicit projection step.
• Reproducibility & Lineage: Every artifact carries provenance pointing back to source files, chunks, upstream artifact IDs, and run manifests.
• Safe Resumption & Caching: Downstream phases can hydrate upstream artifacts across a chain of parent runs without re-executing valid prior work.
• Fine-Grained Invalidation: Runs record deterministic fingerprints of inputs, configs, prompts, and model parameters to detect exactly which phases or artifacts require recomputation.

2. Core Architecture¶

flowchart TD
    subgraph Orchestration ["Pipeline Orchestration (pipeline/pipeline.py)"]
        ManifestStore["JsonRunManifestStore<br/>(runs_dir/run-id/manifest.json)"]
        ArtifactStore["JsonArtifactStore<br/>(artifacts_dir/run-id/kind/id.json)"]
        Context["ArtifactExecutionContext<br/>(run_id, manifest, input, previous)"]
    end

    subgraph PhaseExecution ["Phase Runner Execution"]
        InputView["Typed ArtifactsView<br/>(e.g., Phase2ArtifactsView)"]
        Runner["PhaseRunner.run(input, context)"]
        Collection["ArtifactCollection<br/>([ArtifactEnvelope, ...])"]
    end

    subgraph Downstream ["Persistence & Projection"]
        Projector["ArtifactGraphProjector<br/>(pipeline/projection/)"]
        Neo4j[("Neo4j Projection Graph")]
    end

    ManifestStore --> Context
    Context --> Runner
    InputView --> Runner
    Runner --> Collection
    Collection --> ArtifactStore
    Collection --> Projector
    Projector --> Neo4j

3. The Run Model¶

Located in pipeline/runs/models.py and persisted via pipeline/runs/persistence.py.

3.1 RunManifest¶

The RunManifest is the central execution record for a pipeline run:

• run_id: Unique run identifier (e.g., run-<uuid4>).
• parent_run_id: Identifier of the parent run when resuming or reusing prior phases.
• status: Current lifecycle status (RunStatus.PLANNED, RUNNING, COMPLETED, FAILED, ABORTED).
• schema_version: Active SchemaConfig.version tracking ontology/taxonomy versions.
• source_fingerprint & input_fingerprint: Deterministic hashes of source document paths, bibtex files, and structural anchors.
• config_snapshot: Full JSON-serialized snapshot of PipelineConfig.
• phase_config_fingerprints: Per-phase configuration hashes derived from serialized phase configs (including fine-grained nested leaf fingerprints).
• method_fingerprints: Fine-grained fingerprints of models, extractors, and prompts used by each phase (e.g., LLM parameters, embedding model dimensions, and prompt bundles).
• phase_records: List of RunPhaseRecord tracking execution status and artifact outputs per phase.

3.2 RunPhaseRecord¶

Tracks execution state for an individual phase:

• phase_name and phase_ordinal (1-based index).
• status: RunStatus for this specific phase.
• started_at and completed_at timestamps.
• input_artifact_ids: IDs of artifacts consumed as input.
• output_artifact_ids: IDs of artifacts produced by this phase.
• output_fingerprint: Stable hash of the emitted artifact collection.
• reused: Boolean flag indicating whether the phase was reused from a prior run.

3.3 RunReport¶

At run completion, a RunReport is generated summarizing:

• Overall status and execution durations.
• Counts of artifacts by kind and by phase.
• Breakdown of new vs. reused artifacts.
• Invalidation decisions and reasons.
• Full list of ArtifactReportEntry references.

4. The Artifact Model¶

Located in pipeline/artifacts/models.py and persisted via pipeline/artifacts/store.py.

4.1 ArtifactEnvelope[T]¶

Every artifact emitted by the pipeline is wrapped in a generic ArtifactEnvelope[T]:

class ArtifactEnvelope(BaseModel, Generic[ArtifactPayloadT]):
    artifact_id: str
    identity_key: str | None = None
    kind: ArtifactKind
    run_id: str
    phase_name: str
    method: str
    method_version: str = "v1"
    config_fingerprint: str | None = None
    dependency_fingerprint: str | None = None
    created_at: datetime = Field(default_factory=utc_now)
    provenance: ArtifactProvenance = Field(default_factory=ArtifactProvenance)
    payload: ArtifactPayloadT
    metadata: dict[str, Any] = Field(default_factory=dict)

Dual Identifier System¶

Every artifact carries two complementary identifiers:

1. artifact_id (artifact::<uuid>): Unique identifier representing the specific artifact instance in a particular run. Used in provenance.upstream_artifact_ids to track concrete data lineage.
2. identity_key (<kind>::<canonical_identifier>): Stable semantic identity across runs (e.g., linked_entity::kant_crp_space). Used by the invalidation DAG and cross-run comparison tools to correlate equivalent entities across parameter changes.

4.2 Payload Families¶

The pipeline defines typed payloads for each construction stage:

5. Execution & Inter-Phase Handoff¶

Located in pipeline/artifacts/execution.py.

5.1 ArtifactCollection¶

An ArtifactCollection is a typed container holding a list of ArtifactEnvelope instances. It provides filtering utilities such as collection.of_kind(*kinds).

5.2 Typed ArtifactsView Classes¶

Phases consume upstream data through typed views. Rather than receiving untyped collections or raw database connections, each phase runner specifies an input_view class:

• Phase1ArtifactsView: Provides documents: list[L1Document] and chunks: list[L1Chunk]. Consumed by Phase 2.
• Phase2ArtifactsView: Provides entities: list[L2Entity], local_triples: list[L2Triple], and type distributions. Consumed by Phase 3.
• Phase3ArtifactsView: Provides global_triples: list[L2Triple], chunks: list[L1Chunk], and relation distributions. Consumed by Phase 3b, Phase 4 Maturation, and Phase 4 Argument Mining.
• Phase4ArtifactsView: Provides theory_atoms: list[TheoryAtom] and theory_relations: list[TheoryRelation]. Consumed by Phase 5, Phase 6, and Theoretical Enrichment.
• TheoreticalEnrichmentArtifactsView: Provides enrichments: list[TheoreticalEnrichmentArtifact].

5.3 Runner Interface¶

All pipeline phase runners implement the uniform async interface:

async def run(
    self,
    input: ViewT,
    context: ArtifactExecutionContext,
) -> ArtifactCollection: ...

The orchestrator builds the appropriate view via entry.input_view.from_collection(previous) and passes it to the runner alongside the ArtifactExecutionContext (which contains run_id, manifest, pipeline_input, and previous).

6. Graph Projection & Decoupling¶

Located in pipeline/projection/artifact_projector.py and pipeline/projection/theorynet_projector.py.

Execution is cleanly decoupled from graph storage:

1. Phases emit artifacts: Phase runners have no direct write dependencies on the final projection graph.
2. Independent projection: When config.execution.project_artifacts_to_graph is enabled (default True), the orchestrator passes emitted artifacts through ArtifactGraphProjector.project(art).
3. Dual-Store Architecture: The temporary checkpoint_store (operational graph) is used strictly for working memory, co-occurrence blocking, and intermediate index lookups during extraction, while projection_graph receives canonical projections derived from validated artifacts.
4. Dual Projection Streams:
5. Property Graph Projection (ArtifactGraphProjector): Projects artifact envelopes into concrete Neo4j nodes and relationships.
6. TheoryNet Formalism Projection (TheoryNetProjector): Projects Phase 4 artifacts into the formal mathematical \(\mathcal{G}_{\text{TheoryNet}}\) structure, executing iterative QBAF gradual semantics.

6.1 Artifact-to-Graph Projection Mapping¶

ArtifactGraphProjector maps artifact payloads into the projection graph according to the following rules:

6.2 Current Limitations & Roadmap¶

As the pipeline transitions fully toward an artifact-first runtime, the projection layer currently serves as an explicit bridge. The following enhancements are planned:

• Transactional Projection Batches: Grouping multi-artifact projections into single atomic Cypher transactions.
• Projector Plugins: Pluggable projectors allowing custom export targets (e.g., RDF/OWL, NetworkX, GraphML).
• Projection Planning: Dependency-aware projection ordering ensuring referenced nodes are guaranteed to exist prior to edge creation.
• Projection-Time Schema Validation: Validating projected graph structures against SchemaConfig constraints before committing writes.
• Replay Tooling: Command-line utilities to re-project historical runs from .pipeline_artifacts/ without re-running extraction.

7. Artifact Hydration & Parent-Run Lineage¶

When a run is resumed via resume_from_run() or reuses phases via _choose_reuse_source():

1. The orchestrator identifies the source run (parent_run_id).
2. _hydrate_previous_collection(phase_number, source_run_id) retrieves all artifacts produced prior to the resume boundary.
3. If an artifact was inherited across multiple resume hops, the store traverses the parent-run chain (run_id -> parent_run_id -> ...), always selecting the nearest occurrence of each identity_key.
4. Downstream phases execute seamlessly with a fully populated ArtifactCollection as if all upstream phases had executed in the current process.

8. Configuration Controls¶

The artifact and run model is configured via ExecutionConfig in pipeline/config.py:

9. Related Documentation¶

• Invalidation and Resumption: Invalidation and Resume
• Runtime and Complexity: Pipeline Runtime Analysis
• System Overview: System Architecture Overview
• Pipeline Architecture: Pipeline Architecture
• ADRs: ADR 0001: Neo4j Async Graph Backend, ADR 0014: Deterministic Identity Diffing