Skip to content

Graph API

This page documents the pipeline-facing graph contracts.

The important boundary is the protocol and DTO layer, not the Neo4j backend. GraphReader.get_neighborhood() returns a SubGraph, which is the canonical transport object for structural context passed into Phase 3, Phase 4, and fusion logic.

Key Semantics

  • SubGraph is a serializable neighborhood snapshot, not a live graph handle.
  • center_id identifies the query center; the center node does not need to be duplicated in nodes.
  • nodes and triples expose the retrieved local context in pipeline-native types.
  • Storage backends may vary, but callers should rely only on the documented SubGraph and GraphReader contracts.

Envelope Construction and Traversal

When GraphReader.get_neighborhood() constructs a SubGraph envelope (e.g., for Phase 3 and Phase 4 LLM context), it explicitly filters out structural nodes (Chunk, Chapter, Document).

Why filter structural nodes?

  • Structural nodes like Chunk carry heavy metadata payloads (e.g., 4096-dimensional dense embeddings, phase2_processed flags, source IDs).
  • If traversed, these nodes would leak into the LLM prompt via the envelope formatting (format_envelope), causing severe context window overflows and diluting the semantic reasoning task.
  • By ignoring structural nodes during the apoc.path.subgraphAll traversal, the resulting SubGraph strictly contains semantic entity-to-entity and component-to-component relationships, which is the exact context needed for LLM relation extraction and argument mining.

Batch Persistence and Bulk I/O

The write interface GraphWriter exposes explicit plural persistence contracts (upsert_chunks, upsert_entities, upsert_triples, upsert_relations, upsert_argument_components, upsert_communities) alongside singular operations.

Key write invariants:

  • Single-Transaction Bulk Writes: Plural write operations execute within a single Cypher transaction using UNWIND $batch AS .... This reduces transaction management and round-trip network overhead by up to \(95\%\) during large ingestions.
  • DRY Singular Delegation: To avoid query duplication and ensure consistent label/property mapping, singular methods (upsert_entity, upsert_triple, upsert_chunk, etc.) wrap inputs in single-element lists and delegate to the plural batch implementations.
  • Explicit Contracts over Global Flags: There is no global enable_batching flag. Phases decide whether to batch data (typically via Python's standard itertools.batched) based on their algorithmic constraints.

For the architectural rationale and benchmarks, see ADR 0017: Explicit Batch Contracts & Bulk I/O.

Contracts

episteme_pipeline.contracts.domain.SubGraph

Bases: BaseModel

Depth-limited neighborhood returned by graph read operations.

This model is the pipeline-level transport object for structural context. It intentionally does not expose backend-specific graph handles or third- party graph objects. Consumers should treat it as a compact, serializable snapshot of a local property-graph neighborhood.

Parameters

center_id Identifier of the node around which the neighborhood was requested. The center node itself is referenced here and is not required to appear in nodes. nodes Neighbor nodes materialized for the neighborhood. Implementations should return the reachable context nodes needed by downstream phases. Ordering is not semantically significant. triples Directed relations contained in the neighborhood snapshot. These are graph edges represented in the pipeline's canonical L2Triple shape so downstream components can inspect structure without depending on the persistence backend. depth Maximum traversal depth that was requested to construct the neighborhood.

Source code in packages/episteme-pipeline/episteme_pipeline/contracts/domain.py
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
class SubGraph(BaseModel):
    """Depth-limited neighborhood returned by graph read operations.

    This model is the pipeline-level transport object for structural context.
    It intentionally does not expose backend-specific graph handles or third-
    party graph objects. Consumers should treat it as a compact, serializable
    snapshot of a local property-graph neighborhood.

    Parameters
    ----------
    center_id
        Identifier of the node around which the neighborhood was requested.
        The center node itself is referenced here and is not required to appear
        in ``nodes``.
    nodes
        Neighbor nodes materialized for the neighborhood. Implementations
        should return the reachable context nodes needed by downstream phases.
        Ordering is not semantically significant.
    triples
        Directed relations contained in the neighborhood snapshot. These are
        graph edges represented in the pipeline's canonical ``L2Triple`` shape
        so downstream components can inspect structure without depending on the
        persistence backend.
    depth
        Maximum traversal depth that was requested to construct the
        neighborhood.
    """
    center_id: str
    nodes: list[L2Entity]
    triples: list[L2Triple]
    depth: int

episteme_pipeline.protocols.graph_store.GraphReader

Bases: GraphHandle, ABC

Read surface for graph-backed pipeline components.

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
class GraphReader(GraphHandle, ABC):
    """Read surface for graph-backed pipeline components."""

    @abstractmethod
    async def get_chunks(
        self, filters: dict | None = None, limit: int | None = None
    ) -> list[L1Chunk]: ...

    @abstractmethod
    async def get_entities(
        self,
        labels: list[str] | None = None,
        filters: dict | None = None,
    ) -> list[L2Entity]: ...

    @abstractmethod
    async def get_all_entity_triples(self) -> list[L2Triple]: ...

    @abstractmethod
    async def get_neighborhood(
        self, node_id: str, depth: int = 1
    ) -> SubGraph:
        """Return a depth-limited structural neighborhood around a node.

        Parameters
        ----------
        node_id
            Identifier of the center node whose local graph context should be
            retrieved.
        depth
            Maximum hop distance from ``node_id`` to include in the returned
            neighborhood.

        Returns
        -------
        SubGraph
            Serializable neighborhood snapshot centered on ``node_id``. The
            result must expose the center identifier, neighboring nodes,
            directed relations, and the requested depth without leaking
            backend-specific driver objects.

        Notes
        -----
        This method defines a pipeline contract, not a storage-engine detail.
        Implementations may use Neo4j, in-memory fixtures, or other backends,
        but consumers should only rely on the ``SubGraph`` semantics.
        """
        ...

    @abstractmethod
    async def vector_search(
        self,
        embedding: list[float],
        top_k: int,
        node_label: str | None = None,
    ) -> list[SearchResult]: ...

    @abstractmethod
    async def get_theory_atoms(self) -> list[TheoryAtom]: ...

    @abstractmethod
    async def get_all_theory_relations(self) -> list[TheoryRelation]: ...

    @abstractmethod
    async def find_entities_by_name(
        self, name: str, label: str | None = None
    ) -> list[L2Entity]: ...

    @abstractmethod
    async def get_entity_envelopes(self, entity_id: str) -> list[str]:
        """Return the stored textual envelopes for ``entity_id``.

        An envelope is the sentence-level context a mention was extracted from.
        Phase 4 maturation needs these to synthesise an entity description, and
        used to guard the call with ``hasattr`` because the contract did not
        declare it. Implementations with no envelope storage return an
        empty list.
        """
        ...

    @abstractmethod
    async def get_chunk_entities(self, chunk_id: str) -> list[L2Entity]:
        """Return all entities extracted from a specific chunk."""
        ...

get_chunk_entities(chunk_id) abstractmethod async

Return all entities extracted from a specific chunk.

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
106
107
108
109
@abstractmethod
async def get_chunk_entities(self, chunk_id: str) -> list[L2Entity]:
    """Return all entities extracted from a specific chunk."""
    ...

get_entity_envelopes(entity_id) abstractmethod async

Return the stored textual envelopes for entity_id.

An envelope is the sentence-level context a mention was extracted from. Phase 4 maturation needs these to synthesise an entity description, and used to guard the call with hasattr because the contract did not declare it. Implementations with no envelope storage return an empty list.

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
 94
 95
 96
 97
 98
 99
100
101
102
103
104
@abstractmethod
async def get_entity_envelopes(self, entity_id: str) -> list[str]:
    """Return the stored textual envelopes for ``entity_id``.

    An envelope is the sentence-level context a mention was extracted from.
    Phase 4 maturation needs these to synthesise an entity description, and
    used to guard the call with ``hasattr`` because the contract did not
    declare it. Implementations with no envelope storage return an
    empty list.
    """
    ...

get_neighborhood(node_id, depth=1) abstractmethod async

Return a depth-limited structural neighborhood around a node.

Parameters

node_id Identifier of the center node whose local graph context should be retrieved. depth Maximum hop distance from node_id to include in the returned neighborhood.

Returns

SubGraph Serializable neighborhood snapshot centered on node_id. The result must expose the center identifier, neighboring nodes, directed relations, and the requested depth without leaking backend-specific driver objects.

Notes

This method defines a pipeline contract, not a storage-engine detail. Implementations may use Neo4j, in-memory fixtures, or other backends, but consumers should only rely on the SubGraph semantics.

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
@abstractmethod
async def get_neighborhood(
    self, node_id: str, depth: int = 1
) -> SubGraph:
    """Return a depth-limited structural neighborhood around a node.

    Parameters
    ----------
    node_id
        Identifier of the center node whose local graph context should be
        retrieved.
    depth
        Maximum hop distance from ``node_id`` to include in the returned
        neighborhood.

    Returns
    -------
    SubGraph
        Serializable neighborhood snapshot centered on ``node_id``. The
        result must expose the center identifier, neighboring nodes,
        directed relations, and the requested depth without leaking
        backend-specific driver objects.

    Notes
    -----
    This method defines a pipeline contract, not a storage-engine detail.
    Implementations may use Neo4j, in-memory fixtures, or other backends,
    but consumers should only rely on the ``SubGraph`` semantics.
    """
    ...

episteme_pipeline.protocols.graph_store.GraphWriter

Bases: GraphHandle, ABC

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
class GraphWriter(GraphHandle, ABC):
    @abstractmethod
    async def upsert_node(
        self,
        label: str,
        node_id: str,
        properties: dict,
        *,
        extra_labels: tuple[str, ...] = (),
        create_only_properties: dict | None = None,
    ) -> None:
        """Idempotently write a node identified by ``label`` and ``node_id``.

        ``extra_labels`` are additional labels stamped on the node (the
        ``:Entity`` marker, F-05). ``create_only_properties`` are written once,
        when the node is first created, and never rewritten — first-seen
        timestamps belong here so that re-running an unchanged corpus leaves the
        graph byte-identical (F-11).
        """
        ...

    @abstractmethod
    async def prune_document_children(
        self,
        document_id: str,
        *,
        keep_chunk_ids: list[str],
        keep_chapter_ids: list[str],
    ) -> dict[str, int]:
        """Delete the document's Chapter/Chunk nodes that are no longer produced.

        Chunk identity is content-addressed, so re-ingesting an edited source
        creates new chunk nodes rather than updating the old ones. Callers pass
        the ids the current ingest produced; everything else under
        ``document_id`` is removed. Returns ``{"chunks": n, "chapters": n}``.
        """
        ...

    @abstractmethod
    async def upsert_relation(
        self,
        from_id: str,
        relation_type: str,
        to_id: str,
        properties: dict | None = None,
    ) -> None: ...

    @abstractmethod
    async def upsert_relations(
        self, relations: list[dict]
    ) -> None: ...

    @abstractmethod
    async def upsert_chunk(self, chunk: L1Chunk) -> None: ...

    @abstractmethod
    async def upsert_chunks(self, chunks: list[L1Chunk]) -> None: ...

    @abstractmethod
    async def upsert_entity(self, entity: L2Entity) -> None: ...

    @abstractmethod
    async def upsert_entities(self, entities: list[L2Entity]) -> None: ...

    @abstractmethod
    async def upsert_triple(self, triple: L2Triple) -> None: ...

    @abstractmethod
    async def upsert_triples(self, triples: list[L2Triple]) -> None: ...

    @abstractmethod
    async def upsert_argument_component(
        self, component: TheoryAtom
    ) -> None: ...

    @abstractmethod
    async def upsert_argument_components(
        self, components: list[TheoryAtom]
    ) -> None: ...

    @abstractmethod
    async def upsert_community(self, community_id: str, level: int, entity_ids: list[str]) -> None: ...

    @abstractmethod
    async def upsert_communities(self, communities: list[dict]) -> None: ...

prune_document_children(document_id, *, keep_chunk_ids, keep_chapter_ids) abstractmethod async

Delete the document's Chapter/Chunk nodes that are no longer produced.

Chunk identity is content-addressed, so re-ingesting an edited source creates new chunk nodes rather than updating the old ones. Callers pass the ids the current ingest produced; everything else under document_id is removed. Returns {"chunks": n, "chapters": n}.

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
@abstractmethod
async def prune_document_children(
    self,
    document_id: str,
    *,
    keep_chunk_ids: list[str],
    keep_chapter_ids: list[str],
) -> dict[str, int]:
    """Delete the document's Chapter/Chunk nodes that are no longer produced.

    Chunk identity is content-addressed, so re-ingesting an edited source
    creates new chunk nodes rather than updating the old ones. Callers pass
    the ids the current ingest produced; everything else under
    ``document_id`` is removed. Returns ``{"chunks": n, "chapters": n}``.
    """
    ...

upsert_node(label, node_id, properties, *, extra_labels=(), create_only_properties=None) abstractmethod async

Idempotently write a node identified by label and node_id.

extra_labels are additional labels stamped on the node (the :Entity marker, F-05). create_only_properties are written once, when the node is first created, and never rewritten — first-seen timestamps belong here so that re-running an unchanged corpus leaves the graph byte-identical (F-11).

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
@abstractmethod
async def upsert_node(
    self,
    label: str,
    node_id: str,
    properties: dict,
    *,
    extra_labels: tuple[str, ...] = (),
    create_only_properties: dict | None = None,
) -> None:
    """Idempotently write a node identified by ``label`` and ``node_id``.

    ``extra_labels`` are additional labels stamped on the node (the
    ``:Entity`` marker, F-05). ``create_only_properties`` are written once,
    when the node is first created, and never rewritten — first-seen
    timestamps belong here so that re-running an unchanged corpus leaves the
    graph byte-identical (F-11).
    """
    ...

episteme_pipeline.protocols.graph_store.PhaseCheckpointStore

Bases: GraphHandle, ABC

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
class PhaseCheckpointStore(GraphHandle, ABC):
    @abstractmethod
    async def mark_chunk_processed(self, chunk_id: str, phase: str) -> None: ...

    @abstractmethod
    async def mark_chunks_processed(self, chunk_ids: list[str], phase: str) -> None: ...

    @abstractmethod
    async def get_unprocessed_chunks(
        self, phase: str, limit: int | None = None
    ) -> list[L1Chunk]: ...

    @abstractmethod
    async def filter_unprocessed_items(
        self, item_keys: list[str], phase: str
    ) -> list[str]:
        """Filter item keys and return only those not yet marked processed.

        Parameters
        ----------
        item_keys : list[str]
            Candidate item keys to check.
        phase : str
            Phase identifier tag.

        Returns
        -------
        list[str]
            Subset of item keys that have not been marked in the store.
        """
        ...

    @abstractmethod
    async def commit_phase_batch(
        self,
        phase: str,
        triples: list[L2Triple],
        items: list[PhaseItemRecord],
    ) -> None:
        """Atomically commit triples and item checkpoints within a single transaction.

        Parameters
        ----------
        phase : str
            Phase identifier tag.
        triples : list[L2Triple]
            Triples to upsert.
        items : list[PhaseItemRecord]
            Phase item checkpoint records to persist.
        """
        ...

    @abstractmethod
    async def clear_phase_checkpoints(self, phase: str) -> None:
        """Clear all item checkpoints for the given phase upon invalidation.

        Parameters
        ----------
        phase : str
            Phase identifier tag.
        """
        ...

clear_phase_checkpoints(phase) abstractmethod async

Clear all item checkpoints for the given phase upon invalidation.

Parameters

phase : str Phase identifier tag.

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
251
252
253
254
255
256
257
258
259
260
@abstractmethod
async def clear_phase_checkpoints(self, phase: str) -> None:
    """Clear all item checkpoints for the given phase upon invalidation.

    Parameters
    ----------
    phase : str
        Phase identifier tag.
    """
    ...

commit_phase_batch(phase, triples, items) abstractmethod async

Atomically commit triples and item checkpoints within a single transaction.

Parameters

phase : str Phase identifier tag. triples : list[L2Triple] Triples to upsert. items : list[PhaseItemRecord] Phase item checkpoint records to persist.

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
@abstractmethod
async def commit_phase_batch(
    self,
    phase: str,
    triples: list[L2Triple],
    items: list[PhaseItemRecord],
) -> None:
    """Atomically commit triples and item checkpoints within a single transaction.

    Parameters
    ----------
    phase : str
        Phase identifier tag.
    triples : list[L2Triple]
        Triples to upsert.
    items : list[PhaseItemRecord]
        Phase item checkpoint records to persist.
    """
    ...

filter_unprocessed_items(item_keys, phase) abstractmethod async

Filter item keys and return only those not yet marked processed.

Parameters

item_keys : list[str] Candidate item keys to check. phase : str Phase identifier tag.

Returns

list[str] Subset of item keys that have not been marked in the store.

Source code in packages/episteme-pipeline/episteme_pipeline/protocols/graph_store.py
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
@abstractmethod
async def filter_unprocessed_items(
    self, item_keys: list[str], phase: str
) -> list[str]:
    """Filter item keys and return only those not yet marked processed.

    Parameters
    ----------
    item_keys : list[str]
        Candidate item keys to check.
    phase : str
        Phase identifier tag.

    Returns
    -------
    list[str]
        Subset of item keys that have not been marked in the store.
    """
    ...