Orchestrating AI Workflows with Mathematical Topology

Layer 1: Directed Acyclic Graph (Mature)

1.1 A workflow is a dependency graph

Mathematically, this is a Directed Acyclic Graph (DAG): nodes are tasks, and arrows represent "must be completed before." "Acyclic" ensures that all tasks can be completed in some sequence.

1.2 Topological Sorting: Sequence and Parallelism

Topological sorting provides an execution order that satisfies all dependencies. When expanded by "layers," tasks within the same layer do not depend on each other and can be assigned to different Agents for parallel execution.

The critical path is the longest dependency chain from start to finish; it determines the minimum time required to complete the entire process. To accelerate the whole workflow, it is only useful to shorten the tasks on the critical path.

A cycle implies a deadlock: Agent A is waiting for B's result, while B is waiting for A's result.

1.3 Code

The graphlib in the Python standard library can accomplish this:

from graphlib import TopologicalSorter, CycleError

workflow = {   # key = task, value = tasks it depends on
    "fetch_data":       set(),
    "clean":       {"fetch_data"},
    "factor_A":      {"clean"},
    "factor_B":      {"clean"},
    "causal_test":   {"factor_A", "factor_B"},
    "backtest":       {"factor_A", "factor_B"},
    "risk_review":   {"backtest", "causal_test"},
    "report":       {"risk_review"},
}
cost = {"fetch_data": 3, "clean": 2, "factor_A": 5, "factor_B": 8, "causal_test": 4, "backtest": 6, "risk_review": 2, "report": 1}

ts = TopologicalSorter(workflow); ts.prepare()
layer = 0
while ts.is_active():
    ready = list(ts.get_ready())               # this batch has no mutual dependencies, can run in parallel on different Agent
    print(f"Layer  {layer}  can run in parallel:", ready)
    ts.done(*ready); layer += 1

order = list(TopologicalSorter(workflow).static_order())
finish, prev = {}, {}
for t in order:
    deps = workflow[t]
    start = max((finish[d] for d in deps), default=0)
    prev[t] = max(deps, key=lambda d: finish[d]) if deps else None
    finish[t] = start + cost[t]
path, t = [], "report"
while t: path.append(t); t = prev[t]
print("critical path:", " → ".join(reversed(path)), " total time:", finish["report"])

try:
    list(TopologicalSorter({"A": {"B"}, "B": {"A"}}).static_order())
except CycleError as e:
    print("cycle found（deadlock）:", e.args[1])

Output:

Layer 0 parallelizable: ['Data Retrieval']
Layer 1 parallelizable: ['Cleaning']
Layer 2 parallelizable: ['Factor A', 'Factor B']
Layer 3 parallelizable: ['Causal Testing', 'Backtesting']
Layer 4 parallelizable: ['Risk Control Review']
Layer 5 parallelizable: ['Reporting']
Critical Path: Data Retrieval → Cleaning → Factor B → Backtesting → Risk Control Review → Reporting  Total Time: 22
Cycle detected (deadlock): ['A', 'B', 'A']

(The order of tasks within the same layer may vary depending on execution.)

1.4 Engineering Significance

Risk Control Review is designed as the common successor of "Backtesting" and "Causal Testing": structurally ensuring that any result must pass through risk control before entering the report. This involves writing security rules into the topological structure itself, rather than relying on an Agent to remember to do it (see Security Boundaries of Agent Trading Systems).

Layer 2: Persistent Homology—Visualizing the "Shape" of Agent Outputs (Research Foundation + Concept)

2.1 The Problem

Five Agents independently analyze the same dataset and provide conclusions. Are they:

• Consistent?
• Divided into several factions?
• Is there an "outlier"?

Looking only at the average opinion masks the structure. Persistent Homology in Topological Data Analysis (TDA) is used to describe how the "shape" of data changes across different scales.

2.2 Intuition

Treat each Agent's output (e.g., a scoring vector for a set of stocks, or the embedding vector of a conclusion text) as a point in space. Gradually increase a distance threshold $\varepsilon$ and connect points with a distance less than $\varepsilon$:

• When $\varepsilon$ is very small, each point is its own cluster;
• As $\varepsilon$ increases, points gradually merge into several clusters;
• If a cluster survives for a long $\varepsilon$ interval before merging with others, it represents a stable structure (a true "faction");
• If a point only merges at a very large $\varepsilon$, it is an outlier.

Recording the $\varepsilon$ of each cluster's "birth" and "death" yields a persistence barcode. 0-dimensional persistent homology describes "connected components" (clusters); 1-dimensional describes "loops" (e.g., circular, non-converging opinions between Agents).

2.3 Code: 0-Dimensional Persistent Homology

import numpy as np
rng = np.random.default_rng(0)
emb = np.vstack([rng.normal(0, 0.3, (5, 2)), rng.normal(3, 0.3, (5, 2)),[[8, 8]]])  # Two consensus groups + one outlier Agent
n = len(emb); parent = list(range(n))
def find(i):
    while parent[i] != i: parent[i] = parent[parent[i]]; i = parent[i]
    return i
edges = sorted((np.linalg.norm(emb[i] - emb[j]), i, j) for i in range(n) for j in range(i + 1, n))
deaths = []
for d, i, j in edges:
    a, b = find(i), find(j)
    if a != b: parent[a] = b; deaths.append(round(float(d), 2))
print("Death distances of connected components (persistence barcode):", deaths)

Output:

Distance to connected component extinction (persistence barcode): [0.11, 0.17, 0.24, 0.25, 0.32, 0.36, 0.42, 0.43, 3.27, 6.76]

2.4 How to Read It

• The first 8 death distances are very small (0.1–0.4): Agents within the group merge quickly, indicating consensus within the group;
• 3.27: Two factions do not merge until a distance of 3.27, indicating the existence of two stable, distinctly different opinion clusters;
• 6.76: The last merge involves the outlier Agent.

In a research workflow, this can become a rule: When long-lived branches appear in the barcode, do not simply average them; instead, hand the disagreement over to a human or an "Arbitration Agent" (see "Disagreement is a Valuable Signal" in Deconstructionism and Task Flows).

2.5 Existing Research Foundation

• Persistent homology was developed by Edelsbrunner, Carlsson, and others in the 2000s and is a core tool of Topological Data Analysis;
• In finance, research has used persistent homology to analyze stock return correlation networks, finding that topological features change significantly around market crises (e.g., the 2000 Dot-com bubble and the 2008 Financial Crisis), suggesting it could serve as an early warning signal for structural market shifts;
• Applying this to real-time monitoring of multi-Agent opinion structures is a concept proposed in this article and requires experimental validation.

Layer 3: Category Theory—Workflow Composability (Theoretical Concept)

3.1 Intuition

Category Theory studies "things" (objects) and "transformations between things" (morphisms), and how these transformations compose.

3.2 Conceptual Uses

• Type Checking: When assembling a workflow, automatically check "Can the output of Agent X be directly passed to Agent Y," preventing errors like connecting "factor values" to "returns";
• Replaceability: Two Agents with the same input and output types can be swapped for A/B testing of different models;
• Parallel Composition (Tensor product in monoidal categories): Two independent pipelines can be combined "side-by-side," corresponding to parallel layers in a DAG;
• Concepts like pipelines and monads in functional programming originate from category theory and are already widely used in software engineering.

3.3 Honest Assessment

Category theory here primarily provides a thinking framework and design constraints rather than new algorithms. For small-scale workflows, type systems and DAGs are sufficient; as workflows become very large and are dynamically generated by Agents, formalized composition rules may become more valuable.

Summary