The Risk Isn’t Agency. It’s Interaction

Why Networked AI Systems Create Risk Before Autonomy

When people first started talking about “agentic AI,” I assumed they meant something much stronger than what they actually did.

My background probably biased me. The first models I built were agent-based. My dissertation leaned heavily on them, and my early papers did, too. In that world, the word agent already carried weight.

As shown in my 2015 paper on transit dynamics (Figure 1), an agent is defined by its internal logic and environmental triggers. A commuter transitions from “Waiting” to “In train” based on conditions like “train arrived and not full.” It pointed to systems where components act, interact, and collectively produce behavior that no single component controls.

Figure 1 (originally Figure 3 in our paper “A data-driven agent-based model of congestion and scaling dynamics of rapid transit systems”): State diagrams for train and commuter agents.

So when I heard “agentic AI,” my mind jumped ahead. I thought people were talking about agency.

Not metaphorical agency. Not task autonomy. I mean the stronger idea: systems that set their own objectives, revise them through experience, and participate in the world as decision-making entities in their own right. Something closer to what people usually bundle under AGI, whether or not they like the term.

That’s not what I learned once I read more carefully.

What “agentic AI” usually refers to today is much closer to what we’ve been building for years in agent-based and multi-agent systems. These are systems that can act, plan, call tools, and coordinate with other systems, but whose goals remain externally defined. Yes, there is autonomy, but it’s delegated. The system can decide how to do something, not what it should ultimately care about.

That’s where the “-ic” comes in. No pun intended.

Agentic doesn’t mean Agent in the strong conceptual sense. It means agent-like: capable of action and coordination, able to operate without step-by-step human instruction, but still anchored to objectives set elsewhere.

Seen this way, agentic AI isn’t a radical break from agent-based modeling, but merely a continuation, with more capable components and tighter coupling to real-world systems. And that’s exactly why it’s interesting.

You don’t need genuine agency for things to get complex.

Once multiple systems interact, even if each behaves deterministically in isolation, uncertainty creeps in through timing, context, and feedback. One system’s output becomes another’s input. Decisions alter the environment that future decisions respond to. Loops form.

This alone is enough to produce Emergence.

This is a familiar result in complexity science. Emergent behavior doesn’t require intent or self-directed goals. It requires interaction, heterogeneity, and feedback. Once those are in place, system-level behavior quickly escapes what you can infer from any single component.

That’s the lens I bring to conversations about agentic AI. Not because I think these systems already have agency, but because I know how much can happen long before they ever do.

What worries me is not that these systems will suddenly “wake up,” but that their behavior becomes harder to attribute as interactions multiply.

As agentic systems are chained together, responsibility diffuses. Decisions emerge from interactions rather than from any single model or rule. When something goes wrong, there is no obvious place to point. The failure doesn’t reside in a single component; it spans the network. By the time an issue surfaces, it’s often unclear which decision mattered, which assumption broke, or which interaction amplified a small error into a visible outcome.

This is where cascading failures stop being abstract and become organizational.

Gif images: Takashi Nishikawa and Adilson E. Motter. An example of a cascading failure. “Just a few vulnerable patches in North American power grids are responsible for most of the continent’s largest blackouts.”

For executives, this shows up as risk that is difficult to anticipate and even harder to explain after the fact. For data science teams, it shows up as systems that technically perform as designed, yet produce outcomes no one intended. Each model may pass validation, sure. Each component may meet its metric, okay. Still, the system as a whole behaves in ways that surprise everyone involved.

The uncomfortable part is that none of this requires bad actors, broken models, or reckless deployment. It follows naturally once systems interact at speed, at scale, and with limited human oversight.

That’s why the question isn’t whether agentic AI is powerful enough to shape outcomes. It already is. The more important question is whether organizations are prepared to reason about behavior that no longer lives inside a single model, a single team, or a single decision point.

This is also why monitoring and feedback become central, not optional, in any governance framework. When risk emerges from interaction, design-time controls and one-off reviews aren’t enough. Organizations need ways to observe how systems behave together, how decisions propagate, and how small deviations accumulate over time.

Monitoring preserves the ability to understand system behavior as interactions unfold, while there is still room to learn and intervene.

This came up recently in a research meeting I had with an enterprise data science and AI team preparing to experiment with agentic systems. They were debating whether to deploy them in a fully autonomous mode or keep them semi-autonomous, with tighter human oversight. The technology could support either. The real question was organizational.

My answer was straightforward: start with semi-autonomy. Not because the systems aren’t capable, but because learning still matters more than speed. Once systems begin interacting, the early signals are subtle. You only see them if you’re watching closely, and you only learn from them if there’s still room to intervene. Full autonomy closes that window too quickly.

The risk isn’t that these systems will behave unpredictably in isolation. It’s that they will behave coherently in ways no one explicitly planned. Without feedback loops that surface system-level behavior, organizations lose the ability to attribute outcomes, adjust assumptions, or even ask the right questions about what went wrong.

We’ve seen this pattern before in other complex systems. Finance, infrastructure, and supply chains all learned the same lesson: when interactions dominate, local correctness is not enough. What is important is how the system behaves under stress, uncertainty, and feedback.

Agentic AI brings that lesson into software and decision-making itself. The challenge now is to recognize it early, while these systems are still understandable enough to learn from, rather than later, when the learning comes at a higher cost.