An AI agent interacting with an external system needs permission boundaries enforced outside the model, a named person accountable for its operation, and a way to stop it in real time. Without those safeguards, even an experiment intended to help can become a security incident and a legal dispute. Saying the agent acted against its operator’s intentions is no substitute for controls, and certainly does not provide automatic immunity.
The incident involving Railway and Primcloud illustrates the gap between being able to run an agent and being able to govern one. The lesson is not that autonomy is a mistake. It is that authorization cannot remain a sentence in a prompt.
What happened between Railway and Primcloud?
According to public accounts of the incident, Primcloud founder Josh Manders accused a Railway employee of attempting to break into the company’s infrastructure. The employee, Jake R anz er, acknowledged running agents that registered accounts and performed tests on similar platforms. He said the software wrapper he had built around them went beyond what he intended. He denied malicious intent and said the activity had stopped.
Railway CEO Jake Cooper apologized. In a later explanation, he described a tool the employee had developed in his spare time to identify and report vulnerabilities through bug bounty programs, using an external service based on DeepSeek. According to Cooper’s account, the tool ran once, and the employee stopped it within 15 minutes. Manders responded that Primcloud had no such bounty program and that its policies prohibited this kind of scanning and testing.
The parties’ claims must be kept separate from verified findings. This account does not establish that a successful intrusion occurred, what information may have been exposed, or what the legal outcome will be. It does, however, reveal a clear governance problem: a purpose the operator considers beneficial is not authorization from the system’s owner.
Good intentions are not authorization: Wanting to find vulnerabilities does not grant permission to test a third-party system. Even where a bug bounty program exists, its permitted targets, methods, and conditions must be checked.
This distinction extends well beyond security research. An agent that purchases information, creates accounts, contacts customers, or collects website data can also cross a boundary the organization never intended to cross.
Governance cannot wait for a legal ruling
There is no single legal answer to the question of who is responsible. Criminal liability, civil liability, breach of contract, and privacy violations are different matters. Applicable law, the operator’s knowledge, precautions taken, and the circumstances of deployment can all affect the analysis.
A legal argument developed in one jurisdiction should not be applied wholesale to an enterprise operating elsewhere. Interactions with overseas services may raise questions across multiple jurisdictions. Depending on the activity, organizations need to consider computer misuse laws, privacy requirements, contracts, and terms of service. That requires specific legal advice, not the model’s own interpretation.
But a manager does not need to wait for a court ruling to decide who approves access, who monitors exceptions, and who activates the incident response process. Legal uncertainty is a reason to strengthen governance, not postpone it.
An agent is not a role holder to whom accountability can be transferred. It is a system someone in the organization has chosen to authorize.
The claim that a tool was developed in someone’s spare time does not end the inquiry either. Which accounts, permissions, resources, and data did it use? Was the activity connected to the employee’s role? This does not mean an employer is liable in every case. It does mean organizations should define a clear experimentation policy before an incident occurs.
A prompt is an instruction, not an access control
An agent can receive a reasonable objective and choose an unreasonable way to achieve it. That is one implication of a nondeterministic system: checking its output is not enough. You also need to control the actions it can take.
An instruction such as act only within the law matters, but it cannot replace a technical restriction. A mechanism outside the model should check the target, action type, and authorization before execution. If the same model that plans an action also decides whether it is allowed, execution and oversight are not sufficiently separated.
- Access targets: Unrestricted browsing is a weak control; access should be limited to approved destinations.
- Permissions: A broadly privileged employee account is a weak control; use a dedicated identity with limited permissions.
- Sensitive actions: Model discretion is a weak control; apply appropriate policies and approval requirements.
- Monitoring: Reading conversations after the fact is a weak control; maintain action logs and alerts.
- Stopping: Ending a conversation is a weak control; cancel tasks and revoke permissions.
This is a basic test of control quality, not a complete technical specification. A real deployment must also account for redirects to other destinations, permissions inherited through connected services, retries, and actions already in a queue. A stop button that leaves queued work running can allow an incident to continue after the operator believes it has ended.
Keep a human in the loop without creating a bottleneck
Following the incident, Railway said it would prohibit non-manual model testing against external organizations. That kind of restriction can be an immediate containment measure. As a permanent operating model, however, requiring manual approval for every action can erase the operational advantage of agents.
The answer is risk-based oversight. Routine actions inside an enterprise system, with limited permissions and a recovery path, may be suitable for extensive automation. A new external destination, a permission change, a financial commitment, or a security test needs a different approval process.
The human role shifts from executing every step to supervising multiple processes. For that to work, exceptions must arrive with context: what the agent wanted to do, which system or party was involved, what authorization supported the action, and what impact was expected. A generic alert that requires someone to read an entire conversation is not an effective oversight mechanism.
- Define authority: Document the agent’s purpose, accountable owner, approved targets, and prohibited actions.
- Enforce controls outside the model: Restrict permissions and access through infrastructure components that do not depend on the model’s compliance.
- Test exceptions: In an authorized environment, examine how the agent handles an unapproved target, conflicting instructions, and a tool failure.
- Apply tiered oversight: Allow routine actions while routing sensitive actions to the appropriate human approver.
- Rehearse shutdown: Verify that tasks can be canceled, permissions revoked, and evidence preserved for investigation.
The order matters. Do not start with an agent that has broad access and add controls only after it delivers an impressive demo. Define the boundaries of its role first, then test how much autonomy it can safely exercise within them.
The real cost goes beyond the model bill
The business case must include maintenance, testing, supervision time, and exception handling. Unauthorized activity can add legal expenses, security investigation costs, process downtime, and damage to commercial relationships. These are risks to assess, not figures to invent in support of or opposition to a project.
Alongside labor hours saved, measure the time spent resolving exceptions, the share of actions requiring intervention, and the ability to reconstruct decisions. An agent that saves work in one department while creating constant emergency work in another is not necessarily an efficiency gain.
The organizational capability to build now
This incident is not a reason to stop adopting AI. It is a reason to stop treating agent deployment as an isolated technical skill. Sound design combines deep AI expertise, an understanding of the business process, security, legal input, and management experience. Relevant education and practical experience matter more here than an impressive demonstration.
Enterprises need to advance on two tracks: AI literacy that teaches employees to work effectively with models and recognize their limits, and internal infrastructure for building and managing agents. That infrastructure should include an agent registry, accountable owners, permissions, versions, monitoring, and a shutdown procedure. IT will need to manage not just software, but also the authority and lifecycle of a digital workforce.
Before approving the next agent to act outside the organization, ask a simple question: Can we show exactly who authorized it to do what, in which system, and under which conditions? If the answer exists only in the prompt, the project is not ready for autonomous operation.