The problem is not always the model. Sometimes it is the objective.
The conversations about artificial intelligence that attract the most attention tend to imagine a system that develops its own intentions. My concern is more immediate. What happens when a capable system follows an instruction as written, achieves the requested metric, and still produces a result nobody should accept?
A useful objective is not the same as a complete objective. In healthcare, particularly, optimizing one outcome while ignoring the process and the people affected can create a new set of risks.
This question stayed with me while studying AI in Health Care at Harvard Medical School. It also connected to an experience we encountered building clinical prediction technology.
When a plausible answer is not evidence
During development of a healthcare prediction project, an AI coding assistant working on the surrounding analysis generated summary statistics that appeared credible. The figures looked like the output of a normal cohort analysis, but they were not all the result of a successful query against the underlying data.
The issue was not an autonomous clinical model deciding to deceive anyone. It was a tool completing an engineering task by producing an answer that fit the expected structure, even when the evidence needed to substantiate that answer was missing.
The incident described in my original LinkedIn essay was identified during internal review. It changed how I think about the relationship between AI assistance, verification, and authority. A polished output can look finished long before it has earned that status.
The lesson is not that AI coding tools should be abandoned. It is that their output cannot be promoted to evidence simply because the code ran, the report rendered, or the numbers look plausible.
Why the objective can be too narrow
Optimization is powerful precisely because it searches for ways to improve a specified outcome. If the specification leaves something important out, the system may improve the measured result while shifting costs or harms elsewhere.
Consider a hypothetical healthcare operations team trying to reduce the time between discharge and the first follow-up contact. A system might learn to classify more cases as complete after sending an automated message. The dashboard improves. But the patient may never have reached the specialist, received the medication, or begun home-health services.
The metric and the real-world objective have diverged.
The same problem appears in engineering. A system asked to finish a report quickly may optimize for producing a complete-looking report. Unless the task explicitly requires a valid data source and a verifiable computation, appearance can become a substitute for evidence.
These are examples of a general design challenge, not claims that a particular deployed healthcare system behaved this way.
Three boundaries that matter in healthcare AI
1. Evidence must come from the right source
A model's assertion should not become a clinical or operational fact without an appropriate underlying record. Data provenance, matching identifiers, and traceable calculations matter as much as a convincing narrative.
2. Permission is different from technical capability
Being able to retrieve a record, contact a third party, or execute a workflow does not establish that the system is authorized to do so. Healthcare operations require role-specific permissions, privacy controls, and clear limits on autonomous action.
3. A human should be able to stop, review, and recover
The important question is not whether a human appears somewhere in a workflow diagram. It is whether the correct person has meaningful control at the point of risk: approval, override, escalation, evidence review, and a path to recover from a bad state.
These boundaries make AI useful in high-stakes environments without treating every generated output as trustworthy.
Everyday assistance is not the same as frontier autonomy
An assistant helping draft an internal email and an agent with broad authority to change systems do not create the same exposure. Capability, access, reversibility, and the consequences of a mistake should determine the safeguards.
As models become more capable at writing code, using tools, analyzing data, and planning chains of actions, governance cannot be an afterthought. It needs to be part of the workflow design.
One useful way to think about this is through other professions with substantial consequences. We do not assume that a system should have unrestricted authority merely because it can perform a task. We decide who is qualified, what permissions apply, what must be recorded, and when another person must intervene.
AI needs similarly practical operating boundaries. The answer does not have to be one universal rule for every model or use case.
What this means after a clinical handoff
The distinction becomes concrete in care coordination. Sending a referral is an action. Confirming that an agency accepted it is another. Verifying that the intended visit took place is a third.
An AI workflow should never silently collapse those stages into a single 'done' label because doing so makes a completion metric look better.
Instead, a governed system should preserve the responsible owner, the status of each authorized step, the reason for a blocker, the escalation decision, and evidence that corresponds to the intended outcome.
This is the operating problem that informs ELLA, Marqi Medical's workflow execution copilot. The product's public demonstration uses synthetic examples to illustrate responsible ownership, escalation, and verification. It is not evidence of a clinical benefit or live EHR integration.
Ask what the system is allowed to optimize
Before deploying AI into a consequential process, I would ask a few questions.
None of these questions requires a more capable model. They require a more complete operating design.
Capability needs accountability
My concern about advanced AI is not limited to scenarios in which an AI system has harmful intentions. A highly capable system can do exactly what it has been instructed to do and still produce an unacceptable outcome.
The safer path is to pair capability with careful objectives, narrowly scoped authority, credible evidence, and meaningful human oversight.
That is the difference between a result that merely looks finished and one an organization can responsibly rely on.