Trustworthy medical agents: a Chinese expert consensus on development, evaluation, and clinical implementation

Medical agents are evolving from information-generation tools into complex AI systems capable of perceiving clinical states, conducting multi-step reasoning and task planning, invoking external tools, and taking actions within predefined permissions. Accordingly, the central clinical question is no longer whether such systems can generate plausible answers, but whether a given system, under explicit permissions and effective human oversight, has sufficient evidence to act safely in real-world clinical workflows. Because errors may accumulate and amplify along the “information–recommendation–order–execution” pathway, evaluations centered on accuracy or examination-style benchmarks are insufficient to establish clinical trustworthiness. Drawing on current guidance from the World Health Organization and the International Medical Device Regulators Forum, reporting and evaluation standards such as SPIRIT-AI and CONSORT-AI, and recent clinical research, this paper proposes the Medical Agent Autonomy Matrix (MAAM) and the Medical Agent Assessment and Evidence Framework (MAAE). The evidence synthesis identifies three persistent disconnects: between offline capability and clinical benefit, between model performance and system safety, and between predeployment validation and whole-lifecycle assurance. MAAM–MAAE bridges these gaps by aligning four dimensions: risk level, action authority, evidence maturity, and intensity of human oversight. The framework anchors evaluation on clinical task success rate and the rate of unsafe actions with clinical consequences; organizes evidence across four domains—capability, safety, human–AI clinical interaction, and the system lifecycle; and establishes five sequential gates: clinical necessity, system reliability, simulated safety, clinical feasibility, and clinical effectiveness and governability. Twenty-one actionable consensus statements cover intended use, data and knowledge governance, system architecture, tools and permissions, red-teaming, clinical evaluation, institutional readiness, human oversight, patient protection, continuous monitoring, version changes, cybersecurity, and mapping to international standards. The core conclusion is that as the degree of autonomy, the proximity of actions to patients, and the severity of potential harm increase, evidence thresholds, human oversight, and regulatory scrutiny should be strengthened in parallel. At present, high-risk medical actions must not proceed without real-time authorization and revocable control by qualified health professionals.

 

Key Words: medical agents; trustworthy artificial intelligence; large language models; clinical evaluation; human–machine collaboration; risk stratification; lifecycle governance; patient safety