A peer-reviewed study published in the Asian Journal of Psychiatry has reported that ChatGPT-3.5 achieved a high level of diagnostic accuracy when applied to clinical psychiatric case vignettes.
The study is conducted by Professor Russell Franco D’Souza and colleagues, the study found that the AI model completed the diagnostic tasks without a single critical error, raising the possibility of its utility in supporting clinical decision-making.
Expert Evaluation Using Real-World Psychiatric Cases
Using 100 Cases in Psychiatry as a testing framework, researchers examined ChatGPT-3.5’s diagnostic abilities across a range of realistic patient profiles. The vignettes—each detailing symptoms, background, and diagnostic queries—were submitted sequentially to the model.
ChatGPT’s responses were then graded by two psychiatrists using an academic A-to-D scale without any retraining. The findings: 61 As, 31 Bs, and 8 Cs, with no responses deemed unacceptable.
Strongest Performance in Management Planning
ChatGPT-3.5 showed its strongest performance in outlining management strategies for psychiatric cases—covering treatment plans, patient follow-up, and psychosocial considerations, areas typically reserved for experienced clinicians. Diagnostic accuracy also ranked high, with the model consistently identifying key conditions and appropriately ruling out alternatives.
According to the study, there were zero diagnostic errors. While not every response earned top marks, all fell within acceptable parameters, indicating a reliable foundation of psychiatric knowledge and reasoning.
Responsible Integration Remains Key
The research team framed the study’s results within a wider conversation on responsible AI integration. “It is evident from our study that ChatGPT 3.5 has appreciable knowledge and interpretation skills in Psychiatry,” the authors stated.
They emphasised that moving from promise to practice requires defined protocols for reliability and ethics. The model is proposed as a potential clinical decision support tool—particularly relevant in psychiatry, where clinical interpretation is rarely straightforward.
Proven Capabilities Across Broader Medical Applications
ChatGPT’s performance in psychiatric diagnostics aligns with its growing track record in medical domains. The model has previously passed the United States Medical Licensing Examination (USMLE), often delivering insights comparable to those of human clinicians.
Since its launch in November 2022, ChatGPT attracted more than one million users within five days and has since become a tool for both consumers and professionals.
Its expanding role in healthcare research reflects a wider push to explore large language models in data-heavy clinical settings—particularly in psychiatry, where case analysis and dialogue are central.
Transparency Around Training Data Still Lacking
The authors flagged an important limitation: uncertainty around ChatGPT-3.5’s training inputs. With OpenAI yet to disclose whether 100 Cases in Psychiatry was part of the dataset, it remains unclear if the model had previous exposure.
This ambiguity complicates efforts to measure its performance on unfamiliar material. The study emphasises the need for follow-up testing with cases known to be outside the model’s training history.
Next Steps for AI Deployment in Psychiatric Care
The study positions ChatGPT as a potential contributor to psychiatric care, capable of interpreting detailed clinical scenarios and offering treatment direction. However, the authors caution that deployment in real-world settings must be informed by rigorous evidence and oversight.
They call for future research that expands clinical testing and builds decision-support infrastructure around verified AI outputs. AI could enhance clinical efficiency and influence the standards of psychiatric care with appropriate governance.