The increasing availability of large language models (LLMs) such as ChatGPT has accelerated their use in clinical settings, particularly for answering medical questions, supporting diagnosis, and assisting patient education. With the evolution toward multimodal versions such as ChatGPT-4, it has become possible to combine image analysis with text interpretation, bringing these systems closer to more comprehensive clinical support. However, while specialized deep learning models such as CNNs have already demonstrated high accuracy in image analysis in gastroenterology, the performance of general-purpose LLMs in this domain remains underexplored and still requires robust validation.

Home / Publications / Publication

Home / Publications / Publication

Gráficos ilustrativos
Image reproduced from the article.

Publication Type: Paper Abstract
Original title: Evaluating ChatGPT-4 for the Interpretation of Images from Several Diagnostic Techniques in Gastroenterology
Article publication date: January 2025
Source: Journal of Clinical Medicine (MDPI)
Author(s): Miguel Mascarenhas Saraiva; Tiago Ribeiro; Belén Agudo; João Afonso; Francisco Mendes; Miguel Martins; Pedro Cardoso; Joana Mota; Maria João Almeida; António Costa; Mariano Gonzalez Haba Ruiz; Jessica Widmer; Eduardo Moura; Ahsan Javed; Thiago Manzione; Sidney Nadal; Luis F. Barroso; Vincent de Parades; João Ferreira; Guilherme Macedo

What is the goal, target audience, and areas of digital health it addresses?
     The study aims to systematically evaluate ChatGPT-4’s performance in the automatic interpretation of medical images from multiple diagnostic techniques in gastroenterology. The target audience includes gastroenterologists, clinicians, researchers in artificial intelligence applied to healthcare, and decision-makers involved in digital health transformation. The publication is mainly positioned in the areas of generative and multimodal artificial intelligence, medical image analysis, clinical decision support systems, and integration of clinical and imaging data.

What is the context?
     The increasing availability of large language models (LLMs) such as ChatGPT has accelerated their use in clinical settings, particularly for answering medical questions, supporting diagnosis, and assisting patient education. With the evolution toward multimodal versions such as ChatGPT-4, it has become possible to combine image analysis with text interpretation, bringing these systems closer to more comprehensive clinical support. However, while specialized deep learning models such as CNNs have already demonstrated high accuracy in image analysis in gastroenterology, the performance of general-purpose LLMs in this domain remains underexplored and still requires robust validation.

What are the current approaches?
     The current dominant approaches in medical image analysis rely on convolutional neural networks (CNNs) and other specialized deep learning models trained on large annotated clinical datasets. These models have achieved strong results in lesion detection, endoscopy, pancreatic diagnosis, and cancer detection. In contrast, multimodal LLMs such as ChatGPT-4 combine text and visual interpretation but were not originally designed as dedicated vision models. Their performance may therefore depend on integration with external vision components, which can limit effectiveness in highly specialized clinical tasks.

What is the innovation and how was its impact evaluated?
     The main innovation of the study lies in evaluating a general-purpose model, ChatGPT-4, in tasks traditionally dominated by specialized models. The system was tested on 740 clinical images from five techniques: capsule endoscopy, device-assisted enteroscopy, endoscopic ultrasound (EUS), digital single-operator cholangioscopy (DSOC), and high-resolution anoscopy (HRA). Each image was analyzed using structured prompts, and the outputs were compared with the reference diagnosis used as the gold standard. The study evaluated impact through standard clinical performance metrics, including accuracy, sensitivity, specificity, positive predictive value and negative predictive value (PPV/NPV), as well as AUC.

What are the main results, conclusions, and future implications?
     The results show variable performance across techniques. In capsule endoscopy, accuracy ranged from 50% to 90%, with relevant differences across anatomical locations. In device-assisted enteroscopy, overall accuracy was 67%. In endoscopic ultrasound, accuracy ranged from approximately 40% to 55%, revealing limited ability to differentiate pancreatic lesions. In digital single-operator cholangioscopy, accuracy was about 55% for malignancy detection. In high-resolution anoscopy, accuracy ranged from 47.5% to 67.5%, depending on the stage of the examination.

     Overall, performance was suboptimal and inconsistent, remaining below that of specialized models such as CNNs, which can achieve accuracies above 90% to 99% in similar tasks. The study highlights a significant risk of diagnostic error, including false positives and false negatives. In the short term, direct clinical use is not recommended, and its utility as a standalone tool remains limited. In the medium to long term, there is potential for a hybrid support model combining text and images, but substantial improvements in the visual component will be required. The main identified risks include incorrect diagnoses with negative clinical impact and a possible loss of trust among professionals and patients.

Do you have an innovative idea in healthcare field?

Share it with us and see it come to life.
We will help bring your projects to life!

Newsletter

Receive the latest updates from the InovarSaúde portal.

República Portuguesa logo
logotipo SNS
SPMS logotipo

Follow Us

YouTube
LinkedIn

Co-funded by

PRR Logotipo
república Portuguesa logo
União Europeia Logo

Newsletter

Receive the latest updates from the InovarSaúde portal.

República Portuguesa logo
SNS Logo
SPMS Logo

Follow Us

Co-funded by

PRR Logotipo
República Portuguesa logo
União Europeia Logo

Home / Publications / Publication

Gráficos ilustrativos
Image reproduced from the article.

Publication Type: Paper Abstract
Original title: Evaluating ChatGPT-4 for the Interpretation of Images from Several Diagnostic Techniques in Gastroenterology
Article publication date: January 2025
Source: Journal of Clinical Medicine (MDPI)
Author(s): Miguel Mascarenhas Saraiva; Tiago Ribeiro; Belén Agudo; João Afonso; Francisco Mendes; Miguel Martins; Pedro Cardoso; Joana Mota; Maria João Almeida; António Costa; Mariano Gonzalez Haba Ruiz; Jessica Widmer; Eduardo Moura; Ahsan Javed; Thiago Manzione; Sidney Nadal; Luis F. Barroso; Vincent de Parades; João Ferreira; Guilherme Macedo

What is the goal, target audience, and areas of digital health it addresses?
     The study aims to systematically evaluate ChatGPT-4’s performance in the automatic interpretation of medical images from multiple diagnostic techniques in gastroenterology. The target audience includes gastroenterologists, clinicians, researchers in artificial intelligence applied to healthcare, and decision-makers involved in digital health transformation. The publication is mainly positioned in the areas of generative and multimodal artificial intelligence, medical image analysis, clinical decision support systems, and integration of clinical and imaging data.

What is the context?
     The increasing availability of large language models (LLMs) such as ChatGPT has accelerated their use in clinical settings, particularly for answering medical questions, supporting diagnosis, and assisting patient education. With the evolution toward multimodal versions such as ChatGPT-4, it has become possible to combine image analysis with text interpretation, bringing these systems closer to more comprehensive clinical support. However, while specialized deep learning models such as CNNs have already demonstrated high accuracy in image analysis in gastroenterology, the performance of general-purpose LLMs in this domain remains underexplored and still requires robust validation.

What are the current approaches?
     The current dominant approaches in medical image analysis rely on convolutional neural networks (CNNs) and other specialized deep learning models trained on large annotated clinical datasets. These models have achieved strong results in lesion detection, endoscopy, pancreatic diagnosis, and cancer detection. In contrast, multimodal LLMs such as ChatGPT-4 combine text and visual interpretation but were not originally designed as dedicated vision models. Their performance may therefore depend on integration with external vision components, which can limit effectiveness in highly specialized clinical tasks.

What is the innovation and how was its impact evaluated?
     The main innovation of the study lies in evaluating a general-purpose model, ChatGPT-4, in tasks traditionally dominated by specialized models. The system was tested on 740 clinical images from five techniques: capsule endoscopy, device-assisted enteroscopy, endoscopic ultrasound (EUS), digital single-operator cholangioscopy (DSOC), and high-resolution anoscopy (HRA). Each image was analyzed using structured prompts, and the outputs were compared with the reference diagnosis used as the gold standard. The study evaluated impact through standard clinical performance metrics, including accuracy, sensitivity, specificity, positive predictive value and negative predictive value (PPV/NPV), as well as AUC.

What are the main results, conclusions, and future implications?
     The results show variable performance across techniques. In capsule endoscopy, accuracy ranged from 50% to 90%, with relevant differences across anatomical locations. In device-assisted enteroscopy, overall accuracy was 67%. In endoscopic ultrasound, accuracy ranged from approximately 40% to 55%, revealing limited ability to differentiate pancreatic lesions. In digital single-operator cholangioscopy, accuracy was about 55% for malignancy detection. In high-resolution anoscopy, accuracy ranged from 47.5% to 67.5%, depending on the stage of the examination.

     Overall, performance was suboptimal and inconsistent, remaining below that of specialized models such as CNNs, which can achieve accuracies above 90% to 99% in similar tasks. The study highlights a significant risk of diagnostic error, including false positives and false negatives. In the short term, direct clinical use is not recommended, and its utility as a standalone tool remains limited. In the medium to long term, there is potential for a hybrid support model combining text and images, but substantial improvements in the visual component will be required. The main identified risks include incorrect diagnoses with negative clinical impact and a possible loss of trust among professionals and patients.

Cadeira de rodas

Autonomous Patient Mobility in a Hospital Environment

The internal transport of patients in healthcare institutions, although at first glance it may seem like a simple task, represents a complex, continuous, demanding and time-consuming logistical operation that cuts across all levels of the…

Read more
Voz em IA

The Future of Diagnostics: Speech and AI

Speech is a biomarker that reflects, in a sensitive way, the integrated functioning of several physiological systems, namely the nervous, respiratory, and muscular systems. This complexity makes it a promising resource for detecting changes associated…

Read more
Literatura sobre os cuidados de saúde no futuro

What Literature Reveals About Healthcare in the Future

The healthcare sector is undergoing rapid transformation driven by population aging, increasing complexity of care, and digital advancements, in a context that requires greater integration, sustainability and adaptation to new realities such as the European…

Read more
Perturbação do sono

A Digital Intervention for Insomnia in Oncology

Insomnia is a sleep disorder characterised by persistent difficulties in initiating sleep, maintaining sleep during the night, or achieving restful sleep. These difficulties arise even in the presence of adequate sleeping conditions and are often…

Read more
Sistema de telemonitorização remota

Digital Technology Revolutionising Post-cardiac Surgery

According to the World Health Organisation, cardiovascular disease remains the leading cause of death worldwide, responsible for around 17.9 million deaths a year. Its high prevalence is associated with unhealthy lifestyles characterised by poor diet,…

Read more
Sistema robótico autónomo INSIDE

Autonomous Robotics System for Autism Therapy

Autism spectrum disorder is a neurodevelopmental condition with significant clinical, social and economic repercussions throughout life. According to the World Health Organization, it is estimated to affect approximately 1 in 160 children worldwide. Its origin…

Read more
Enfermeira com um telefone

Mobile Application to Improve Workflows in Nursing Homes

Portugal has one of the highest aging populations in the world, placing increasing pressure on elderly care services, especially in nursing homes. Healthcare professionals in these facilities are often overwhelmed due to the increasing number…

Read more
troca de informações de saúde e interoperabilidade

New Era of Interoperability in Healthcare Systems

The growing use of electronic health records, digital diagnostic systems and remote monitoring technologies has led to a significant increase in the volume and complexity of health data. This increase intensifies the need for continuous,…

Read more
robótica colaborativa

Collaborative Robotics Improves Working Conditions

Workers face growing challenges in the industrial environment. Among the most critical are fatigue and inappropriate postures, often associated with repetitive tasks and working conditions that lack ergonomic suitability. These factors represent significant risks for…

Read more
Benefícios da Eletrônica Médica

Detection of Anxiety and Panic Attacks in Real Time

The growing number of people with anxiety disorders, along with increased awareness of mental health, drives the need for new technological tools that provide remote and continuous monitoring of anxiety and panic disorders. Thus, the…

Read more
tele-ecografia

A Novel Approach for Robotic-assisted Tele-echography

Currently, robotic systems for ultrasound diagnostic procedures fall into two main categories: portable robots that require manual positioning and fully autonomous robotic systems that independently control the ultrasound probe’s orientation and positioning. Portable robots rely…

Read more
Personalização e tecnologia na gestão da Diabetes

Personalization and Technology in Diabetes Management

IPDM has significant potential to improve diabetes management and drive health system reforms to become high-performing, effective, equitable, accessible, and sustainable. Evidence and good practices inspire health system transformation. Adopting person-centred approaches like co-creation and…

Read more
TEF-HEALTH Logo

SPMS Integrates the TEF-Health Initiative

SPMS participates in the TEF-Health initiative as a partner in a consortium composed of 51 entities from 9 European Union countries. This action is co-financed by the European Commission and has a duration of five…

Read more
Global Digital Health Partnership Logo

SPMS Represents Portugal as Vice-president of GDHP

The GDHP is an intergovernmental organization in the digital health sector that facilitates cooperation and collaboration between government representatives and the World Health Organization (WHO). Its purpose is to foster policymaking that promote the digitalization…

Read more
Portugal INCoDe.2030

Digital Transformation of Health at INCoDe.2030 in Tomar

The “National Digital Skills Initiative e.2030, Portugal” (INCoDe.2030) is an initiative that aims to improve the Portuguese population’s level of digital skills, placing Portugal at the level of the most advanced European countries in this…

Read more
HealthData@PT Logo

HealthData@PT: New SPMS Initiative for Health Data

Action HealthData@PT is launched in the context of the implementation of the European Health Data Space, and is an initiative approved by the European Commission under the EU4Health 2021-2027 programme. This initiative contributes to the…

Read more

Do you have an innovative idea in healthcare field?

Share it with us and see it come to life.
We will help bring your projects to life!

Newsletter

Receive the latest updates from the Inovarsaúde portal.

República Portuguesa logo
SNS Logo
SPMS Logo

Follow Us

YouTube
LinkedIn

Co-funded by

PRR Logotipo
República Portuguesa logo
União Europeia Logo
Scroll to Top