Background and Aims
Methods
Results
Conclusions
Abbreviations:
ADR (adenoma detection rate), AI (artificial intelligence), AMR (adenoma miss rate), ANN (artificial neural network), BE (Barrett’s esophagus), CAD (computer-aided diagnosis), CADe (CAD studies for colon polyp detection), CADx (CAD studies for colon polyp classification), CI (confidence interval), CNN (convolutional neural network), CRC (colorectal cancer), DL (deep learning), GI (gastroenterology), HDWL (high-definition white light), HD-WLE (high-definition white light endoscopy), ML (machine learning), NBI (narrow-band imaging), NPV (negative predictive value), PIVI (preservation and Incorporation of Valuable Endoscopic Innovations), SVM (support vector machine), VLE (volumetric laser endomicroscopy), WCE (wireless capsule endoscopy), WL (white light)Introduction



Term | Definition/Description |
---|---|
Artificial intelligence (AI) | Branch of computer science that develops machines to perform tasks that would usually require human intelligence |
Machine learning (ML) | Subfield of AI in which algorithms are trained to perform tasks by learning patterns from data rather than by explicit programming |
Representation learning (RL) | Subtype of ML in which algorithms learn the best features required to classify data on their own |
Deep learning (DL) | Type of RL in which algorithms learn a composition of features that reflect a hierarchy of structures in the data and provide detailed image classification output |
Deep reinforcement learning (DRL) | Technique combining DL and sequential learning to achieve a specific goal over several steps in a dynamic environment |
Training dataset | Dataset used to select the ideal parameters of a model after iterative adjustments |
Validation dataset | A (usually) distinct dataset used to test and adjust the parameters of a model |
Neural networks | Model of layers consisting of connected nodes broadly similar to neurons in a biological nervous system |
Support vector machine (SVM) | Classification technique that enables identification of an optimal separation plane between categories by receiving data inputs in a testing dataset and providing outputs that can be used in a separate validation dataset |
Recurrent neural networks | DL architecture for tasks involving sequential inputs such as speech or language and used for speech recognition and natural language processing and understanding (eg, predictive text suggestions for next words in a sequence) |
Convolutional neural networks (CNN) | DL architecture that adaptively learns hierarchies of features through back-propagation and is used for detection and recognition tasks in images (eg, face recognition) |
Computer-aided detection/diagnosis | Use of a computer algorithm to provide detection or a diagnosis of a specified object/region of interest |
Transfer learning | Ability of a trained CNN model to perform a separate task by using a relatively small dataset for the new task |
Procedure | Application |
---|---|
Colonoscopy | Detection of polyps (real time and on still images and video) |
Classification of polyps (neoplastic vs hyperplastic) | |
Detection of malignancy within polyps (depth of invasion on endocytoscopic images) | |
Presence of inflammation on endocytoscopic images | |
Wireless capsule endoscopy (WCE) | Lesion detection and classification (bleeding, ulcers, polyps) |
Assessment of intestinal motility | |
Celiac disease (assessment of villous atrophy, intestinal motility) | |
Improve efficiency of image review | |
Deletion of duplicate images and uninformative image frames (eg, images with debris) | |
Upper endoscopy | Identify anatomical location |
Diagnosis of Helicobacter pylori infection status | |
Gastric cancer detection and assessing depth of invasion | |
Esophageal squamous dysplasia | |
Detection and delineation of early dysplasia in Barrett’s esophagus | |
Real-time image segmentation in volumetric laser endomicroscopy (VLE) in Barrett’s esophagus | |
Endoscopic ultrasound (EUS) | Differentiation of pancreatic cancer from chronic pancreatitis and normal pancreas |
Differentiation of autoimmune pancreatitis from chronic pancreatitis | |
EUS elastography |
Applications in endoscopy
Colorectal polyps: detection, classification, and cancer prediction
Polyp detection
Polyp classification
Study | Design | Real time or delayed? | Lesion number (learning/validation) | Type of computer aided design | Imaging technology | Lesion size and type | Sensitivity/Specificity/Negative predictive value accuracy for neoplasia | Accuracy for surveillance interval |
---|---|---|---|---|---|---|---|---|
Takemura 2010 28 | Retrospective | Image analysis ex vivo. Not real time capable. | 72 polyps/134 polyps | Automated classification | Magnifying chromoendoscopy (Kudo pit pattern) | NR | NR/NR/NR/98.5% | NS |
No SA | ||||||||
Tischendorf 2010 29 | Post hoc analysis of prospective data | Image analysis ex vivo. Not real time capable. | 209 polyps/NS | Automated classification with SVM | Magnifying NBI | 8.1 mm avg (2-40 mm) | 90%/70%/NR | NS |
SA excluded | 85.3% | |||||||
Gross 2011 31 | Post hoc analysis of prospective data | Image analysis ex vivo. Not real time capable. | 434 polyps/NS | Automated classification with SVM | Magnifying NBI | 2-10 mm (SA; n = 2) | 95%/90.3/NR/93.1% | NS |
Takemura 2012 32 | Retrospective | Image analysis ex vivo | NR/371 polyps | Automated classification with SVM | Magnifying NBI | NR | 97.8%/97.9%/NR/97.8% | NS |
No SA | ||||||||
Kominami 2016 30 | Prospective | Real time analysis of ex vivo images | NR/118 polyps | Automated classification with SVM | Magnifying NBI | ≤5 mm: 88 >5 mm: 30 | For ≤5 mm:93%/93.3%/93%/93.2% | 92.7% |
SA excluded | ||||||||
Chen 2018 34 | Prospective validation | Image analysis ex vivo. | 2157/284 polyps | Automated classification with CNN | Magnifying NBI | SA excluded | 96.3%/78.1%/91.5%/90.1% | NS |
Real time capability. | ||||||||
Byrne 2019 33 | Prospective validation | Ex vivo video images. Real time capability (50 ms delay) | Test set: 125 videos | Automated classification with CNN | Near focus NBI | SA excluded | 98%/83%/97% | NS |
94% | ||||||||
Jin 2020 42 | Prospective validation | Image analysis ex vivo | 2150/300 | Automated classification with CNN | NBI | ≤5 mm:300 | 83.3%/91.7%/NR/86.7% | NS |
SA excluded | ||||||||
Mori 2015 37 | Retrospective | Ex vivo of still images | NR/176 polyps | Automated classification (type NS) | Endocytoscopy | ≤10 mm:176 | 92%/79.5%/NR/89.2% | NR |
SA excluded | ||||||||
Mori 2016 35 | Retrospective | Ex vivo of still images. Real time capability. | 6051/205 polyps | Automated classification with SVM | Endocytoscopy | ≤5 mm: 139 | 89%/88%/76%/89% | 96% |
6-10 mm: 66 | ||||||||
No SA | ||||||||
Misawa 2016 36 | Prospective | Ex vivo of still images | 979/100 | Automated classification with SVM | Endocytoscopy with NBI | Mean 8.6 ± 10.3 mm | 84.5%/97.6%/82%/90% | NR |
No SA | ||||||||
Mori 2018 13 | Prospective | Real time colonoscopy | NS/475 polyps | Automated classification with SVM | Endocytoscopy with NBI and MB | ≤5 mm: 475 | Rectosigmoid: NR/NR/96.4%/98.1% | NR |
No SA |
Detecting malignancy in colorectal polyps
Colonoscopy in inflammatory bowel disease
Improving quality and training in colonoscopy
Analysis of wireless capsule endoscopy images
EGD
Anatomical location and quality assessment
Diagnosis of Helicobacter pylori infection
Diagnosis of gastric cancer and premalignant gastric lesions
Evaluation of esophageal cancer and dysplasia
Analysis of EUS images
Areas for future research
Summary
Supplementary data
- Video 1
Colonoscopy with CADe (Skout, Iterative Scopes, Cambridge, Mass) real-time identification of a 3 mm ascending colon polyp as denoted by a bounding box.
- Video 2
The artificial intelligence system (combined CADe) automatically detects colonic polyps which are noted by the oval bounding box. The individual polyps are then interrogated with the endocytoscope and the AI system (CADx) characterizes the polyp as neoplastic or non-neoplastic with a probability level.
References
- Canadian Association of Radiologists white paper on artificial intelligence in radiology.Can Assoc Radiol J. 2018; 69: 120-135
- Deep learning.Nature. 2015; 521: 436-444
- Predicting the future - big data, machine learning, and clinical medicine.N Engl J Med. 2016; 375: 1216-1219
- Artificial intelligence in healthcare: past, present and future.Stroke Vasc Neurol. 2017; 2: 230-243
- Applications of artificial neural networks in medical science.Curr Clin Pharmacol. 2007; 2: 217-226
- Deep learning: a primer for radiologists.Radiographics. 2017; 37: 2113-2131
Jia Y, Shelhamer E, Donahue J, et al. Caffe: convolutional architecture for fast feature embedding. MM ’14: Proceedings of the 22nd ACM International Conference on Multimedia 2014:675-678.
- Deep learning.MIT Press, Cambridge, MA2016
- An introduction to deep reinforcement learning.Foundation and Trends in Machine Learning. 2018; 11: 219-354
- Machine learning in cardiovascular medicine: are we there yet?.Heart. 2018; 104: 1156-1164
- Adding artificial intelligence to gastrointestinal endoscopy.Lancet. 2020; 395: 485
- Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study.Gut. 2019; 68: 1813-1819
- Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study.Ann Intern Med. 2018; 169: 357-366
- Adenoma detection rate and risk of colorectal cancer and death.N Engl J Med. 2014; 370: 2541
- Adenoma miss rates associated with a 3-minute versus 6-minute colonoscopy withdrawal time: a prospective, randomized trial.Gastrointest Endosc. 2017; 85: 1273-1280
- Factors influencing the miss rate of polyps in a back-to-back colonoscopy study.Endoscopy. 2012; 44: 470-475
- Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps.Endoscopy. 2016; 48: 837-842
- Automated polyp detection in colonoscopy videos using shape and context information.IEEE Trans Med Imaging. 2016; 35: 630-644
- Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.Gastroenterology. 2018; 155: 1069-1078.e8
- Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial.Gastroenterology. 2020; 159: 512-520.e7
Wang P, Liu P, Glissen Brown JR, et al. Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study. Gastroenterology. Epub 2020 Jun 17.
- Use of endoscopic impression, artificial intelligence, and pathologist interpretation to resolve discrepancies between endoscopy and pathology analyses of diminutive colorectal polyps.Gastroenterology. 2020; 158: 783-785.e1
- The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on real-time endoscopic assessment of the histology of diminutive colorectal polyps.Gastrointest Endosc. 2011; 73: 419-422
- Real-time optical biopsy of colon polyps with narrow-band imaging in community practice does not yet meet key thresholds for clinical decisions.Gastroenterology. 2013; 144: 81-91
- New classification for probe-based confocal laser endomicroscopy in the colon.Endoscopy. 2011; 43: 1076-1081
- Real-time characterization of diminutive colorectal polyp histology using narrow-band imaging: implications for the resect and discard strategy.Gastroenterology. 2016; 150: 406-418
- ASGE Technology Committee systematic review and meta-analysis assessing the ASGE PIVI thresholds for adopting real-time endoscopic assessment of the histology of diminutive colorectal polyps.Gastrointest Endosc. 2015; 81: 502.e1-502.e16
- Quantitative analysis and development of a computer-aided system for identification of regular pit patterns of colorectal lesions.Gastrointest Endosc. 2010; 72: 1047-1051
- Computer-aided classification of colorectal polyps based on vascular patterns: a pilot study.Endoscopy. 2010; 42: 203-207
- Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy.Gastrointest Endosc. 2016; 83: 643-649
- Computer-based classification of small colorectal polyps by using narrow-band imaging with optical magnification.Gastrointest Endosc. 2011; 74: 1354-1359
- Computer-aided system for predicting the histology of colorectal tumors by using narrow-band imaging magnifying colonoscopy (with video).Gastrointest Endosc. 2012; 75: 179-185
- Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model.Gut. 2019; 68: 94-100
- Accurate classification of diminutive colorectal polyps using computer-aided analysis.Gastroenterology. 2018; 154: 568-575
- Impact of an automated system for endocytoscopic diagnosis of small colorectal lesions: an international web-based study.Endoscopy. 2016; 48: 1110-1118
- Characterization of colorectal lesions using a computer-aided diagnostic system for narrow-band imaging endocytoscopy.Gastroenterology. 2016; 150: 1531-1532.e3
- Novel computer-aided diagnostic system for colorectal lesions by using endocytoscopy (with videos).Gastrointest Endosc. 2015; 81: 621-629
- Automatic optical diagnosis of small colorectal lesions by laser-induced autofluorescence.Endoscopy. 2015; 47: 56-62
- Computer-aided diagnosis of neoplastic colorectal lesions using 'real-time' numerical color analysis during autofluorescence endoscopy.Eur J Gastroenterol Hepatol. 2013; 25: 488-494
- Efficacy of a novel auto-fluorescence imaging system with computer-assisted color analysis for assessment of colorectal lesions.World J Gastroenterol. 2013; 19: 7146-7153
- Software for automated classification of probe-based confocal laser endomicroscopy videos of colorectal polyps.World J Gastroenterol. 2012; 18: 5560-5569
- Improved accuracy in optical diagnosis of colorectal polyps using convolutional neural networks with visual explanations.Gastroenterology. 2020; 158: 2169-2179.e8
- Simultaneous detection and characterization of diminutive polyps with the use of artificial intelligence during colonoscopy.VideoGIE. 2019; 4: 7-10
Mori Y, Kudo SE, East JE, et al. Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video). Gastrointest Endosc. Epub 2020 Mar 30.
- Long-term outcomes after resection for submucosal invasive colorectal cancers.Gastroenterology. 2013; 144: 551-559
- A large-scale multicenter study of long-term outcomes after endoscopic resection for submucosal invasive colorectal cancer.Endoscopy. 2013; 45: 718-724
- Colorectal polypectomy and endoscopic mucosal resection (EMR): European Society of Gastrointestinal Endoscopy (ESGE) Clinical Guideline.Endoscopy. 2017; 49: 270-297
- Narrow band imaging, magnifying chromoendoscopy, and gross morphological features for the optical diagnosis of T1 colorectal cancer and deep submucosal invasion: a systematic review and meta-analysis.Am J Gastroenterol. 2017; 112: 54-64
- Accuracy of diagnosing invasive colorectal cancer using computer-aided endocytoscopy.Endoscopy. 2017; 49: 798-802
- Endoscopic diagnostic support system for cT1b colorectal cancer using deep learning.Oncology. 2019; 96: 44-50
- Fully automated diagnostic system with artificial intelligence using endocytoscopy to identify the presence of histologic inflammation associated with ulcerative colitis (with video).Gastrointest Endosc. 2019; 89: 408-415
Takenaka K, Ohtsuka K, Fujii T, et al. Development and validation of a deep neural network for accurate evaluation of endoscopic images from patients with ulcerative colitis. Gastroenterology. Epub 2020 Feb 12.
- Use of artificial intelligence-based analytics from live colonoscopies to optimize the quality of the colonoscopy examination in real time: proof of concept.Gastroenterology. 2020; 158: 1219-1221.e2
- A novel artificial intelligence system for the assessment of bowel preparation (with video).Gastrointest Endosc. 2020; 91: 428-435.e2
- Wireless capsule endoscopy.Gastrointest Endosc. 2013; 78: 805-815
- Detection of lesions during capsule endoscopy: physician performance is disappointing.Am J Gastroenterol. 2012; 107: 554-560
- Generic feature learning for wireless capsule endoscopy analysis.Comput Biol Med. 2016; 79: 163-172
- Software for enhanced video capsule endoscopy: challenges for essential progress.Nat Rev Gastroenterol Hepatol. 2015; 12: 172-186
- Computer-aided decision support systems for endoscopy in the gastrointestinal tract: a review.IEEE Rev Biomed Eng. 2011; 4: 73-88
- A deep convolutional neural network for bleeding detection in wireless capsule endoscopy images.Conf Proc IEEE Eng Med Biol Soc. 2016; 2016: 639-642
- Classification of functional bowel disorders by objective physiological criteria based on endoluminal image analysis.Am J Physiol Gastrointest Liver Physiol. 2015; 309: G413-G419
- CAD-CAP: a 25,000-image database serving the development of artificial intelligence for capsule endoscopy.Endosc Int Open. 2020; 8: E415-E420
- Bleeding frame and region detection in the wireless capsule endoscopy video.IEEE J Biomed Health Inform. 2016; 20: 624-630
- Computer-aided bleeding detection in WCE video.IEEE J Biomed Health Inform. 2014; 18: 636-642
- Deep learning for polyp recognition in wireless capsule endoscopy images.Med Phys. 2017; 44: 1379-1389
- A neural network algorithm for detection of GI angiectasia during small-bowel capsule endoscopy.Gastrointest Endosc. 2019; 89: 189-194
- Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network.Gastrointest Endosc. 2019; 89: 357-363.e2
- Computer-aided detection of small intestinal ulcer and erosion in wireless capsule endoscopy images.Phys Med Biol. 2018; 63: 165001
- Hookworm detection in wireless capsule endoscopy images with deep learning.IEEE Trans Image Process. 2018; 27: 2379-2392
- Gastroenterologist-level identification of small-bowel diseases and normal variants by capsule endoscopy using a deep-learning model.Gastroenterology. 2019; 157: 1044-1054.e5
- Automatic anatomical classification of esophagogastroduodenoscopy images using deep convolutional neural networks.Sci Rep. 2018; 8: 7497
- Randomised controlled trial of WISENSE, a real-time quality improving system for monitoring blind spots during esophagogastroduodenoscopy.Gut. 2019; 68: 2161-2169
- Comparing blind spots of unsedated ultrafine, sedated, and unsedated conventional gastroscopy with and without artificial intelligence: a prospective, single-blind, 3-parallel-group, randomized, single-center trial.Gastrointest Endosc. 2020; 91: 332-339.e3
- Application of convolutional neural networks in the diagnosis of helicobacter pylori infection based on endoscopic images.EBioMedicine. 2017; 25: 106-111
- Deep learning analyzes Helicobacter pylori infection by upper gastrointestinal endoscopy images.Endosc Int Open. 2018; 6: E139-E144
- High accuracy of convolutional neural network for evaluation of Helicobacter pylori infection based on endoscopic images: preliminary experience.Clin Transl Gastroenterol. 2019; 10: e00109
- Application of convolutional neural networks for evaluating Helicobacter pylori infection status on the basis of endoscopic images.Scand J Gastroenterol. 2019; 54: 158-163
- Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images.Gastric Cancer. 2018; 21: 653-660
- Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy.Gastrointest Endosc. 2019; 89: 806-815.e1
- Real-time gastric polyp detection using convolutional neural networks.PloS One. 2019; 14e0214133
- Deep-learning based detection of gastric precancerous conditions.Gut. 2020; 69: 4-6
- Diagnosing chronic atrophic gastritis by gastroscopy using artificial intelligence.Dig Liver Dis. 2020; 52: 566-572
- Real-time artificial intelligence for detection of upper gastrointestinal cancer by endoscopy: a multicentre, case-control, diagnostic study.Lancet Oncol. 2019; 20: 1645-1654
- Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks.Gastrointest Endosc. 2019; 89: 25-32
- Deep-learning system detects neoplasia in patients with Barrett's esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking.Gastroenterology. 2020; 158: 915-929.e4
- Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).Gastrointest Endosc. 2020; 91: 1242-1250
- Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).Gastrointest Endosc. 2020; 91: 1264-1271.e1
- Computer-aided detection of early Barrett's neoplasia using volumetric laser endomicroscopy.Gastrointest Endosc. 2017; 86: 839-846
- Endoscopic surveillance of Barrett's esophagus using volumetric laser endomicroscopy with artificial intelligence image enhancement.Gastroenterology. 2019; 157: 303-305
- Differentiation of pancreatic cancer and chronic pancreatitis using computer-aided diagnosis of endoscopic ultrasound (EUS) images: a diagnostic test.PloS One. 2013; 8: e63820
- Differential diagnosis of pancreatic cancer from normal tissue with digital imaging processing and pattern recognition based on a support vector machine of EUS images.Gastrointest Endosc. 2010; 72: 978-985
- Digital image analysis of EUS images accurately differentiates pancreatic cancer from chronic pancreatitis and normal tissue.Gastrointest Endosc. 2008; 67: 861-867
- Neural network analysis of EUS images to differentiate between pancreatic malignancy and pancreatitis.Gastrointest Endosc. 2001; 54: 625-629
- A new descriptor for computer-aided diagnosis of EUS imaging to distinguish autoimmune pancreatitis from chronic pancreatitis.Gastrointest Endosc. 2015; 82: 831-836.e1
- Neural network analysis of dynamic sequences of EUS elastography used for the differential diagnosis of chronic pancreatitis and pancreatic cancer.Gastrointest Endosc. 2008; 68: 1086-1094
- Diagnosis of autoimmune pancreatitis: the Mayo Clinic experience.Clin Gastroenterol Hepatol. 2006; 4: 1010-1016
- Endoscopic ultrasound elastography.Pancreatology. 2011; 11: 34-39
- Efficacy of an artificial neural network-based approach to endoscopic ultrasound elastography in diagnosis of focal pancreatic masses.Clin Gastroenterol Hepatol. 2012; 10: 84-90.e1
Article info
Publication history
Footnotes
Disclosure: Dr Pannala is a consultant for HCL Technologies and has received travel compensation from Boston Scientific Corporation. Dr Krishnan is a consultant for Olympus Medical. Dr Melson received an investigator-initiated grant from Boston Scientific and has stock options with Virgo Imaging. Dr Schulman is a consultant for Boston Scientific, Apollo Endosurgery, and MicroTech and has research funding from GI Dynamics. Dr Sullivan is a consultant and performs contracted research for Allurion Technologies, Aspire Bariatrics, Baronova, Obalon Therapeutics; is a consultant, performs contracted research, and has stock in Elira; is a consultant for USGI Medical, GI Dynamics, Phenomix Sciences Nitinotes, Spatz FGIA, and Endotools; and performs contracted research for Finch Therapeutics and ReBiotix. Dr Trikudanathan is a speaker and has received honorarium and travel from Boston Scientific, and is on the advisory board for Abbvie. Dr Trindade is a consultant for Olympus Corporation of the Americas and PENTAX of America, Inc., has received food and beverage from Boston Scientific, and has a research grant from NinePoint Medical, Inc. Dr Watson is a consultant and speaker for Apollo Endosurgery and Boston Scientific and a consultant for Medtronic and Neptune Medical Inc. Dr Lichtenstein is a consultant for Allergan Inc. and Augmenix; a speaker for Aries Pharmaceutical; a consultant and speaker for Gyrus ACMI, Inc., and Olympus Corporation of the Americas; and has received a tuition payment from Erbe USA Inc. All other authors disclosed no financial relationships.
Identification
Copyright
User license
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) |
Permitted
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
Not Permitted
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier's open access license policy