Medical Sieve Radiology Grand Challenge - overview

With the growth in digital imaging such as MR, CT and ultrasound imaging, radiologists and cardiologists have to examine a large number of images a day. A typical emergency room radiologist may look at as many as 200 imaging study cases a day containing thousands of images, such as in lower body CT angiography where there can be as many as 3000 images per study. Cognitive fatigue is therefore a common problem facing radiologists and cardiologists. Moreover, due to the volume of imagery to examine, these specialists have little time left to assimilate all pertinent clinical information from patient records before they interpret the images. As a result, the current method of interpretation is largely based on imaging alone without the benefit of holistic clinical context leading to large diagnosis error rates particularly for coincidental diagnosis. 

Medical Sieve, an IBM Research Grand Challenge, led by Dr. Tanveer Syeda-Mahmood ( aims at developing a cognitive assistant system for radiologists and cardiologists that acts as a medical sieve filtering and quickly detecting anomalies using holistic clinical information derived from imaging, text and clinical data. As a grand challenge, its goal is to build a provably good cognitive clinical assistant with as much capability as an entry-level radiologist to assist radiologists with accuracy and efficiency leading to:

  • Significant innovation
  • Capture mindshare and position IBM
  • Launch of new business for IBM

Significant Innovation

Over the last few years the project has made great strides on all of these grounds resulting in over 135 peer-reviewed publications including 7 best paper awards at prestigious forums such as MICCAI and over 113 patents filed. Team members have launched awareness of machine learning-driven clinical decision support through a series of workshops held at MICCAI conference over the years which helped establish a strong computational foundation for this field. 

Launch of New Business

Progress in medical imaging research through Medical Sieve and other related projects led to the acquisition of Merge Healthcare by IBM and the subsequent formation of the Watson Health Imaging business. The concept of a cognitive assistant summarizing the patient record for providing the holistic clinical context for radiologists is now already generally available as the product IBM Watson Imaging Patient Synopsis. Medical Sieve technologies have also led to another retrospective healthcare informatics solution called IBM Watson Imaging Clinical Review that enables discrepancy detection in health records by discovering disease mentions in radiology reports and comparing them to problem lists and recorded diagnosis in electronic health record systems.

Capturing Mindshare

Medical Sieve researchers at IBM have led the wave of interest in deep learning in radiology through a grand demonstration of full cognitive capabilities of a visual reasoning engine shown through a collaboration with Radiological Society of North America (RSNA) called the Eyes of Watson in which over 3000 radiologists participated in 2016. The interest shown in this exhibition continues at RSNA through the machine learning showcase in subsequent years. The project will be conducting a new clinical study at RSNA in 2018 aimed at evaluating AI technologies in a radiology read setting through its RAD+AI exhibit at RSNA 2018.

Medical Sieve Research

Over the years, the project has addressed many modalities in cardiac imaging (Almaden Labs), breast imaging (Haifa Labs) and other specialties, including many firsts in :

  • Automatic detection of coronary stenosis in X-ray angiography
  • Automatic 17-segment cardiac MRI characterization
  • Automatic BIRAD estimation from mammography
  • Automatic detection of heart failure from cardiac MRI
  • Automatic detection of valvular diseases from echocardiograms
  • Automatic Agatston scoring from CTA
  • Automatic detection of traumatic brain injuries                              

Medical Sieve's philosophy in developing cognitive assistants is to do a systematic modeling of the radiologist's interpretation process by bringing together multiple facets of artificial intelligence ranging from multimodal image and text analytics, deep learning, clinical knowledge and clinical reasoning technologies. Attention is also given to the seamless integration of these technologies into the clinical workflow. This comprehensive approach to building practical systems founded on solid theoretical research of the team in the respective fields of AI differs significantly from work of other researchers who focus on one or more aspects of AI.

For more details on the project, please contact the Principal Investigator and Chief Scientist, Dr. Tanveer Syeda-Mahmood ( or Dr. Mehdi Moradi (

You can watch a demo of our Medical Sieve Cognitive Assistant Application here.