Computer Vision: How computers see
TL;DR
Computer vision has evolved from manually-defined feature detection to ML-trained models that now anchor most FDA-cleared AI medical devices:
- Machine learning replaced hand-defined features with learned ones: Early computer vision required people to manually specify what to search for in an image, while modern ML-based models learn layered filters (“convolutions”) during training, producing more robust and reliable detection.
- Image classification and segmentation serve distinct medical purposes: Classification assigns probabilities to an entire image (useful for flagging radiographical scans), while segmentation assigns probabilities per pixel, enabling tasks like margin detection or isolating abnormal regions within a scan.
- Radiological imaging dominates FDA-cleared ML medical devices today: The vast majority of ML-enabled devices cleared by the FDA deal with medical imaging systems, and computer vision is expected to remain the field’s mainstay ML application even as other techniques expand into new disciplines.
Computer Vision and Medical Devices
Computer vision plays a big role in many aspects of everyday life. While subtle, it is used in a wide range of consumer, industrial, and medical products, including cell phone cameras, cars (not just the self-driving variety!), assembly lines, and the software a doctor may use to interpret medical scans.
Computer Vision Benefits
The primary benefit of computer vision is how it can boost the speed and efficiency of human users. The human visual cortex contains roughly 4-6 billion neurons, each much more expressive and interconnected than their digital counterparts. The cost of this incredible flexibility in interpretation is speed.
A human’s mean visual reaction time is roughly 250 milliseconds for simple reactive tasks. Any additional required interpretation can increase this time to seconds or minutes per item. Using computer vision to automate simple tasks (and to recognize what items need expert interpretation!) or to guide the user’s attention can allow individual practitioners to do the work of dozens of unassisted users.
Computer Vision Evolution
Computer vision as a practice has evolved massively since its inception. In the early era of the field, people would be required to define the features that would be searched for within an image. Detection of these human-defined “features” allows for automation in well-controlled conditions but lacks flexibility.
Improved computational power enables these features to be “learned” through training a computer vision model using machine learning (ML). This results in more subtle and robust feature detection which improves the reliability of the final model.
ML-based computer vision models generally harness many layers of “convolutions”, which are essentially sets of filters that it learns during training. These filters are usually much smaller than any of the objects the model is learning to detect. Combining many of them across different scales, an ML-trained computer vision model can reach a surprising level of robustness and accuracy across many different tasks, including image classification and segmentation.
A more recent development is multimodal AI, an improvement on top of language models, which allows the AI to incorporate information from input images and perform assistant-like tasks.
Computer Vision Applications
Classifying images is one of the primary tasks for computer vision models. While simple on its face, classifying images is a highly nuanced task that can span an incredible breadth of complexity depending on both the images and categories.
For image classification, which may be used to flag medical radiographical images, the output of the model is generally one or multiple probabilities, corresponding to each of the classes of image the model must recognize. Throughout the model, different filters are learned that characterize the differences between the classes, which contribute both positively and negatively to the outputs.
Image segmentation can be considered per-pixel image classification instead of class probabilities for the entire image. Each pixel or section of an image is given probabilities, which allow for complex shapes to be isolated from within a picture. In medical applications, this type of computer vision model could be used for margin detection, or to recognize abnormal sections of a medical scan, or even as part of a cell-counting pipeline for laboratory work.
Modern models, such as Meta’s Segment Anything 2, can even perform segmentation across videos by learning more complex features not only across an image’s spatial pixels, but also across the temporal dimension.
Computer Vision Future
While a mature field, computer vision is still constantly evolving and incorporating new techniques and technologies as they emerge. These models utilize the same feature-detection and classification principles as segmentation models. They also include the information with the text inputs to a language model, which can then respond based on information contained in the image inputs. A recent example of this is NVidia’s Describe Anything Model, which seamlessly blends images and language.
The vast majority of ML-enabled medical devices cleared by the FDA are radiological – dealing with medical imaging systems. While we can expect to see other ML techniques further utilized across other medical disciplines, computer vision will remain a mainstay application of ML in the sector.
Thor Tronrud, PhD, is a Machine Learning Scientist at StarFish Medical who specializes in the development and application of machine learning tools. Previously an astrophysicist working with magnetohydrodynamic simulations, Thor joined StarFish in 2021 and has applied machine learning techniques to problems including image segmentation, signal analysis, and language processing.
Image: StarFish Medical
Related Resources
Medtech leaders expect connected products to take over their portfolios within five years. Three years of actual FDA clearance filings show a real climb, just a much more gradual one.
Medical device data monetization turns connected therapy data into revenue from providers, payers, and pharma, not just device sales.
Connected health devices are multiplying fast. We examine the gap between remote monitoring’s promise and its reality.
Modern medical devices are no longer confined to hospital settings. Wearable cardiac monitors, home respiratory systems, and remote patient monitoring devices now operate within broader digital health networks.
Explore Our Services
Our multidisciplinary teams have all the in-house capabilities to transform your medical device from concept to final product.
Medical device design and development with a commercial launch focus. See how we support clinical trials and facilitate seamless tech transfers.
We help you navigate complex regulatory landscapes to achieve commercial success. Explore our integrated approach to quality and regulatory services.