Instance Segmentation of Cells and Nuclei Made Simple Using Deep Learning

By Dr. Mike Woerdemann - 15 March, 2022

Instance segmentation of nuclei and cells using deep learning

Instance segmentation is the task of splitting an image into individual instances of an object, such as individual cells or nuclei. While this task required post-processing work in the past, we updated our TruAI™ deep-learning technology to dramatically simplify instance segmentation.

In this post, we explain this easy instance segmentation method through various image examples. Read on to learn about its benefits for microscopy image analysis and the new capabilities it enables.

How Instance Segmentation Works in Microscopy Images

Automated analysis of microscopy images typically requires segmentation—the division of an image into its objects, parts, and background. Relatively easy segmentation applications include images with well-separated nuclei (see Figure 1).

Figure 1. BPAE cell sample with nuclei visualized through DAPI staining. Sample was acquired using the FLUOVIEW™ FV3000 confocal microscope at 20x magnification. The sample is partly bleached, resulting in different levels of fluorescence signal. A low integration time was chosen to show the capabilities of TruAI technology.

Even here, it makes sense to use TruAI technology to segment noisy nuclei and nuclei over a broad range of signal levels. TruAI technology can easily address this standard application (see Figure 2). The image is segmented into two classes: foreground and background. TruAI technology can detect up to 16 different foreground classes if needed.

Deep learning image segmentation of nuclei

Figure 2. The same image as in Figure 1 overlaid with the TruAI probability map, which indicates nuclei pixels in red.

Nuclei that are overlapping or touching each other may require some simple post-processing steps to separate them into individual nuclei. The reason is that the neural network detects all pixels as either nuclei or background, and there are no background pixels between touching nuclei to distinguish them as separate objects. The segmentation of individual nuclei is called instance segmentation.

Figure 3. Detailed view of Figure 1 (left) and Figure 2 (right). Segmentation of the nuclei pixels is very good, but touching nuclei must be separated using post-processing steps.

Segmentation is also useful for more complex analysis tasks, such as confluent cells (see Figure 4). For instance, you can use segmentation to measure an area covered by cells.

Figure 4. The same field of view as in Figure 1 but now also showing the fluorescence signal for 488 phalloidin marking the F-actin.

Deep learning image segmentation of F-action

Figure 5. The segmentation result of the F-actin channel with a standard segmentation neural network. Although the neural network was trained with 1 px borders between touching cells, the segmentation result can’t reliably separate individual cells.

However, splitting the segmented area of cells into individual cells (instance segmentation) can be a challenging post-processing task (see Figure 6).

Figure 6. Detailed view of Figure 4 (left) and Figure 5 (right). The borders between adjacent cells show a dip in probability for the foreground class, but a separation of individual cells would require sophisticated post-processing by an experienced user.

To simplify this workflow for images with a broad range of complexity, we have added an easy instance segmentation feature to our TruAI technology for the scanR screening station.*

The instance segmentation feature is AI-based and requires no manual post-processing steps or adjustment of parameters. Once a neural network model is trained, it can be applied to new images with a single click for immediate analysis results (see Figure 7).

Figure 7. Get the results for nuclei (red) and cell (yellow) detection in one click using TruAI instance segmentation. This simple instance segmentation method requires no post-processing steps.

Figure 8 below shows how TruAI instance segmentation directly delivers the instances of both cells and nuclei without further post-processing steps.

Figure 8. Using TruAI instance segmentation, the neural networks directly output the correct separation of the nuclei (top) and cells (bottom).

Figure 9 below compares the results of cell separation using standard segmentation vs. TruAI instance segmentation.

Cell separation results using standard segmentation vs. deep learning instance segmentation

Figure 9. Evolution from standard segmentation (top right) to TruAI instance segmentation (bottom left) when analyzing complex images (top left). The result (bottom right) is a nearly perfect separation of the cells.

AI-Based Segmentation: A Powerful Toolset for Microscopy Image Analysis

Our updated TruAI technology provides a powerful toolset to train individual neural network models for a broad range of image analysis applications. While training an optimal neural network requires carefully labeled examples (supported by our user-friendly software), applying the trained neural network to new images is an easy and robust task.

TruAI instance segmentation also enables us to deliver reliable pretrained deep-learning models for standard applications, such as nuclei or cell detection. Look out for an upcoming post to learn more about the pretrained models and see examples.

In the meantime, check out other instance segmentation applications: