1 00:00:01,08 --> 00:00:05,00 In this third video in the series on cellSens software’s deep-learning function, 2 00:00:05,00 --> 00:00:10,20 we’ll explain how to manually label images to prepare the deep-learning training data. 3 00:00:10,20 --> 00:00:14,20 Once trained, the TruAI deep-learning technology can automatically 4 00:00:14,20 --> 00:00:18,06 analyze and detect objects in tissue images. 5 00:00:18,06 --> 00:00:23,09 This is a stitched image of mouse kidney. It has green and red channels. 6 00:00:23,09 --> 00:00:27,01 As you can see, there are numerous glomeruli in the kidney 7 00:00:27,01 --> 00:00:31,10 Since the shapes of the glomeruli are complex, they are impossible to detect 8 00:00:31,10 --> 00:00:37:08 using conventional threshold-based object detection relying on fluorescence intensity. 9 00:00:37,08 --> 00:00:43,08 To use the TruAI deep-learning detection, first we need to prepare the training data. 10 00:00:43,08 --> 00:00:50,09 In the count and measure layout tab, go to the training labels tool window. 11 00:00:50,09 --> 00:01:01,12 Click the new training label class button and change the name to glomeruli. 12 00:01:01,12 --> 00:01:06,17 Repeat that step to make another label named BG for background. 13 00:01:06,17 --> 00:01:09,09 Since this image has green and red channels, 14 00:01:09,09 --> 00:01:15,24 we should change the label colors to improve their visibility. 15 00:01:15,24 --> 00:01:13,01 In many cases, the standard network is OK. 16 00:01:13,01 --> 00:01:20,10 In this case, we will use blue and magenta. 17 00:01:20,10 --> 00:01:24,02 Then check the background checkbox for the BG label. 18 00:01:24,02 --> 00:01:29,23 With the BG label still selected, click the create training labels-fill button. 19 00:01:29,23 --> 00:01:32,14 Now you need to define the background. 20 00:01:32,14 --> 00:01:36,18 You can do this manually on the image using the drawing tool. 21 00:01:36,18 --> 00:01:40,11 You will want to select several different patterns when defining the background, 22 00:01:40,11 --> 00:01:44,04 so the annotated areas should include the center and edge of the kidney 23 00:01:44,04 --> 00:01:46,06 and the renal cortex, and so on. 24 00:01:46,06 --> 00:01:50,21 This is because we are attempting to identify glomeruli within the tissue, 25 00:01:50,21 --> 00:01:56,04 so the tissue pattern is the background rather than the black background in the image. 26 00:01:56,04 --> 00:01:59,10 Then, select the glomeruli label and paint the glomeruli 27 00:01:59,10 --> 00:02:02,18 in each background area using the brush tool. 28 00:02:02,18 --> 00:02:07,11 Enlarging the brush slightly may make it easier to label the glomeruli. 29 00:02:07,11 --> 00:02:16,20 Using a pen tablet rather than a mouse can improve the efficiency of this step. 30 00:02:16,20 --> 00:02:19,21 All glomeruli in the background area must be labeled 31 00:02:19,21 --> 00:02:23,19 to produce accurate training data for the deep-learning detection. 32 00:02:35,07 --> 00:02:39,18 Click the eye icon in the visible column to show and hide the background label 33 00:02:39,18 --> 00:02:48,01 to ensure that no glomeruli are missed. 34 00:02:48,01 --> 00:02:52,20 The image outside these labeled areas is not used for the deep-learning training phase 35 00:02:52,20 --> 00:02:57,17 so only the glomeruli that are inside the background area need to be annotated. 36 00:02:57,17 --> 00:03:03,17 Move to the next background area. 37 00:03:03,17 --> 00:03:08,09 This one doesn’t have any glomeruli so you can leave it as is. 38 00:03:08,09 --> 00:03:13,22 Check the next area, and again label any glomeruli. 39 00:03:24,07 --> 00:03:28,12 There is one glomerulus that is located on the border of the background area, 40 00:03:28,12 --> 00:03:32,21 so we need to modify the border using the eraser tool. 41 00:03:42,14 --> 00:03:46,23 Repeat this procedure for all the other background labeled areas. 42 00:03:46,23 --> 00:03:53,05 As you can see, a total of 45 glomeruli have been labeled in our seven background areas. 43 00:03:53,05 --> 00:03:56,07 At this point, we need to save the data. 44 00:03:56,07 --> 00:04:02,12 Click new training to start the deep-learning training phase. 45 00:04:02,12 --> 00:04:05,12 Enter a name for this new training. 46 00:04:05,12 --> 00:04:09,13 The image used for the training phase is already selected. 47 00:04:09,13 --> 00:04:11,22 Since the training image has two channels, 48 00:04:11,22 --> 00:04:16,00 you should click the plus icon to select two input channels. 49 00:04:16,00 --> 00:04:21,21 This additional information will help improve the accuracy of the trained neural network. 50 00:04:21,21 --> 00:04:25,15 The training labels that we created are already selected. 51 00:04:25,15 --> 00:04:29,15 Make sure that the background checkbox is checked for the BG label. 52 00:04:29,15 --> 00:04:32,11 Click next, and then start. 53 00:04:32,11 --> 00:04:35,14 The training of the neural network has begun. 54 00:04:35,14 --> 00:04:46,13 Refer to the second part of this video series for more detail on the training phase. 55 00:04:46,13 --> 00:04:48,11 The training is now complete. 56 00:04:48,11 --> 00:04:53,01 Verify the validation images of the checkpoint that has the highest similarity value 57 00:04:53,01 --> 00:04:57,07 to ensure that the glomeruli are properly detected in blue. 58 00:04:57,07 --> 00:04:59,14 If they are not all properly detected, 59 00:04:59,14 --> 00:05:11,22 then verify another checkpoint to find the one with the best validation images. 60 00:05:11,22 --> 00:05:15,22 In this case, the validation images at checkpoint 5 are good. 61 00:05:15,22 --> 00:05:22,22 Save it for the next step. 62 00:05:22,22 --> 00:05:28,22 Go to the count and measure layout tab and apply the trained neural network. 63 00:05:41,06 --> 00:05:53,21 It takes a few minutes to process since the image is so large. 64 00:05:53,21 --> 00:06:05,21 The inference results are shown on a probability map layer. 65 00:06:05,21 --> 00:06:09,08 Zoom in on the image to check the accuracy. 66 00:06:09,08 --> 00:06:13,09 We can see that the glomeruli outside the labeled area were detected 67 00:06:13,09 --> 00:06:15,18 and are now labeled in blue. 68 00:06:15,18 --> 00:06:19,10 We need to verify that these glomeruli, which were not hand labeled, 69 00:06:19,10 --> 00:06:23,17 are properly detected to validate the accuracy of the inference results. 70 00:06:23,17 --> 00:06:26,10 If the inference results are not accurate enough, 71 00:06:26,10 --> 00:06:30,10 try increasing the number of background areas and hand-labeled glomeruli 72 00:06:30,10 --> 00:06:33,23 when preparing the data for the training phase. 73 00:06:33,23 --> 00:06:38,01 Once properly trained, cellSens software’s deep-learning technology 74 00:06:38,01 --> 00:06:43,07 can improve the efficiency of your image analysis while reducing your workload. 75 00:06:43,07 --> 00:06:45,23 If you’d like to learn more about cellSens software 76 00:06:45,23 --> 00:06:49,03 and Olympus’ TruAI deep-learning technology, 77 00:06:49,03 --> 00:06:53,03 visit olympus-lifescience.com.