# EasySegment Supervised To create an **EasySegment Supervised** tool: * API: * To create an **EasySegment Supervised** project in **Deep Learning Studio**: 1. Start **Deep Learning Studio**. 2. Create a new project and select `EasySegment Supervised` in the `New deep learning project` dialog. ![image](assets/images/easydeeplearning/easysegment_4.webp) ## Properties ![image](assets/images/easydeeplearning/easysegment_5.webp) The supervised segmenter tool has 5 parameters: 1. The `Capacity` of the neural network (default: `Normal`) represents the quantity of information it is capable of learning. A larger capacity makes the tool slower. In the API: * The capacity is represented by the enumerate type [`ESupervisedSegmenterCapacity`](../../../doxygen_cpp/namespaceeVision_1_1EasyDeepLearning.html#a1f13ba56908e5bcc675a2b84f7c197ca) [.NET](../../../doxygen_cs/namespaceeVision_1_1EasyDeepLearning.html#ab0d58f35eb7a3914a67df69362ed6473) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterCapacity.html). * [`SetCapacity`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a234d36fbde19d092a242525b244b37ef) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a7255705dbe72d5ef712d757ee29ae2b6) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#ac2b307162f4d6babed66765153c447f9) sets the capacity of the tool. 2. The `Image type` (default: `Monochrome (1 channel)`): In the API: * To use monochrome (grayscale, 1 channel) images, set [`SetForceGrayscale`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#ae2b292d4e08a23dc94e34faef0ba4172) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a4bd496970b81bf534f7578243fd0d330) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#ae2b4c289857121b51dabd62a4f2107ab) to `true`. * To use color (3 channels) images, set [`SetForceGrayscale`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EUnsupervisedSegmenter.html#a695437c832162d1bcead1701b7b4bb20) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EUnsupervisedSegmenter.html#a0d0734f94def8070b908916dcdf2c5d3) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EUnsupervisedSegmenter.html#af0b943a353b64003627f244fbe6359ce) to `false`. 3. The `Sampling density` ([`SetSamplingDensity`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#aba29457bc8da07bcf5470e00d3755f6d) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a753e740db07c711f967a68aaace9892b) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#abed343bfa96c78dc877139491c2605dc)) is the parameter of the sliding window algorithm used to process whole images using patches of size ([`SetPatchSize`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a2c129297f1cea21c0895aeaffb4e0b5c) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#af556d5082b21aa3a84708a729c1a4dd4) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#ac591325880f17afc221194233ba329cb)). * It indicates how much overlap there is between the image patches: 100 - (100 / `SamplingDensity`) (%) * In practice, the stride between 2 consecutive patches is: `PatchSize` / `SampleDensity` (pixels) 4. The `Patch size` ([`SetPatchSize`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a2c129297f1cea21c0895aeaffb4e0b5c) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#af556d5082b21aa3a84708a729c1a4dd4) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#ac591325880f17afc221194233ba329cb)) is the size of the patches processed by the neural network. * By default, the patch size is determined automatically from the images in the training dataset. * You can also select the resolution of the patch size from the drop down list. ![image](assets/images/easydeeplearning/easysegment_6.webp) 5. Use the `Scale` ([`SetScale`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a9d75067bf7a037db3dcc9bcbd4a4d84e) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#ada373e013b5754bd77f2d3d5c14d8c33) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a4fd96402784b7f207d121e99bf22e1b9)) to automatically resize your images to a lower resolution and accelerate the processing. In **Deep Learning Studio**: * If the dataset contains images with different resolutions, the `Input width` and the `Input height` indicate the range of the resolutions with the given scale. * If all the images in the dataset have the same resolution, set either the `Input width` or the `Input height` to change the scale. ## Training To train your tool, see [Training a Deep Learning Tool](../easydeeplearning/training_a_deep_learning_tool.html.md). ## Evaluating the Results * There are 3 types of metrics for the supervised segmentation tool: * The ***pixel-based metrics*** that quantify the performance of the tool at the pixel level. * The ***blob-based metrics*** that quantify the performance of the tool at the blob level. A blob is a contiguous region of pixels that have the same foreground segmentation label. By definition, there is no background blob. * The ***image-based metrics*** that quantify the performance of the tool at the image level. These metrics are related to the capacity of the tool to correctly detect background images (images with no blobs) and foreground images (images with blobs). * In **Deep Learning Studio**: * The metrics are available in the `Validation and results` tab. * Most metrics depends on the value of the `Classification threshold`. ![image](assets/images/easydeeplearning/supervised_evaluating_1.webp) * In the API: * The metrics are represented by an [`ESupervisedSegmenterMetrics`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterMetrics.html) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterMetrics.html) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterMetrics.html) object. ### The Pixel-Based Metrics ![image](assets/images/easydeeplearning/supervised_evaluating_3.webp) * The pixel-based metrics are: * The weighted ***Intersection over Union*** (IoU) that is the weighted average of the IoU over all the labels (see per-label metrics). * The weighted ***pixel accuracy*** that is the weighted average of the accuracy over all the labels (see per-label metrics). * The ***pixel confusion matrix*** that shows the number of pixels from a given label that are predicted to belong to another label. * The per label metrics are: * The ***Intersection over Union*** (IoU) that is the ratio of the intersection between the ground truth and the prediction for the label to the union of the ground truth and prediction for the label. ![](assets/images/easydeeplearning/supervised_evaluating_2.svg) * The ***accuracy*** that is the proportion of the pixels of the label that are correctly predicted. ### The Blob-Based Metrics ![image](assets/images/easydeeplearning/supervised_evaluating_4.webp) * The metrics related to the correct prediction of blobs are: * The ***recall*** that is the ratio of the correctly predicted blobs to the total number of ground truth blobs. * The ***precision*** that is the ratio of correctly predicted blobs to the total number of predicted blobs. * The ***F1-Score*** that is the harmonic mean of the ***recall*** and the ***precision***. * The ***average precision*** that is the average of the ***precision*** for different threshold weighted by the ***recall*** values. * The ***best achievable F1-Score*** that is the maximum ***F1-Score*** achievable by selecting an appropriate threshold. * The confusion matrix: * It shows the number of blobs of a given true label that are predicted to be of the corresponding predicted label. * The image list of the `Dataset results` tab only shows the images containing the blobs corresponding to the selected cells of the matrix. * Each matrix element shows the number of ground truth blob and the number of corresponding predicted blobs separated by a `"/"` as a ground truth blob can correspond to one or more blobs in the prediction and inversely. * A dash (-) indicates that blobs are not defined for this category. * For example, in the screenshot above, there are: * 0 predicted blobs of the label `Defect` that correspond to the `Background` (false positive). * 9 ground truth blob of the label `Defect` that does not correspond to any predicted blob (false negative). * 163 predicted blobs of the label `Defect` that correspond to 158 ground truth blob of the same label (true positive). * The metrics for each individual foreground label are: * ***Recall*** * ***Precision*** * ***F1-Score*** ### The Image-Based Metrics * The metrics related to the correct detection of the class of the images (background / foreground or good / defective in the context of defect detection): * The ***image detection accuracy*** that is the proportion of image correctly predicted to have foreground blobs or not. * The ***foreground image detection rate*** that is the proportion of correctly predicted images with foreground blobs. * The ***background image detection rate*** that is the proportion of correctly predicted images with no foreground blobs. * The ***image detection AUC*** ( ***Area under the ROC Curve***, see ***ROC Curve*** below). * The ***best achievable image detection accuracy*** that is the maximum ***image detection accuracy*** obtained by changing the threshold. ![image](assets/images/easydeeplearning/supervised_evaluating_6.webp) * The confusion matrix: * It shows the number of images of a given true label that are predicted to be of the corresponding predicted label. * The image list of the `Dataset results` tab only shows the images corresponding to the selected cells of the matrix. ![image](assets/images/easydeeplearning/supervised_evaluating_8.webp) * The 2 available graphics are: * The ***ROC curve*** that plots the true positive rate ( ***foreground image detection rate***) versus the false positive rate (1 minus the ***background image detection rate***) for various threshold values. Click on a point on the plot to set the threshold at the corresponding value. ![image](assets/images/easydeeplearning/supervised_evaluating_7.webp) * The ***score histogram*** that plots: * In green: the cumulative histogram of the scores of the background images. * In orange: the cumulative histogram of the scores of the foreground images. * The blue line corresponds to the current value of the threshold. ![image](assets/images/easydeeplearning/supervised_evaluating_5.webp) ## Applying the Tool to New Images To get the result, a supervised segmenter follows these steps: 1. For each pixel, the supervised segmenter tool computes the probabilities that it belongs to each of the segmentation labels. 2. From these probabilities, it extracts a set of potential foreground blobs (groups of contiguous pixels for which the highest probability corresponds to the same foreground segmentation label). 3. For each one of these potential foreground blobs: * It computes a score. * It removes, from the predicted segmentation map, the blobs with a score that is below or equal to the threshold of the supervised segmenter tool. 4. The score of an image is the maximum among the scores of the potential foreground blobs. #### NOTE In the context of defect detection, an image is considered to be without defect when its score is below or equal to the threshold of the supervised segmenter tool. #### SEE ALSO The `Supervised segmentation` **eVision Studio** sample illustrates inference with **EasySegment Supervised** on the `Coffee` dataset. See [Additional Resources](../deep_learning_studio.html.md#easydeeplearning-dlstudio-additional-resources) for details about the dataset and how to get it. ### Deep Learning Studio * Include the new images in the test set 1. Add new images to the dataset from the `Dataset and annotations` tab. 2. In the `Validation and results` tab, refresh the results ![image](assets/images/easydeeplearning/easysegment_8.webp) * Open the `Inference tests` tab to apply the segmenter to new images and display detailed results for these images. * To visualize the segmentation of an image, check the `Predicted segmentation` option (`CTRL`+`P`) in the `Result` menu (`ALT`+`R`) of the image viewer. ![](assets/images/easydeeplearning/supervised_using_1.webp) * If the image has a ground truth, check the `Ground truth segmentation` option (`CTRL`+`G`) to display it. It appears with a green pattern drawn over it. ![](assets/images/easydeeplearning/supervised_using_2.webp)![](assets/images/easydeeplearning/supervised_using_3.webp) * To accept the whole predicted segmentation as ground truth, click on the `Use prediction as ground truth` button (`CTRL`+`U`). * To accept a single predicted blob as ground truth, right click on the blob and select `Accept into ground truth` in the menu. ![](assets/images/easydeeplearning/supervised_using_4.webp) * A list of blobs with various characteristics is available in the `Results` tab of the image viewer. ![](assets/images/easydeeplearning/supervised_using_5.webp) ### API * To apply the supervised segmenter to an image use [`ESupervisedSegmenter::Apply`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#ab853c91d3892b8b558a152abd8063145) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#ae3960ce7acd9b65bc1b5bf2e3ac22f3a) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenter.html#a36a18c88551f8616f42ff830e995169a). This method returns a [`ESupervisedSegmenterResult`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html) object. * Use [`GetProbabilityMap`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a32dc408b3c6eead745e2c78050283f45) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a6ccc06fb0018092cabad436c0b82b67b) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a1e031a395d4f25576f195940c487db0d) to retrieve the probability map for a given label. The probability map pixels contain the index of the predicted label. * Use [`Draw`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a07348dc73cf07e98a52d8539724ff68e) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a4512dea382dbe344e62371f192bad0cb) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#af763e0dc32844d41ecd5d1d8acdcb4ee) to draw the segmentation with the segmentation label colors of the dataset used for training. * Use [`GetBlobs`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a0349838f3960742d5dcda564b2b03851) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a37c7274814a42a1f7adef3710839e7e4) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a92793823c8218f94fbcfdc452da2a1c9) to retrieve the filtered list of blobs. * Use [`GetScore`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a638d64b0e192f37027eca2b6c36ea83e) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a71e218f7146a2ead6480942b74626950) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a6396ab034871227e49aac396dc45d8f5) to retrieve the score of an image. * Use [`GetRegionForLabel`](../../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a6a778024971da7ebbbefeca937b38853) [.NET](../../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#aaae5ae9efe10ebcd6fbd08ef24cb3dff) [Python](../../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1ESupervisedSegmenterResult.html#a64435b5636ec9e403c964a3e28b2df06) to obtain an `ERegion` object containing the pixels of the specified label.