Collection of metrics used to evaluate the state of a eVision.EasyDeepLearning.ESupervisedSegmenter tool. More...


Public Member Functions | |
| None | __init__ (self) |
| Constructs an eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object. | |
| None | __init__ (self, eVision.EasyDeepLearning.ESupervisedSegmenterMetrics other) |
| Copy constructor of an eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object. | |
| None | Load (self, str path) |
| Loads an supervised segmentation metric. | |
| None | Load (self, eVision.ESerializer|None serializer) |
| Loads an supervised segmentation metric. | |
| None | Save (self, str path) |
| Saves an supervised segmentation metric. | |
| None | Save (self, eVision.ESerializer|None serializer) |
| Saves an supervised segmentation metric. | |
| bool | IsValid (self) |
| Indicates whether the object contains at least one result. | |
| float | GetPixelLabelAccuracy (self, str label) |
| Pixel label accuracy. | |
| float | GetLabelError (self, str label) |
| Label error. | |
| int | GetPixelConfusion (self, str trueLabel, str predictedLabel) |
| Pixel-wise confusion between the given true and predicted labels. | |
| float | GetNormalizedPixelConfusion (self, str trueLabel, str predictedLabel) |
| Pixel-wise normalized confusion between the given true and predicted labels. | |
| float | GetIntersectionOverUnion (self, str label) |
| Intersection over union. | |
| int | GetGroundtruthBlobConfusion (self, str groundtruthLabel, str predictedLabel) |
| Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label. | |
| int | GetGroundtruthBlobConfusion (self, int groundtruthLabelIndex, int predictedLabelIndex) |
| Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label. | |
| int | GetPredictedBlobConfusion (self, str groundtruthLabel, str predictedLabel) |
| Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label. | |
| int | GetPredictedBlobConfusion (self, int groundtruthLabelIndex, int predictedLabelIndex) |
| Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label. | |
| str | GetLabel (self, int labelIdx) |
| Label recognized by the segmenter that produced the results aggregated in the metrics. | |
| float | WeightedPixelAccuracy (self) |
| Weighted accuracy. | |
| float | BalancedPixelAccuracy (self) |
| Balanced accuracy. | |
| float | PixelAccuracy (self) |
| Accuracy. | |
| float | WeightedError (self) |
| Weighted error. | |
| float | BalancedError (self) |
| Balanced error. | |
| float | Error (self) |
| Error. | |
| float | WeightedIntersectionOverUnion (self) |
| Weighted Intersection over Union (IoU). | |
| float | BalancedIntersectionOverUnion (self) |
| Balanced Intersection over Union (IoU). | |
| float | BlobDetectionAveragePrecision (self) |
| Average precision for blob detection. | |
| float | BlobDetectionBestFScore (self) |
| Best F1-Score of the segmenter for blob detection. | |
| float | BlobDetectionBestFScoreThreshold (self) |
| Classification threshold that achieves the best F1-Score for blob detection. | |
| float | BlobDetectionRecall (self) |
| Recall for blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold). | |
| float | BlobDetectionPrecision (self) |
| Precision for blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold). | |
| float | BlobDetectionFScore (self) |
| F1-Score for defective blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold). | |
| int | NumLabels (self) |
| Number of labels recognized by the segmenter that produced the results aggregated in the metrics. | |
| Public Member Functions inherited from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics | |
| None | __init__ (self) |
| Constructs an empty eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics object. | |
| None | __init__ (self, eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics other) |
| Copy constructor. | |
| bool | IsDefectDetectionMetricsValid (self) |
| Whether the defect detection metrics are valid or not. | |
| float | GetAccuracy (self, int index) |
| The accuracy of the segmenter. | |
| float | GetAccuracy (self) |
| The accuracy of the segmenter. | |
| float | GetBestWeightedAccuracy (self, float goodWeight, float badWeight) |
| Best achievable weighted accuracy. | |
| float | GetBestWeightedAccuracyClassificationThreshold (self, float goodWeight, float badWeight) |
| Classification threshold giving the best achievable weighted accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy). | |
| int | GetConfusion (self, eVision.EasyDeepLearning.EConfusionMatrixElement element, int index) |
| Confusion value of one label with another. | |
| int | GetConfusion (self, eVision.EasyDeepLearning.EConfusionMatrixElement element) |
| Confusion value of one label with another. | |
| float | GetRecall (self, int index) |
| Recall for the current threshold. | |
| float | GetRecall (self) |
| Recall for the current threshold. | |
| float | GetPrecision (self, int index) |
| Precision for the current threshold. | |
| float | GetPrecision (self) |
| Precision for the current threshold. | |
| float | GetFScore (self, int index) |
| The F1-Score for the current threshold. | |
| float | GetFScore (self) |
| The F1-Score for the current threshold. | |
| eVision.EasyDeepLearning.EROCPoint | GetPrecisionRecallCurvePoint (self, int index) |
| Get a point on the precision/recall curve. | |
| eVision.EasyDeepLearning.EROCPoint | GetPrecisionRecallCurvePoint (self) |
| Get a point on the precision/recall curve. | |
| eVision.EasyDeepLearning.EROCPoint | GetROCPoint (self, int index) |
| ROC (Receiver Operating Characteristic) point. | |
| eVision.EasyDeepLearning.EROCPoint | GetROCPoint (self) |
| ROC (Receiver Operating Characteristic) point. | |
| int | NumDefectiveSample (self) |
| Number of defective sample added to the metric. | |
| int | NumGoodSample (self) |
| Number of good sample added to the metric. | |
| float | BestAccuracy (self) |
| Best achievable accuracy. | |
| float | BestBalancedAccuracy (self) |
| Best achievable balanced accuracy. | |
| float | BestAccuracyClassificationThreshold (self) |
| Classification threshold giving the best achievable accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracy). | |
| float | BestBalancedAccuracyClassificationThreshold (self) |
| Classification threshold giving the best achievable balanced accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy). | |
| float | AreaUnderROCCurve (self) |
| The area under ROC curve (AUC) of the classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint). | |
| float | AveragePrecision (self) |
| Average precision. | |
| float | BestFScore (self) |
| Best F1-Score. | |
| float | BestFScoreThreshold (self) |
| Classification threshold that yields the best F1-Score. | |
| int | PrecisionRecallCurveIndex (self) |
| Index in the precision/recall curve for the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. | |
| int | NumPrecisionRecallCurvePoint (self) |
| Number of point in the precision/recall curve. | |
| int | NumberOfClassifiers (self) |
| Number of different possible classifiers. | |
| float | ClassificationThreshold (self) |
| Classification threshold. | |
| ClassificationThreshold (self, float value) | |
| Classification threshold. | |
Collection of metrics used to evaluate the state of a eVision.EasyDeepLearning.ESupervisedSegmenter tool.
A metric is a value summarizing the quality of a collection of supervised segmentation results (see eVision.EasyDeepLearning.ESupervisedSegmenterResult) with respect to their ground truth. The eVision.EasyDeepLearning.ESupervisedSegmenterMetrics contains three types of metrics:
The pixel metrics are the error (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Error), the pixel accuracy (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.PixelAccuracy), the pixel confusion (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelConfusion), the pixel per-label accuracy (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelLabelAccuracy) and the intersection over union (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetIntersectionOverUnion).
The blob based metrics are two confusion matrices (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion and eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion), and various defect detection metrics (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionRecall, eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionPrecision, and eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionFScore). Note that the metrics for defective blob detection are different from the metrics for defective image detection because, in the case of blob detection, we have no "good" ground truth results.
See eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics for a description of the defect detection metrics. These metrics are available when eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.IsDefectDetectionMetricsValid is true, i.e. when both images that are entirely background and images with non-background pixels have been added to the metrics.
Most metrics depends upon the eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.
Remarks:
License(s): BasicTypes
| None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.__init__ | ( | self | ) |
Constructs an eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object.
| None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.__init__ | ( | self, | |
| eVision.EasyDeepLearning.ESupervisedSegmenterMetrics | other ) |
Copy constructor of an eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object.
| other | Reference to the eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object that should be copied |
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedError | ( | self | ) |
Balanced error.
The balanced error is the weighted error (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedError) where each label is given an equal weight.
Remarks:
The error is also called the cross-entropy loss.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedIntersectionOverUnion | ( | self | ) |
Balanced Intersection over Union (IoU).
The balanced intersection over union is the weighted intersection over union (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedIntersectionOverUnion) where each label is given equal weight.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedPixelAccuracy | ( | self | ) |
Balanced accuracy.
The balanced accuracy is the weighted accuracy (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedPixelAccuracy) where each label is given an equal weight.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionAveragePrecision | ( | self | ) |
Average precision for blob detection.
Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label. The average precision for blob detection is the average of the precision (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionPrecision) over all the possible classification threshold values. As such, the average precision does not depend on a specific classification threshold.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScore | ( | self | ) |
Best F1-Score of the segmenter for blob detection.
See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScoreThreshold for the corresponding threshold.
See eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScoreThreshold | ( | self | ) |
Classification threshold that achieves the best F1-Score for blob detection.
See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScore for the corresponding F1-Score.
See eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionFScore | ( | self | ) |
F1-Score for defective blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScore for the corresponding F1-Score.
See eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionPrecision | ( | self | ) |
Precision for blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
The precision is the proportion of detected defective blobs that match ground truth defective blobs.
Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionRecall | ( | self | ) |
Recall for blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
The recall is the proportion of ground truth defective blobs that are matched to predicted defective blobs.
Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Error | ( | self | ) |
Error.
Remarks:
The error is also called the cross-entropy loss.
| int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion | ( | self, | |
| int | groundtruthLabelIndex, | ||
| int | predictedLabelIndex ) |
Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.
See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion for the number of predicted blobs that match to these ground truth blobs.
Remarks:
A ground truth blob is matched to the subset of predicted blobs from the same predicted label that has the best intersection over union with the ground truth blob. Otherwise, the ground truth blob is matched to background.
| groundtruthLabelIndex | Ground truth label index |
| predictedLabelIndex | Predicted label index |
| int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion | ( | self, | |
| str | groundtruthLabel, | ||
| str | predictedLabel ) |
Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.
See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion for the number of predicted blobs that match to these ground truth blobs.
Remarks:
A ground truth blob is matched to the subset of predicted blobs from the same predicted label that has the best intersection over union with the ground truth blob. Otherwise, the ground truth blob is matched to background.
| groundtruthLabel | Ground truth label |
| predictedLabel | Predicted label |
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetIntersectionOverUnion | ( | self, | |
| str | label ) |
Intersection over union.
The intersection over union if the ratio between the number of correctly classified pixels from the given label to the total number of pixels that belongs to that label or are predicted as being from that label.
Assuming that the given label is the positive class, the intersection over union is expressed as IOU = TP / (TP + FP + FN) where TP is the number of true positive, FP the number of false positive (pixels that belongs to label but are not predicted as label) and FN the number of false negative (pixels predicted as label but that do not belong to label).
Remarks:
The intersection over union is also called the Jaccard index.
| label | Label |
| str eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetLabel | ( | self, | |
| int | labelIdx ) |
Label recognized by the segmenter that produced the results aggregated in the metrics.
| labelIdx | Index of the label between 0 and eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.NumLabels - 1. |
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetLabelError | ( | self, | |
| str | label ) |
Label error.
| label | Label |
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetNormalizedPixelConfusion | ( | self, | |
| str | trueLabel, | ||
| str | predictedLabel ) |
Pixel-wise normalized confusion between the given true and predicted labels.
The normalized confusion is the ratio of the number of pixels belonging to the 'trueLabel' that are classified as 'predictedLabel' to the total number of pixels belonging to the 'trueLabel'.
| trueLabel | True label |
| predictedLabel | Predicted label |
| int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelConfusion | ( | self, | |
| str | trueLabel, | ||
| str | predictedLabel ) |
Pixel-wise confusion between the given true and predicted labels.
The confusion is the number of pixels belonging to the 'trueLabel' that are classified as 'predictedLabel'.
| trueLabel | True label |
| predictedLabel | Predicted label |
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelLabelAccuracy | ( | self, | |
| str | label ) |
Pixel label accuracy.
The label accuracy is the ratio of the number of correctly classified pixels of the given class to the total number of ground truth pixels from that class. If there are no ground truth pixels from the requested label, the method returns '-1'.
| label | Label |
| int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion | ( | self, | |
| int | groundtruthLabelIndex, | ||
| int | predictedLabelIndex ) |
Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.
See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion for the number of ground truth blobs that match to these predicted blobs.
Remarks:
A predicted blob may be counted several times in the confusion matrix for predicted blobs:
| groundtruthLabelIndex | Ground truth label index |
| predictedLabelIndex | Predicted label index |
| int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion | ( | self, | |
| str | groundtruthLabel, | ||
| str | predictedLabel ) |
Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.
See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion for the number of ground truth blobs that match to these predicted blobs.
Remarks:
A predicted blob may be counted several times in the confusion matrix for predicted blobs:
| groundtruthLabel | Ground truth label |
| predictedLabel | Predicted label |
| bool eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.IsValid | ( | self | ) |
Indicates whether the object contains at least one result.
| None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Load | ( | self, | |
| eVision.ESerializer | None | serializer ) |
Loads an supervised segmentation metric.
The given eVision.ESerializer must have been created for reading.
| serializer | The serializer. |
Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.
| None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Load | ( | self, | |
| str | path ) |
Loads an supervised segmentation metric.
The given eVision.ESerializer must have been created for reading.
| path | The file path. |
Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.
| int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.NumLabels | ( | self | ) |
Number of labels recognized by the segmenter that produced the results aggregated in the metrics.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.PixelAccuracy | ( | self | ) |
Accuracy.
The accuracy is the ratio of the number of correctly classified pixels to the total number of pixels.
Remarks:
The accuracy will be skewed towards the most represented labels in the dataset. For example, if your dataset has 1% of defective pixels, the accuracy will mostly reflect the results on the 99% of good pixels and will say nothing about the capability of the model to detect the defect labels.
| None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Save | ( | self, | |
| eVision.ESerializer | None | serializer ) |
Saves an supervised segmentation metric.
The given eVision.ESerializer must have been created for writing.
| serializer | The serializer. |
Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.
| None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Save | ( | self, | |
| str | path ) |
Saves an supervised segmentation metric.
The given eVision.ESerializer must have been created for writing.
| path | The file path. |
Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedError | ( | self | ) |
Weighted error.
The weighted error is the weighted average of each label error (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetLabelError) with respect to the dataset label weights.
Remarks:
The error is also called the cross-entropy loss.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedIntersectionOverUnion | ( | self | ) |
Weighted Intersection over Union (IoU).
The weighted intersection over union is the weighted averaged of each label intersection over union (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetIntersectionOverUnion) with respect to the dataset label weights.
| float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedPixelAccuracy | ( | self | ) |
Weighted accuracy.
The weighted accuracy is the weighted average of each label accuracy (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelLabelAccuracy) with respect to the dataset label weights.