eVision Python API
Loading...
Searching...
No Matches
eVision.EasyDeepLearning.ESupervisedSegmenterMetrics Class Reference

Collection of metrics used to evaluate the state of a eVision.EasyDeepLearning.ESupervisedSegmenter tool. More...

Inheritance diagram for eVision.EasyDeepLearning.ESupervisedSegmenterMetrics:
Collaboration diagram for eVision.EasyDeepLearning.ESupervisedSegmenterMetrics:

Public Member Functions

None __init__ (self)
 Constructs an eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object.
None __init__ (self, eVision.EasyDeepLearning.ESupervisedSegmenterMetrics other)
 Copy constructor of an eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object.
None Load (self, str path)
 Loads an supervised segmentation metric.
None Load (self, eVision.ESerializer|None serializer)
 Loads an supervised segmentation metric.
None Save (self, str path)
 Saves an supervised segmentation metric.
None Save (self, eVision.ESerializer|None serializer)
 Saves an supervised segmentation metric.
bool IsValid (self)
 Indicates whether the object contains at least one result.
float GetPixelLabelAccuracy (self, str label)
 Pixel label accuracy.
float GetLabelError (self, str label)
 Label error.
int GetPixelConfusion (self, str trueLabel, str predictedLabel)
 Pixel-wise confusion between the given true and predicted labels.
float GetNormalizedPixelConfusion (self, str trueLabel, str predictedLabel)
 Pixel-wise normalized confusion between the given true and predicted labels.
float GetIntersectionOverUnion (self, str label)
 Intersection over union.
int GetGroundtruthBlobConfusion (self, str groundtruthLabel, str predictedLabel)
 Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.
int GetGroundtruthBlobConfusion (self, int groundtruthLabelIndex, int predictedLabelIndex)
 Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.
int GetPredictedBlobConfusion (self, str groundtruthLabel, str predictedLabel)
 Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.
int GetPredictedBlobConfusion (self, int groundtruthLabelIndex, int predictedLabelIndex)
 Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.
str GetLabel (self, int labelIdx)
 Label recognized by the segmenter that produced the results aggregated in the metrics.
float WeightedPixelAccuracy (self)
 Weighted accuracy.
float BalancedPixelAccuracy (self)
 Balanced accuracy.
float PixelAccuracy (self)
 Accuracy.
float WeightedError (self)
 Weighted error.
float BalancedError (self)
 Balanced error.
float Error (self)
 Error.
float WeightedIntersectionOverUnion (self)
 Weighted Intersection over Union (IoU).
float BalancedIntersectionOverUnion (self)
 Balanced Intersection over Union (IoU).
float BlobDetectionAveragePrecision (self)
 Average precision for blob detection.
float BlobDetectionBestFScore (self)
 Best F1-Score of the segmenter for blob detection.
float BlobDetectionBestFScoreThreshold (self)
 Classification threshold that achieves the best F1-Score for blob detection.
float BlobDetectionRecall (self)
 Recall for blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
float BlobDetectionPrecision (self)
 Precision for blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
float BlobDetectionFScore (self)
 F1-Score for defective blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
int NumLabels (self)
 Number of labels recognized by the segmenter that produced the results aggregated in the metrics.
Public Member Functions inherited from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics
None __init__ (self)
 Constructs an empty eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics object.
None __init__ (self, eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics other)
 Copy constructor.
bool IsDefectDetectionMetricsValid (self)
 Whether the defect detection metrics are valid or not.
float GetAccuracy (self, int index)
 The accuracy of the segmenter.
float GetAccuracy (self)
 The accuracy of the segmenter.
float GetBestWeightedAccuracy (self, float goodWeight, float badWeight)
 Best achievable weighted accuracy.
float GetBestWeightedAccuracyClassificationThreshold (self, float goodWeight, float badWeight)
 Classification threshold giving the best achievable weighted accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy).
int GetConfusion (self, eVision.EasyDeepLearning.EConfusionMatrixElement element, int index)
 Confusion value of one label with another.
int GetConfusion (self, eVision.EasyDeepLearning.EConfusionMatrixElement element)
 Confusion value of one label with another.
float GetRecall (self, int index)
 Recall for the current threshold.
float GetRecall (self)
 Recall for the current threshold.
float GetPrecision (self, int index)
 Precision for the current threshold.
float GetPrecision (self)
 Precision for the current threshold.
float GetFScore (self, int index)
 The F1-Score for the current threshold.
float GetFScore (self)
 The F1-Score for the current threshold.
eVision.EasyDeepLearning.EROCPoint GetPrecisionRecallCurvePoint (self, int index)
 Get a point on the precision/recall curve.
eVision.EasyDeepLearning.EROCPoint GetPrecisionRecallCurvePoint (self)
 Get a point on the precision/recall curve.
eVision.EasyDeepLearning.EROCPoint GetROCPoint (self, int index)
 ROC (Receiver Operating Characteristic) point.
eVision.EasyDeepLearning.EROCPoint GetROCPoint (self)
 ROC (Receiver Operating Characteristic) point.
int NumDefectiveSample (self)
 Number of defective sample added to the metric.
int NumGoodSample (self)
 Number of good sample added to the metric.
float BestAccuracy (self)
 Best achievable accuracy.
float BestBalancedAccuracy (self)
 Best achievable balanced accuracy.
float BestAccuracyClassificationThreshold (self)
 Classification threshold giving the best achievable accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracy).
float BestBalancedAccuracyClassificationThreshold (self)
 Classification threshold giving the best achievable balanced accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy).
float AreaUnderROCCurve (self)
 The area under ROC curve (AUC) of the classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint).
float AveragePrecision (self)
 Average precision.
float BestFScore (self)
 Best F1-Score.
float BestFScoreThreshold (self)
 Classification threshold that yields the best F1-Score.
int PrecisionRecallCurveIndex (self)
 Index in the precision/recall curve for the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.
int NumPrecisionRecallCurvePoint (self)
 Number of point in the precision/recall curve.
int NumberOfClassifiers (self)
 Number of different possible classifiers.
float ClassificationThreshold (self)
 Classification threshold.
 ClassificationThreshold (self, float value)
 Classification threshold.

Detailed Description

Collection of metrics used to evaluate the state of a eVision.EasyDeepLearning.ESupervisedSegmenter tool.

A metric is a value summarizing the quality of a collection of supervised segmentation results (see eVision.EasyDeepLearning.ESupervisedSegmenterResult) with respect to their ground truth. The eVision.EasyDeepLearning.ESupervisedSegmenterMetrics contains three types of metrics:

  • pixel based metrics that are related to the quality of the segmentation masks
  • blob based metrics that are related to the ability of the supervised segmentation tool to detect foreground blobs
  • defect detection metrics that are related to the ability of the supervised segmentation tool to differentiate between images that contains foreground pixels (defective images) and images that are entirely background (good images).

The pixel metrics are the error (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Error), the pixel accuracy (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.PixelAccuracy), the pixel confusion (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelConfusion), the pixel per-label accuracy (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelLabelAccuracy) and the intersection over union (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetIntersectionOverUnion).

The blob based metrics are two confusion matrices (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion and eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion), and various defect detection metrics (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionRecall, eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionPrecision, and eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionFScore). Note that the metrics for defective blob detection are different from the metrics for defective image detection because, in the case of blob detection, we have no "good" ground truth results.

See eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics for a description of the defect detection metrics. These metrics are available when eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.IsDefectDetectionMetricsValid is true, i.e. when both images that are entirely background and images with non-background pixels have been added to the metrics.

Most metrics depends upon the eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.

Remarks:

License(s): BasicTypes

Constructor & Destructor Documentation

◆ __init__() [1/2]

None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.__init__ ( self)

◆ __init__() [2/2]

None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.__init__ ( self,
eVision.EasyDeepLearning.ESupervisedSegmenterMetrics other )

Copy constructor of an eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object.

Parameters
otherReference to the eVision.EasyDeepLearning.ESupervisedSegmenterMetrics object that should be copied

Member Function Documentation

◆ BalancedError()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedError ( self)

Balanced error.

The balanced error is the weighted error (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedError) where each label is given an equal weight.

Remarks:

The error is also called the cross-entropy loss.

◆ BalancedIntersectionOverUnion()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedIntersectionOverUnion ( self)

Balanced Intersection over Union (IoU).

The balanced intersection over union is the weighted intersection over union (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedIntersectionOverUnion) where each label is given equal weight.

◆ BalancedPixelAccuracy()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedPixelAccuracy ( self)

Balanced accuracy.

The balanced accuracy is the weighted accuracy (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedPixelAccuracy) where each label is given an equal weight.

◆ BlobDetectionAveragePrecision()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionAveragePrecision ( self)

Average precision for blob detection.

Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label. The average precision for blob detection is the average of the precision (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionPrecision) over all the possible classification threshold values. As such, the average precision does not depend on a specific classification threshold.

◆ BlobDetectionBestFScore()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScore ( self)

Best F1-Score of the segmenter for blob detection.

See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScoreThreshold for the corresponding threshold.

See eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ BlobDetectionBestFScoreThreshold()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScoreThreshold ( self)

Classification threshold that achieves the best F1-Score for blob detection.

See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScore for the corresponding F1-Score.

See eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ BlobDetectionFScore()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionFScore ( self)

F1-Score for defective blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).

See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScore for the corresponding F1-Score.

See eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ BlobDetectionPrecision()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionPrecision ( self)

Precision for blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).

The precision is the proportion of detected defective blobs that match ground truth defective blobs.

Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ BlobDetectionRecall()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionRecall ( self)

Recall for blob detection given the current classification threshold (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold).

The recall is the proportion of ground truth defective blobs that are matched to predicted defective blobs.

Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ Error()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Error ( self)

Error.

Remarks:

The error is also called the cross-entropy loss.

◆ GetGroundtruthBlobConfusion() [1/2]

int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion ( self,
int groundtruthLabelIndex,
int predictedLabelIndex )

Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.

See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion for the number of predicted blobs that match to these ground truth blobs.

Remarks:

A ground truth blob is matched to the subset of predicted blobs from the same predicted label that has the best intersection over union with the ground truth blob. Otherwise, the ground truth blob is matched to background.

Parameters
groundtruthLabelIndexGround truth label index
predictedLabelIndexPredicted label index

◆ GetGroundtruthBlobConfusion() [2/2]

int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion ( self,
str groundtruthLabel,
str predictedLabel )

Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.

See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion for the number of predicted blobs that match to these ground truth blobs.

Remarks:

A ground truth blob is matched to the subset of predicted blobs from the same predicted label that has the best intersection over union with the ground truth blob. Otherwise, the ground truth blob is matched to background.

Parameters
groundtruthLabelGround truth label
predictedLabelPredicted label

◆ GetIntersectionOverUnion()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetIntersectionOverUnion ( self,
str label )

Intersection over union.

The intersection over union if the ratio between the number of correctly classified pixels from the given label to the total number of pixels that belongs to that label or are predicted as being from that label.

Assuming that the given label is the positive class, the intersection over union is expressed as IOU = TP / (TP + FP + FN) where TP is the number of true positive, FP the number of false positive (pixels that belongs to label but are not predicted as label) and FN the number of false negative (pixels predicted as label but that do not belong to label).

Remarks:

The intersection over union is also called the Jaccard index.

Parameters
labelLabel

◆ GetLabel()

str eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetLabel ( self,
int labelIdx )

Label recognized by the segmenter that produced the results aggregated in the metrics.

Parameters
labelIdxIndex of the label between 0 and eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.NumLabels - 1.

◆ GetLabelError()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetLabelError ( self,
str label )

Label error.

Parameters
labelLabel

◆ GetNormalizedPixelConfusion()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetNormalizedPixelConfusion ( self,
str trueLabel,
str predictedLabel )

Pixel-wise normalized confusion between the given true and predicted labels.

The normalized confusion is the ratio of the number of pixels belonging to the 'trueLabel' that are classified as 'predictedLabel' to the total number of pixels belonging to the 'trueLabel'.

Parameters
trueLabelTrue label
predictedLabelPredicted label

◆ GetPixelConfusion()

int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelConfusion ( self,
str trueLabel,
str predictedLabel )

Pixel-wise confusion between the given true and predicted labels.

The confusion is the number of pixels belonging to the 'trueLabel' that are classified as 'predictedLabel'.

Parameters
trueLabelTrue label
predictedLabelPredicted label

◆ GetPixelLabelAccuracy()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelLabelAccuracy ( self,
str label )

Pixel label accuracy.

The label accuracy is the ratio of the number of correctly classified pixels of the given class to the total number of ground truth pixels from that class. If there are no ground truth pixels from the requested label, the method returns '-1'.

Parameters
labelLabel

◆ GetPredictedBlobConfusion() [1/2]

int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion ( self,
int groundtruthLabelIndex,
int predictedLabelIndex )

Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.

See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion for the number of ground truth blobs that match to these predicted blobs.

Remarks:

A predicted blob may be counted several times in the confusion matrix for predicted blobs:

  • Once if the more than 50% of the blob intersects ground truth background.
  • Once for each non-background label the predicted blob has an intersection with.
Parameters
groundtruthLabelIndexGround truth label index
predictedLabelIndexPredicted label index

◆ GetPredictedBlobConfusion() [2/2]

int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion ( self,
str groundtruthLabel,
str predictedLabel )

Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.

See eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion for the number of ground truth blobs that match to these predicted blobs.

Remarks:

A predicted blob may be counted several times in the confusion matrix for predicted blobs:

  • Once if the more than 50% of the blob intersects ground truth background.
  • Once for each non-background label the predicted blob has an intersection with.
Parameters
groundtruthLabelGround truth label
predictedLabelPredicted label

◆ IsValid()

bool eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.IsValid ( self)

Indicates whether the object contains at least one result.

◆ Load() [1/2]

None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Load ( self,
eVision.ESerializer | None serializer )

Loads an supervised segmentation metric.

The given eVision.ESerializer must have been created for reading.

Parameters
serializerThe serializer.

Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.

◆ Load() [2/2]

None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Load ( self,
str path )

Loads an supervised segmentation metric.

The given eVision.ESerializer must have been created for reading.

Parameters
pathThe file path.

Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.

◆ NumLabels()

int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.NumLabels ( self)

Number of labels recognized by the segmenter that produced the results aggregated in the metrics.

◆ PixelAccuracy()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.PixelAccuracy ( self)

Accuracy.

The accuracy is the ratio of the number of correctly classified pixels to the total number of pixels.

Remarks:

The accuracy will be skewed towards the most represented labels in the dataset. For example, if your dataset has 1% of defective pixels, the accuracy will mostly reflect the results on the 99% of good pixels and will say nothing about the capability of the model to detect the defect labels.

◆ Save() [1/2]

None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Save ( self,
eVision.ESerializer | None serializer )

Saves an supervised segmentation metric.

The given eVision.ESerializer must have been created for writing.

Parameters
serializerThe serializer.

Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.

◆ Save() [2/2]

None eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Save ( self,
str path )

Saves an supervised segmentation metric.

The given eVision.ESerializer must have been created for writing.

Parameters
pathThe file path.

Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.

◆ WeightedError()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedError ( self)

Weighted error.

The weighted error is the weighted average of each label error (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetLabelError) with respect to the dataset label weights.

Remarks:

The error is also called the cross-entropy loss.

◆ WeightedIntersectionOverUnion()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedIntersectionOverUnion ( self)

Weighted Intersection over Union (IoU).

The weighted intersection over union is the weighted averaged of each label intersection over union (see eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetIntersectionOverUnion) with respect to the dataset label weights.

◆ WeightedPixelAccuracy()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedPixelAccuracy ( self)

Weighted accuracy.

The weighted accuracy is the weighted average of each label accuracy (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelLabelAccuracy) with respect to the dataset label weights.