eVision Python API
Loading...
Searching...
No Matches
eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics Class Reference

Collection of metrics used to evaluate a defect detection problem where the detection is based on the thresholding of a score produced by the underlying deep learning tool, e.g. More...

Inheritance diagram for eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics:

Public Member Functions

None __init__ (self)
 Constructs an empty eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics object.
None __init__ (self, eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics other)
 Copy constructor.
None Load (self, str path)
 Loads the defect detection metrics.
None Load (self, eVision.ESerializer|None serializer)
 Loads the defect detection metrics.
None Save (self, str path)
 Saves the defect detection metrics.
None Save (self, eVision.ESerializer|None serializer)
 Saves the defect detection metrics.
bool IsDefectDetectionMetricsValid (self)
 Whether the defect detection metrics are valid or not.
float GetAccuracy (self, int index)
 The accuracy of the segmenter.
float GetAccuracy (self)
 The accuracy of the segmenter.
float GetBestWeightedAccuracy (self, float goodWeight, float badWeight)
 Best achievable weighted accuracy.
float GetBestWeightedAccuracyClassificationThreshold (self, float goodWeight, float badWeight)
 Classification threshold giving the best achievable weighted accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy).
int GetConfusion (self, eVision.EasyDeepLearning.EConfusionMatrixElement element, int index)
 Confusion value of one label with another.
int GetConfusion (self, eVision.EasyDeepLearning.EConfusionMatrixElement element)
 Confusion value of one label with another.
float GetRecall (self, int index)
 Recall for the current threshold.
float GetRecall (self)
 Recall for the current threshold.
float GetPrecision (self, int index)
 Precision for the current threshold.
float GetPrecision (self)
 Precision for the current threshold.
float GetFScore (self, int index)
 The F1-Score for the current threshold.
float GetFScore (self)
 The F1-Score for the current threshold.
eVision.EasyDeepLearning.EROCPoint GetPrecisionRecallCurvePoint (self, int index)
 Get a point on the precision/recall curve.
eVision.EasyDeepLearning.EROCPoint GetPrecisionRecallCurvePoint (self)
 Get a point on the precision/recall curve.
eVision.EasyDeepLearning.EROCPoint GetROCPoint (self, int index)
 ROC (Receiver Operating Characteristic) point.
eVision.EasyDeepLearning.EROCPoint GetROCPoint (self)
 ROC (Receiver Operating Characteristic) point.
int NumDefectiveSample (self)
 Number of defective sample added to the metric.
int NumGoodSample (self)
 Number of good sample added to the metric.
float BestAccuracy (self)
 Best achievable accuracy.
float BestBalancedAccuracy (self)
 Best achievable balanced accuracy.
float BestAccuracyClassificationThreshold (self)
 Classification threshold giving the best achievable accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracy).
float BestBalancedAccuracyClassificationThreshold (self)
 Classification threshold giving the best achievable balanced accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy).
float AreaUnderROCCurve (self)
 The area under ROC curve (AUC) of the classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint).
float AveragePrecision (self)
 Average precision.
float BestFScore (self)
 Best F1-Score.
float BestFScoreThreshold (self)
 Classification threshold that yields the best F1-Score.
int PrecisionRecallCurveIndex (self)
 Index in the precision/recall curve for the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.
int NumPrecisionRecallCurvePoint (self)
 Number of point in the precision/recall curve.
int NumberOfClassifiers (self)
 Number of different possible classifiers.
float ClassificationThreshold (self)
 Classification threshold.
 ClassificationThreshold (self, float value)
 Classification threshold.

Detailed Description

Collection of metrics used to evaluate a defect detection problem where the detection is based on the thresholding of a score produced by the underlying deep learning tool, e.g.

a eVision.EasyDeepLearning.EUnsupervisedSegmenter tool or a eVision.EasyDeepLearning.ESupervisedSegmenter tool.

These metrics are only valid when results for good and defective images are included in the metrics (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.IsDefectDetectionMetricsValid). The definition of what is considered a good or a defective image depends on the deep learning tool used.

The defect detection metrics are separated in two main categories:

  • Metrics dependent on the ROC (Receiver Operating Characteristic) curve. They require at least one good and one defective sample to be defined.
  • Metrics dependent on the Precision/Recall curve. They require at least one defective sample to be defined.

The metrics related to the ROC curve are:

The metrics related to the precision/recall curve are:

The ROC and Precision/Recall curve are both obtained by computing some metrics for different values of the eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. Thus, each point on the curves gives different metrics, except for the area under the ROC curve and the average precision.

By default, the value of the metrics corresponds to the classification threshold of the corresponding deep learning tool. However, you can specify an index to retrieve the value of the metrics for other value of the eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. Metrics based on the ROC curve are indexed between 0 and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers-1 and metrics based on the Precision/Recall curve are indexed between 0 and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumPrecisionRecallCurvePoint-1.

Remarks:

License(s): BasicTypes

Constructor & Destructor Documentation

◆ __init__() [1/2]

None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.__init__ ( self)

◆ __init__() [2/2]

None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.__init__ ( self,
eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics other )

Copy constructor.

Parameters
otherOther object

Member Function Documentation

◆ AreaUnderROCCurve()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.AreaUnderROCCurve ( self)

The area under ROC curve (AUC) of the classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint).

It's value is between 0 and 1.

In the context of unsupervised segmentation, the AUC is equal to the probability that good images will have a lower reconstruction error than defective images.

This metrics measure discrimination capacity of a model. It tells how much the model is capable of distinguishing between classes. The higher the AUC, the better the model is.

◆ AveragePrecision()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.AveragePrecision ( self)

Average precision.

The average precision is the area under the precision-recall curve. The precision is the ratio between the number of true positive and the number of predicted positive and the recall, also called the true positive rate, is the ratio between the number of true positive and the number of positive sample.

◆ BestAccuracy()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracy ( self)

Best achievable accuracy.

The classification threshold corresponding to this accuracy is given by eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracyClassificationThreshold.

◆ BestAccuracyClassificationThreshold()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracyClassificationThreshold ( self)

Classification threshold giving the best achievable accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracy).

◆ BestBalancedAccuracy()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy ( self)

Best achievable balanced accuracy.

The classification threshold corresponding to this accuracy is given by eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracyClassificationThreshold.

◆ BestBalancedAccuracyClassificationThreshold()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracyClassificationThreshold ( self)

Classification threshold giving the best achievable balanced accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy).

◆ BestFScore()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestFScore ( self)

Best F1-Score.

The F1-Score is the harmonic mean of the precision and recall (true positive rate).

◆ BestFScoreThreshold()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestFScoreThreshold ( self)

Classification threshold that yields the best F1-Score.

The F1-Score is the harmonic mean of the precision and recall (true positive rate).

◆ ClassificationThreshold() [1/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold ( self)

Classification threshold.

By default, the classification threshold will be equal to the classification threshold of the last unsupervised segmenter used to produce the results that compose this metric.

Some metrics such as eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint, eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetAccuracy or eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetConfusion depends on the classification threshold. By default, these methods will returns the metric corresponding to the classification threshold.

Remarks:

Modifying the classification threshold in this class doesn't modify the classification threshold of the unsupervised segmenter.

◆ ClassificationThreshold() [2/2]

eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold ( self,
float value )

Classification threshold.

By default, the classification threshold will be equal to the classification threshold of the last unsupervised segmenter used to produce the results that compose this metric.

Some metrics such as eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint, eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetAccuracy or eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetConfusion depends on the classification threshold. By default, these methods will returns the metric corresponding to the classification threshold.

Remarks:

Modifying the classification threshold in this class doesn't modify the classification threshold of the unsupervised segmenter.

◆ GetAccuracy() [1/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetAccuracy ( self)

The accuracy of the segmenter.

The accuracy is the number of images that were correctly classified over the total number of images that was used to evaluate the classifier.

◆ GetAccuracy() [2/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetAccuracy ( self,
int index )

The accuracy of the segmenter.

The accuracy is the number of images that were correctly classified over the total number of images that was used to evaluate the classifier.

Parameters
indexThe index of the classifier to use. If the index is equal to '-1', the index corresponding to eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold will be used.

◆ GetBestWeightedAccuracy()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy ( self,
float goodWeight,
float badWeight )

Best achievable weighted accuracy.

The weighted accuracy is the weighted average of the true positive rate and the true negative rate (which is equal to 1 minus the false positive rate). See eVision.EasyDeepLearning.EROCPoint.

The classification threshold corresponding to this accuracy is given by eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracyClassificationThreshold.

Remarks:

When using a dataset as the source for the label weights, the good weight is the weight of the "good" label and the bad weight is the sum of the weights of all the other labels.

Parameters
goodWeightWeight for the good label
badWeightWeight for the bad label

Reimplemented in eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics, and eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics.

◆ GetBestWeightedAccuracyClassificationThreshold()

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracyClassificationThreshold ( self,
float goodWeight,
float badWeight )

Classification threshold giving the best achievable weighted accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy).

Parameters
goodWeightWeight for the good label
badWeightWeight for the bad label

Reimplemented in eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics, and eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics.

◆ GetConfusion() [1/2]

int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetConfusion ( self,
eVision.EasyDeepLearning.EConfusionMatrixElement element )

Confusion value of one label with another.

The confusion value of a label with another is the number of images belonging to this label that are classified as belonging to the other label.

For a eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics there are only 2 labels (good and defective) so the confusion matrix is only composed of 4 values which are called matrix element (see eVision.EasyDeepLearning.EConfusionMatrixElement).

The confusion matrix is computed for a given threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold) which means an index can be passed to the method (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers).

Parameters
elementThe element from which to obtain the confusion value

◆ GetConfusion() [2/2]

int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetConfusion ( self,
eVision.EasyDeepLearning.EConfusionMatrixElement element,
int index )

Confusion value of one label with another.

The confusion value of a label with another is the number of images belonging to this label that are classified as belonging to the other label.

For a eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics there are only 2 labels (good and defective) so the confusion matrix is only composed of 4 values which are called matrix element (see eVision.EasyDeepLearning.EConfusionMatrixElement).

The confusion matrix is computed for a given threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold) which means an index can be passed to the method (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers).

Parameters
elementThe element from which to obtain the confusion value
indexThe index of the classifier to use. If the index is '-1', the index corresponding to eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold will be used.

◆ GetFScore() [1/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore ( self)

◆ GetFScore() [2/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore ( self,
int index )

The F1-Score for the current threshold.

The F1-Score is the harmonic mean of eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetRecall and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecision.

Parameters
indexIndex for the precision/recall curve point or -1 to use the index corresponding to the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.

◆ GetPrecision() [1/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecision ( self)

Precision for the current threshold.

The precision is the proportion of detected defective instances that were correctly identified as such.

◆ GetPrecision() [2/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecision ( self,
int index )

Precision for the current threshold.

The precision is the proportion of detected defective instances that were correctly identified as such.

Parameters
indexIndex for the precision/recall curve point or -1 to use the index corresponding to the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.

◆ GetPrecisionRecallCurvePoint() [1/2]

eVision.EasyDeepLearning.EROCPoint eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecisionRecallCurvePoint ( self)

Get a point on the precision/recall curve.

Remarks:

The precision/recall curve is defined when there is at least one ground truth positive sample in the metric.

◆ GetPrecisionRecallCurvePoint() [2/2]

eVision.EasyDeepLearning.EROCPoint eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecisionRecallCurvePoint ( self,
int index )

Get a point on the precision/recall curve.

Remarks:

The precision/recall curve is defined when there is at least one ground truth positive sample in the metric.

Parameters
indexIndex for the precision/recall curve point or -1 to use the index corresponding to the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.

◆ GetRecall() [1/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetRecall ( self)

Recall for the current threshold.

It is the proportion of defective instances that were correctly identified as such. It is also called the true positive rate.

◆ GetRecall() [2/2]

float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetRecall ( self,
int index )

Recall for the current threshold.

It is the proportion of defective instances that were correctly identified as such. It is also called the true positive rate.

Parameters
indexIndex for the precision/recall curve point or -1 to use the index corresponding to the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.

◆ GetROCPoint() [1/2]

eVision.EasyDeepLearning.EROCPoint eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint ( self)

ROC (Receiver Operating Characteristic) point.

A ROC point is a point from the ROC curve which is the plot of the true positive rate against the false positive rate (see eVision.EasyDeepLearning.EConfusionMatrixElement) obtained at various classification threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold).

The ROC points are strictly ordered by decreasing threshold order meaning the true positive rate and false positive rate (see eVision.EasyDeepLearning.EConfusionMatrixElement) are sorted in increasing order.

Remarks:

Each ROC point corresponds to a different classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers).

It means that the ROC curve is the perfect tool to choose a threshold depending on the false and true positive rate values that best suit your application.

◆ GetROCPoint() [2/2]

eVision.EasyDeepLearning.EROCPoint eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint ( self,
int index )

ROC (Receiver Operating Characteristic) point.

A ROC point is a point from the ROC curve which is the plot of the true positive rate against the false positive rate (see eVision.EasyDeepLearning.EConfusionMatrixElement) obtained at various classification threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold).

The ROC points are strictly ordered by decreasing threshold order meaning the true positive rate and false positive rate (see eVision.EasyDeepLearning.EConfusionMatrixElement) are sorted in increasing order.

Remarks:

Each ROC point corresponds to a different classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers).

It means that the ROC curve is the perfect tool to choose a threshold depending on the false and true positive rate values that best suit your application.

Parameters
indexThe index of the classifier to use. If the index is equal to '-1', the index corresponding to eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold will be used.

◆ IsDefectDetectionMetricsValid()

bool eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.IsDefectDetectionMetricsValid ( self)

Whether the defect detection metrics are valid or not.

Defect detection metrics are completely valid when the metrics has results for at least one good and one defective images. Some metrics might be valid with only defective or good results.

◆ Load() [1/2]

None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.Load ( self,
eVision.ESerializer | None serializer )

◆ Load() [2/2]

None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.Load ( self,
str path )

◆ NumberOfClassifiers()

int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers ( self)

Number of different possible classifiers.

Each classifier is obtained by choosing a different classification threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold and corresponds to a point in the ROC curve (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint).

Remarks:

The number of classifiers is equal to the number of results added to the metrics that have a unique classification score (i.e. different from the classification score of all the other results) plus one classifier with a threshold below all the results's score. This classifier might be invalid if the lowest score is equal to 0 (i.e. the threshold will be negative which is invalid).

◆ NumDefectiveSample()

int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumDefectiveSample ( self)

Number of defective sample added to the metric.

◆ NumGoodSample()

int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumGoodSample ( self)

Number of good sample added to the metric.

◆ NumPrecisionRecallCurvePoint()

int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumPrecisionRecallCurvePoint ( self)

Number of point in the precision/recall curve.

Remarks:

The precision/recall curve is defined when there is at least one ground truth positive sample in the metric. The number of point in the precision/recall curve is equal to the number of positive sample plus 1.

◆ PrecisionRecallCurveIndex()

int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.PrecisionRecallCurveIndex ( self)

Index in the precision/recall curve for the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.

The value of the index is between 0 and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumPrecisionRecallCurvePoint - 1. The index is -1 if the precision/recall curve is not defined.

◆ Save() [1/2]

None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.Save ( self,
eVision.ESerializer | None serializer )

◆ Save() [2/2]

None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.Save ( self,
str path )