Collection of metrics used to evaluate a defect detection problem where the detection is based on the thresholding of a score produced by the underlying deep learning tool, e.g. More...

Public Member Functions | |
| None | __init__ (self) |
| Constructs an empty eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics object. | |
| None | __init__ (self, eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics other) |
| Copy constructor. | |
| None | Load (self, str path) |
| Loads the defect detection metrics. | |
| None | Load (self, eVision.ESerializer|None serializer) |
| Loads the defect detection metrics. | |
| None | Save (self, str path) |
| Saves the defect detection metrics. | |
| None | Save (self, eVision.ESerializer|None serializer) |
| Saves the defect detection metrics. | |
| bool | IsDefectDetectionMetricsValid (self) |
| Whether the defect detection metrics are valid or not. | |
| float | GetAccuracy (self, int index) |
| The accuracy of the segmenter. | |
| float | GetAccuracy (self) |
| The accuracy of the segmenter. | |
| float | GetBestWeightedAccuracy (self, float goodWeight, float badWeight) |
| Best achievable weighted accuracy. | |
| float | GetBestWeightedAccuracyClassificationThreshold (self, float goodWeight, float badWeight) |
| Classification threshold giving the best achievable weighted accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy). | |
| int | GetConfusion (self, eVision.EasyDeepLearning.EConfusionMatrixElement element, int index) |
| Confusion value of one label with another. | |
| int | GetConfusion (self, eVision.EasyDeepLearning.EConfusionMatrixElement element) |
| Confusion value of one label with another. | |
| float | GetRecall (self, int index) |
| Recall for the current threshold. | |
| float | GetRecall (self) |
| Recall for the current threshold. | |
| float | GetPrecision (self, int index) |
| Precision for the current threshold. | |
| float | GetPrecision (self) |
| Precision for the current threshold. | |
| float | GetFScore (self, int index) |
| The F1-Score for the current threshold. | |
| float | GetFScore (self) |
| The F1-Score for the current threshold. | |
| eVision.EasyDeepLearning.EROCPoint | GetPrecisionRecallCurvePoint (self, int index) |
| Get a point on the precision/recall curve. | |
| eVision.EasyDeepLearning.EROCPoint | GetPrecisionRecallCurvePoint (self) |
| Get a point on the precision/recall curve. | |
| eVision.EasyDeepLearning.EROCPoint | GetROCPoint (self, int index) |
| ROC (Receiver Operating Characteristic) point. | |
| eVision.EasyDeepLearning.EROCPoint | GetROCPoint (self) |
| ROC (Receiver Operating Characteristic) point. | |
| int | NumDefectiveSample (self) |
| Number of defective sample added to the metric. | |
| int | NumGoodSample (self) |
| Number of good sample added to the metric. | |
| float | BestAccuracy (self) |
| Best achievable accuracy. | |
| float | BestBalancedAccuracy (self) |
| Best achievable balanced accuracy. | |
| float | BestAccuracyClassificationThreshold (self) |
| Classification threshold giving the best achievable accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracy). | |
| float | BestBalancedAccuracyClassificationThreshold (self) |
| Classification threshold giving the best achievable balanced accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy). | |
| float | AreaUnderROCCurve (self) |
| The area under ROC curve (AUC) of the classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint). | |
| float | AveragePrecision (self) |
| Average precision. | |
| float | BestFScore (self) |
| Best F1-Score. | |
| float | BestFScoreThreshold (self) |
| Classification threshold that yields the best F1-Score. | |
| int | PrecisionRecallCurveIndex (self) |
| Index in the precision/recall curve for the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. | |
| int | NumPrecisionRecallCurvePoint (self) |
| Number of point in the precision/recall curve. | |
| int | NumberOfClassifiers (self) |
| Number of different possible classifiers. | |
| float | ClassificationThreshold (self) |
| Classification threshold. | |
| ClassificationThreshold (self, float value) | |
| Classification threshold. | |
Collection of metrics used to evaluate a defect detection problem where the detection is based on the thresholding of a score produced by the underlying deep learning tool, e.g.
a eVision.EasyDeepLearning.EUnsupervisedSegmenter tool or a eVision.EasyDeepLearning.ESupervisedSegmenter tool.
These metrics are only valid when results for good and defective images are included in the metrics (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.IsDefectDetectionMetricsValid). The definition of what is considered a good or a defective image depends on the deep learning tool used.
The defect detection metrics are separated in two main categories:
The metrics related to the ROC curve are:
The metrics related to the precision/recall curve are:
The ROC and Precision/Recall curve are both obtained by computing some metrics for different values of the eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. Thus, each point on the curves gives different metrics, except for the area under the ROC curve and the average precision.
By default, the value of the metrics corresponds to the classification threshold of the corresponding deep learning tool. However, you can specify an index to retrieve the value of the metrics for other value of the eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. Metrics based on the ROC curve are indexed between 0 and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers-1 and metrics based on the Precision/Recall curve are indexed between 0 and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumPrecisionRecallCurvePoint-1.
Remarks:
License(s): BasicTypes
| None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.__init__ | ( | self | ) |
Constructs an empty eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics object.
| None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.__init__ | ( | self, | |
| eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics | other ) |
Copy constructor.
| other | Other object |
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.AreaUnderROCCurve | ( | self | ) |
The area under ROC curve (AUC) of the classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint).
It's value is between 0 and 1.
In the context of unsupervised segmentation, the AUC is equal to the probability that good images will have a lower reconstruction error than defective images.
This metrics measure discrimination capacity of a model. It tells how much the model is capable of distinguishing between classes. The higher the AUC, the better the model is.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.AveragePrecision | ( | self | ) |
Average precision.
The average precision is the area under the precision-recall curve. The precision is the ratio between the number of true positive and the number of predicted positive and the recall, also called the true positive rate, is the ratio between the number of true positive and the number of positive sample.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracy | ( | self | ) |
Best achievable accuracy.
The classification threshold corresponding to this accuracy is given by eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracyClassificationThreshold.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracyClassificationThreshold | ( | self | ) |
Classification threshold giving the best achievable accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestAccuracy).
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy | ( | self | ) |
Best achievable balanced accuracy.
The classification threshold corresponding to this accuracy is given by eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracyClassificationThreshold.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracyClassificationThreshold | ( | self | ) |
Classification threshold giving the best achievable balanced accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy).
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestFScore | ( | self | ) |
Best F1-Score.
The F1-Score is the harmonic mean of the precision and recall (true positive rate).
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.BestFScoreThreshold | ( | self | ) |
Classification threshold that yields the best F1-Score.
The F1-Score is the harmonic mean of the precision and recall (true positive rate).
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold | ( | self | ) |
Classification threshold.
By default, the classification threshold will be equal to the classification threshold of the last unsupervised segmenter used to produce the results that compose this metric.
Some metrics such as eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint, eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetAccuracy or eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetConfusion depends on the classification threshold. By default, these methods will returns the metric corresponding to the classification threshold.
Remarks:
Modifying the classification threshold in this class doesn't modify the classification threshold of the unsupervised segmenter.
| eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold | ( | self, | |
| float | value ) |
Classification threshold.
By default, the classification threshold will be equal to the classification threshold of the last unsupervised segmenter used to produce the results that compose this metric.
Some metrics such as eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint, eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetAccuracy or eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetConfusion depends on the classification threshold. By default, these methods will returns the metric corresponding to the classification threshold.
Remarks:
Modifying the classification threshold in this class doesn't modify the classification threshold of the unsupervised segmenter.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetAccuracy | ( | self | ) |
The accuracy of the segmenter.
The accuracy is the number of images that were correctly classified over the total number of images that was used to evaluate the classifier.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetAccuracy | ( | self, | |
| int | index ) |
The accuracy of the segmenter.
The accuracy is the number of images that were correctly classified over the total number of images that was used to evaluate the classifier.
| index | The index of the classifier to use. If the index is equal to '-1', the index corresponding to eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold will be used. |
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy | ( | self, | |
| float | goodWeight, | ||
| float | badWeight ) |
Best achievable weighted accuracy.
The weighted accuracy is the weighted average of the true positive rate and the true negative rate (which is equal to 1 minus the false positive rate). See eVision.EasyDeepLearning.EROCPoint.
The classification threshold corresponding to this accuracy is given by eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracyClassificationThreshold.
Remarks:
When using a dataset as the source for the label weights, the good weight is the weight of the "good" label and the bad weight is the sum of the weights of all the other labels.
| goodWeight | Weight for the good label |
| badWeight | Weight for the bad label |
Reimplemented in eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics, and eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracyClassificationThreshold | ( | self, | |
| float | goodWeight, | ||
| float | badWeight ) |
Classification threshold giving the best achievable weighted accuracy (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy).
| goodWeight | Weight for the good label |
| badWeight | Weight for the bad label |
Reimplemented in eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics, and eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics.
| int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetConfusion | ( | self, | |
| eVision.EasyDeepLearning.EConfusionMatrixElement | element ) |
Confusion value of one label with another.
The confusion value of a label with another is the number of images belonging to this label that are classified as belonging to the other label.
For a eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics there are only 2 labels (good and defective) so the confusion matrix is only composed of 4 values which are called matrix element (see eVision.EasyDeepLearning.EConfusionMatrixElement).
The confusion matrix is computed for a given threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold) which means an index can be passed to the method (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers).
| element | The element from which to obtain the confusion value |
| int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetConfusion | ( | self, | |
| eVision.EasyDeepLearning.EConfusionMatrixElement | element, | ||
| int | index ) |
Confusion value of one label with another.
The confusion value of a label with another is the number of images belonging to this label that are classified as belonging to the other label.
For a eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics there are only 2 labels (good and defective) so the confusion matrix is only composed of 4 values which are called matrix element (see eVision.EasyDeepLearning.EConfusionMatrixElement).
The confusion matrix is computed for a given threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold) which means an index can be passed to the method (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers).
| element | The element from which to obtain the confusion value |
| index | The index of the classifier to use. If the index is '-1', the index corresponding to eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold will be used. |
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore | ( | self | ) |
The F1-Score for the current threshold.
The F1-Score is the harmonic mean of eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetRecall and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecision.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetFScore | ( | self, | |
| int | index ) |
The F1-Score for the current threshold.
The F1-Score is the harmonic mean of eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetRecall and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecision.
| index | Index for the precision/recall curve point or -1 to use the index corresponding to the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. |
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecision | ( | self | ) |
Precision for the current threshold.
The precision is the proportion of detected defective instances that were correctly identified as such.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecision | ( | self, | |
| int | index ) |
Precision for the current threshold.
The precision is the proportion of detected defective instances that were correctly identified as such.
| index | Index for the precision/recall curve point or -1 to use the index corresponding to the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. |
| eVision.EasyDeepLearning.EROCPoint eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecisionRecallCurvePoint | ( | self | ) |
Get a point on the precision/recall curve.
Remarks:
The precision/recall curve is defined when there is at least one ground truth positive sample in the metric.
| eVision.EasyDeepLearning.EROCPoint eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetPrecisionRecallCurvePoint | ( | self, | |
| int | index ) |
Get a point on the precision/recall curve.
Remarks:
The precision/recall curve is defined when there is at least one ground truth positive sample in the metric.
| index | Index for the precision/recall curve point or -1 to use the index corresponding to the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. |
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetRecall | ( | self | ) |
Recall for the current threshold.
It is the proportion of defective instances that were correctly identified as such. It is also called the true positive rate.
| float eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetRecall | ( | self, | |
| int | index ) |
Recall for the current threshold.
It is the proportion of defective instances that were correctly identified as such. It is also called the true positive rate.
| index | Index for the precision/recall curve point or -1 to use the index corresponding to the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold. |
| eVision.EasyDeepLearning.EROCPoint eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint | ( | self | ) |
ROC (Receiver Operating Characteristic) point.
A ROC point is a point from the ROC curve which is the plot of the true positive rate against the false positive rate (see eVision.EasyDeepLearning.EConfusionMatrixElement) obtained at various classification threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold).
The ROC points are strictly ordered by decreasing threshold order meaning the true positive rate and false positive rate (see eVision.EasyDeepLearning.EConfusionMatrixElement) are sorted in increasing order.
Remarks:
Each ROC point corresponds to a different classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers).
It means that the ROC curve is the perfect tool to choose a threshold depending on the false and true positive rate values that best suit your application.
| eVision.EasyDeepLearning.EROCPoint eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint | ( | self, | |
| int | index ) |
ROC (Receiver Operating Characteristic) point.
A ROC point is a point from the ROC curve which is the plot of the true positive rate against the false positive rate (see eVision.EasyDeepLearning.EConfusionMatrixElement) obtained at various classification threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold).
The ROC points are strictly ordered by decreasing threshold order meaning the true positive rate and false positive rate (see eVision.EasyDeepLearning.EConfusionMatrixElement) are sorted in increasing order.
Remarks:
Each ROC point corresponds to a different classifier (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers).
It means that the ROC curve is the perfect tool to choose a threshold depending on the false and true positive rate values that best suit your application.
| index | The index of the classifier to use. If the index is equal to '-1', the index corresponding to eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold will be used. |
| bool eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.IsDefectDetectionMetricsValid | ( | self | ) |
Whether the defect detection metrics are valid or not.
Defect detection metrics are completely valid when the metrics has results for at least one good and one defective images. Some metrics might be valid with only defective or good results.
| None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.Load | ( | self, | |
| eVision.ESerializer | None | serializer ) |
Loads the defect detection metrics.
The given eVision.ESerializer must have been created for reading.
| serializer | The serializer. |
Reimplemented in eVision.EasyDeepLearning.ESupervisedSegmenterMetrics, eVision.EasyDeepLearning.ESupervisedSegmenterMetrics, eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics, and eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics.
| None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.Load | ( | self, | |
| str | path ) |
Loads the defect detection metrics.
The given eVision.ESerializer must have been created for reading.
| path | The file path. |
Reimplemented in eVision.EasyDeepLearning.ESupervisedSegmenterMetrics, eVision.EasyDeepLearning.ESupervisedSegmenterMetrics, eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics, and eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics.
| int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumberOfClassifiers | ( | self | ) |
Number of different possible classifiers.
Each classifier is obtained by choosing a different classification threshold (see eVision.EasyDeepLearning.EUnsupervisedSegmenter.ClassificationThreshold and corresponds to a point in the ROC curve (see eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.GetROCPoint).
Remarks:
The number of classifiers is equal to the number of results added to the metrics that have a unique classification score (i.e. different from the classification score of all the other results) plus one classifier with a threshold below all the results's score. This classifier might be invalid if the lowest score is equal to 0 (i.e. the threshold will be negative which is invalid).
| int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumDefectiveSample | ( | self | ) |
Number of defective sample added to the metric.
| int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumGoodSample | ( | self | ) |
Number of good sample added to the metric.
| int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumPrecisionRecallCurvePoint | ( | self | ) |
Number of point in the precision/recall curve.
Remarks:
The precision/recall curve is defined when there is at least one ground truth positive sample in the metric. The number of point in the precision/recall curve is equal to the number of positive sample plus 1.
| int eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.PrecisionRecallCurveIndex | ( | self | ) |
Index in the precision/recall curve for the current eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.ClassificationThreshold.
The value of the index is between 0 and eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.NumPrecisionRecallCurvePoint - 1. The index is -1 if the precision/recall curve is not defined.
| None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.Save | ( | self, | |
| eVision.ESerializer | None | serializer ) |
Saves the defect detection metrics.
The given eVision.ESerializer must have been created for writing.
| serializer | The eVision.ESerializer object that is written to. |
Reimplemented in eVision.EasyDeepLearning.ESupervisedSegmenterMetrics, eVision.EasyDeepLearning.ESupervisedSegmenterMetrics, eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics, and eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics.
| None eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.Save | ( | self, | |
| str | path ) |
Saves the defect detection metrics.
The given eVision.ESerializer must have been created for writing.
| path | The file path. |
Reimplemented in eVision.EasyDeepLearning.ESupervisedSegmenterMetrics, eVision.EasyDeepLearning.ESupervisedSegmenterMetrics, eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics, and eVision.EasyDeepLearning.EUnsupervisedSegmenterMetrics.