Collection of metrics used to evaluate a defect detection problem where the detection is based on the thresholding of a score produced by the underlying deep learning tool, e.g. a EUnsupervisedSegmenter tool or a ESupervisedSegmenter tool. More...
#include <eVision/EasyDeepLearning/EDeepLearningDefectDetectionMetrics.h>
Public Member Functions | |
| EDeepLearningDefectDetectionMetrics () | |
| Constructs an empty EDeepLearningDefectDetectionMetrics object. | |
| EDeepLearningDefectDetectionMetrics (const eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics &other) | |
| Copy constructor. | |
| virtual | ~EDeepLearningDefectDetectionMetrics () throw () |
| eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics & | operator= (const eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics &other) |
| Copy operator. | |
| virtual void | Load (const std::string &path) |
| Loads the defect detection metrics. The given ESerializer must have been created for reading. | |
| virtual void | Load (eVision::ESerializer *serializer) |
| Loads the defect detection metrics. The given ESerializer must have been created for reading. | |
| virtual void | Save (const std::string &path) |
| Saves the defect detection metrics. The given ESerializer must have been created for writing. | |
| virtual void | Save (eVision::ESerializer *serializer) |
| Saves the defect detection metrics. The given ESerializer must have been created for writing. | |
| bool | IsDefectDetectionMetricsValid () const |
| Whether the defect detection metrics are valid or not. Defect detection metrics are completely valid when the metrics has results for at least one good and one defective images. Some metrics might be valid with only defective or good results. | |
| int | GetNumDefectiveSample () const |
| Number of defective sample added to the metric. | |
| int | GetNumGoodSample () const |
| Number of good sample added to the metric. | |
| float | GetAccuracy (int index) const |
| The accuracy of the segmenter. | |
| float | GetAccuracy () const |
| The accuracy of the segmenter. | |
| float | GetBestAccuracy () const |
| Best achievable accuracy. | |
| float | GetBestBalancedAccuracy () const |
| Best achievable balanced accuracy. | |
| float | GetBestWeightedAccuracy (float goodWeight, float badWeight) const |
| Best achievable weighted accuracy. | |
| float | GetBestAccuracyClassificationThreshold () const |
| Classification threshold giving the best achievable accuracy (see EDeepLearningDefectDetectionMetrics::GetBestAccuracy). | |
| float | GetBestBalancedAccuracyClassificationThreshold () const |
| Classification threshold giving the best achievable balanced accuracy (see EDeepLearningDefectDetectionMetrics::GetBestBalancedAccuracy). | |
| float | GetBestWeightedAccuracyClassificationThreshold (float goodWeight, float badWeight) const |
| Classification threshold giving the best achievable weighted accuracy (see EDeepLearningDefectDetectionMetrics::GetBestWeightedAccuracy). | |
| uint32_t | GetConfusion (eVision::EasyDeepLearning::EConfusionMatrixElement element, int index) const |
| Confusion value of one label with another. | |
| uint32_t | GetConfusion (eVision::EasyDeepLearning::EConfusionMatrixElement element) const |
| Confusion value of one label with another. | |
| float | GetAreaUnderROCCurve () const |
| The area under ROC curve (AUC) of the classifier (see EDeepLearningDefectDetectionMetrics::GetROCPoint). It's value is between 0 and 1. | |
| float | GetAveragePrecision () const |
| Average precision. | |
| float | GetBestFScore () const |
| Best F1-Score. | |
| float | GetBestFScoreThreshold () const |
| Classification threshold that yields the best F1-Score. | |
| float | GetRecall (int index) const |
| Recall for the current threshold. | |
| float | GetRecall () const |
| Recall for the current threshold. | |
| float | GetPrecision (int index) const |
| Precision for the current threshold. | |
| float | GetPrecision () const |
| Precision for the current threshold. | |
| float | GetFScore (int index) const |
| The F1-Score for the current threshold. | |
| float | GetFScore () const |
| The F1-Score for the current threshold. | |
| int | GetPrecisionRecallCurveIndex () const |
| Index in the precision/recall curve for the current EDeepLearningDefectDetectionMetrics::GetClassificationThreshold. | |
| int | GetNumPrecisionRecallCurvePoint () const |
| Number of point in the precision/recall curve. | |
| eVision::EasyDeepLearning::EROCPoint | GetPrecisionRecallCurvePoint (int index) const |
| Get a point on the precision/recall curve. | |
| eVision::EasyDeepLearning::EROCPoint | GetPrecisionRecallCurvePoint () const |
| Get a point on the precision/recall curve. | |
| int | GetNumberOfClassifiers () const |
| Number of different possible classifiers. | |
| eVision::EasyDeepLearning::EROCPoint | GetROCPoint (int index) const |
| ROC (Receiver Operating Characteristic) point. | |
| eVision::EasyDeepLearning::EROCPoint | GetROCPoint () const |
| ROC (Receiver Operating Characteristic) point. | |
| float | GetClassificationThreshold () const |
| Classification threshold. | |
| void | SetClassificationThreshold (float threshold) |
| Classification threshold. | |
| EDeepLearningDefectDetectionMetrics (CppWrapperInternal::InternalConstructor) | |
Public Attributes | |
| CppWrapperInternal::WrappedClass | wrappedClass_ |
Collection of metrics used to evaluate a defect detection problem where the detection is based on the thresholding of a score produced by the underlying deep learning tool, e.g. a EUnsupervisedSegmenter tool or a ESupervisedSegmenter tool.
These metrics are only valid when results for good and defective images are included in the metrics (see EDeepLearningDefectDetectionMetrics::IsDefectDetectionMetricsValid). The definition of what is considered a good or a defective image depends on the deep learning tool used.
The defect detection metrics are separated in two main categories:
The metrics related to the ROC curve are:
The metrics related to the precision/recall curve are:
The ROC and Precision/Recall curve are both obtained by computing some metrics for different values of the EDeepLearningDefectDetectionMetrics::GetClassificationThreshold. Thus, each point on the curves gives different metrics, except for the area under the ROC curve and the average precision.
By default, the value of the metrics corresponds to the classification threshold of the corresponding deep learning tool. However, you can specify an index to retrieve the value of the metrics for other value of the EDeepLearningDefectDetectionMetrics::GetClassificationThreshold. Metrics based on the ROC curve are indexed between 0 and EDeepLearningDefectDetectionMetrics::GetNumberOfClassifiers-1 and metrics based on the Precision/Recall curve are indexed between 0 and EDeepLearningDefectDetectionMetrics::GetNumPrecisionRecallCurvePoint-1.
License(s): BasicTypes
| eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::EDeepLearningDefectDetectionMetrics | ( | ) |
Constructs an empty EDeepLearningDefectDetectionMetrics object.
| eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::EDeepLearningDefectDetectionMetrics | ( | const eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics & | other | ) |
Copy constructor.
| other | Other object |
|
virtual | ||||||||||||
| eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::EDeepLearningDefectDetectionMetrics | ( | CppWrapperInternal::InternalConstructor | ) |
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetAccuracy | ( | ) | const |
The accuracy of the segmenter.
The accuracy is the number of images that were correctly classified over the total number of images that was used to evaluate the classifier.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetAccuracy | ( | int | index | ) | const |
The accuracy of the segmenter.
The accuracy is the number of images that were correctly classified over the total number of images that was used to evaluate the classifier.
| index | The index of the classifier to use. If the index is equal to '-1', the index corresponding to EUnsupervisedSegmenter::SetClassificationThreshold will be used. |
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetAreaUnderROCCurve | ( | ) | const |
The area under ROC curve (AUC) of the classifier (see EDeepLearningDefectDetectionMetrics::GetROCPoint). It's value is between 0 and 1.
In the context of unsupervised segmentation, the AUC is equal to the probability that good images will have a lower reconstruction error than defective images.
This metrics measure discrimination capacity of a model. It tells how much the model is capable of distinguishing between classes. The higher the AUC, the better the model is.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetAveragePrecision | ( | ) | const |
Average precision.
The average precision is the area under the precision-recall curve. The precision is the ratio between the number of true positive and the number of predicted positive and the recall, also called the true positive rate, is the ratio between the number of true positive and the number of positive sample.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetBestAccuracy | ( | ) | const |
Best achievable accuracy.
The classification threshold corresponding to this accuracy is given by EDeepLearningDefectDetectionMetrics::GetBestAccuracyClassificationThreshold.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetBestAccuracyClassificationThreshold | ( | ) | const |
Classification threshold giving the best achievable accuracy (see EDeepLearningDefectDetectionMetrics::GetBestAccuracy).
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetBestBalancedAccuracy | ( | ) | const |
Best achievable balanced accuracy.
The classification threshold corresponding to this accuracy is given by EDeepLearningDefectDetectionMetrics::GetBestBalancedAccuracyClassificationThreshold.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetBestBalancedAccuracyClassificationThreshold | ( | ) | const |
Classification threshold giving the best achievable balanced accuracy (see EDeepLearningDefectDetectionMetrics::GetBestBalancedAccuracy).
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetBestFScore | ( | ) | const |
Best F1-Score.
The F1-Score is the harmonic mean of the precision and recall (true positive rate).
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetBestFScoreThreshold | ( | ) | const |
Classification threshold that yields the best F1-Score.
The F1-Score is the harmonic mean of the precision and recall (true positive rate).
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetBestWeightedAccuracy | ( | float | goodWeight, |
| float | badWeight ) const |
Best achievable weighted accuracy.
The weighted accuracy is the weighted average of the true positive rate and the true negative rate (which is equal to 1 minus the false positive rate). See EROCPoint.
The classification threshold corresponding to this accuracy is given by EDeepLearningDefectDetectionMetrics::GetBestWeightedAccuracyClassificationThreshold.
| goodWeight | Weight for the good label |
| badWeight | Weight for the bad label |
When using a dataset as the source for the label weights, the good weight is the weight of the "good" label and the bad weight is the sum of the weights of all the other labels.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetBestWeightedAccuracyClassificationThreshold | ( | float | goodWeight, |
| float | badWeight ) const |
Classification threshold giving the best achievable weighted accuracy (see EDeepLearningDefectDetectionMetrics::GetBestWeightedAccuracy).
| goodWeight | Weight for the good label |
| badWeight | Weight for the bad label |
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetClassificationThreshold | ( | ) | const |
Classification threshold.
By default, the classification threshold will be equal to the classification threshold of the last unsupervised segmenter used to produce the results that compose this metric.
Some metrics such as EDeepLearningDefectDetectionMetrics::GetROCPoint, EDeepLearningDefectDetectionMetrics::GetAccuracy or EDeepLearningDefectDetectionMetrics::GetConfusion depends on the classification threshold. By default, these methods will returns the metric corresponding to the classification threshold.
Modifying the classification threshold in this class doesn't modify the classification threshold of the unsupervised segmenter.
| uint32_t eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetConfusion | ( | eVision::EasyDeepLearning::EConfusionMatrixElement | element | ) | const |
Confusion value of one label with another.
The confusion value of a label with another is the number of images belonging to this label that are classified as belonging to the other label.
For a EDeepLearningDefectDetectionMetrics there are only 2 labels (good and defective) so the confusion matrix is only composed of 4 values which are called matrix element (see EConfusionMatrixElement).
The confusion matrix is computed for a given threshold (see EUnsupervisedSegmenter::SetClassificationThreshold) which means an index can be passed to the method (see EDeepLearningDefectDetectionMetrics::GetNumberOfClassifiers).
| element | The element from which to obtain the confusion value |
| uint32_t eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetConfusion | ( | eVision::EasyDeepLearning::EConfusionMatrixElement | element, |
| int | index ) const |
Confusion value of one label with another.
The confusion value of a label with another is the number of images belonging to this label that are classified as belonging to the other label.
For a EDeepLearningDefectDetectionMetrics there are only 2 labels (good and defective) so the confusion matrix is only composed of 4 values which are called matrix element (see EConfusionMatrixElement).
The confusion matrix is computed for a given threshold (see EUnsupervisedSegmenter::SetClassificationThreshold) which means an index can be passed to the method (see EDeepLearningDefectDetectionMetrics::GetNumberOfClassifiers).
| element | The element from which to obtain the confusion value |
| index | The index of the classifier to use. If the index is '-1', the index corresponding to EUnsupervisedSegmenter::SetClassificationThreshold will be used. |
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetFScore | ( | ) | const |
The F1-Score for the current threshold.
The F1-Score is the harmonic mean of EDeepLearningDefectDetectionMetrics::GetRecall and EDeepLearningDefectDetectionMetrics::GetPrecision.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetFScore | ( | int | index | ) | const |
The F1-Score for the current threshold.
The F1-Score is the harmonic mean of EDeepLearningDefectDetectionMetrics::GetRecall and EDeepLearningDefectDetectionMetrics::GetPrecision.
| index | Index for the precision/recall curve point or -1 to use the index corresponding to the current EDeepLearningDefectDetectionMetrics::GetClassificationThreshold. |
| int eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetNumberOfClassifiers | ( | ) | const |
Number of different possible classifiers.
Each classifier is obtained by choosing a different classification threshold (see EUnsupervisedSegmenter::SetClassificationThreshold and corresponds to a point in the ROC curve (see EDeepLearningDefectDetectionMetrics::GetROCPoint).
The number of classifiers is equal to the number of results added to the metrics that have a unique classification score (i.e. different from the classification score of all the other results) plus one classifier with a threshold below all the results's score. This classifier might be invalid if the lowest score is equal to 0 (i.e. the threshold will be negative which is invalid).
| int eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetNumDefectiveSample | ( | ) | const |
Number of defective sample added to the metric.
| int eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetNumGoodSample | ( | ) | const |
Number of good sample added to the metric.
| int eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetNumPrecisionRecallCurvePoint | ( | ) | const |
Number of point in the precision/recall curve.
The precision/recall curve is defined when there is at least one ground truth positive sample in the metric. The number of point in the precision/recall curve is equal to the number of positive sample plus 1.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetPrecision | ( | ) | const |
Precision for the current threshold.
The precision is the proportion of detected defective instances that were correctly identified as such.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetPrecision | ( | int | index | ) | const |
Precision for the current threshold.
The precision is the proportion of detected defective instances that were correctly identified as such.
| index | Index for the precision/recall curve point or -1 to use the index corresponding to the current EDeepLearningDefectDetectionMetrics::GetClassificationThreshold. |
| int eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetPrecisionRecallCurveIndex | ( | ) | const |
Index in the precision/recall curve for the current EDeepLearningDefectDetectionMetrics::GetClassificationThreshold.
The value of the index is between 0 and EDeepLearningDefectDetectionMetrics::GetNumPrecisionRecallCurvePoint - 1. The index is -1 if the precision/recall curve is not defined.
| eVision::EasyDeepLearning::EROCPoint eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetPrecisionRecallCurvePoint | ( | ) | const |
Get a point on the precision/recall curve.
The precision/recall curve is defined when there is at least one ground truth positive sample in the metric.
| eVision::EasyDeepLearning::EROCPoint eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetPrecisionRecallCurvePoint | ( | int | index | ) | const |
Get a point on the precision/recall curve.
| index | Index for the precision/recall curve point or -1 to use the index corresponding to the current EDeepLearningDefectDetectionMetrics::GetClassificationThreshold. |
The precision/recall curve is defined when there is at least one ground truth positive sample in the metric.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetRecall | ( | ) | const |
Recall for the current threshold.
It is the proportion of defective instances that were correctly identified as such. It is also called the true positive rate.
| float eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetRecall | ( | int | index | ) | const |
Recall for the current threshold.
It is the proportion of defective instances that were correctly identified as such. It is also called the true positive rate.
| index | Index for the precision/recall curve point or -1 to use the index corresponding to the current EDeepLearningDefectDetectionMetrics::GetClassificationThreshold. |
| eVision::EasyDeepLearning::EROCPoint eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetROCPoint | ( | ) | const |
ROC (Receiver Operating Characteristic) point.
A ROC point is a point from the ROC curve which is the plot of the true positive rate against the false positive rate (see EConfusionMatrixElement) obtained at various classification threshold (see EUnsupervisedSegmenter::SetClassificationThreshold).
The ROC points are strictly ordered by decreasing threshold order meaning the true positive rate and false positive rate (see EConfusionMatrixElement) are sorted in increasing order.
Each ROC point corresponds to a different classifier (see EDeepLearningDefectDetectionMetrics::GetNumberOfClassifiers).
It means that the ROC curve is the perfect tool to choose a threshold depending on the false and true positive rate values that best suit your application.
| eVision::EasyDeepLearning::EROCPoint eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::GetROCPoint | ( | int | index | ) | const |
ROC (Receiver Operating Characteristic) point.
A ROC point is a point from the ROC curve which is the plot of the true positive rate against the false positive rate (see EConfusionMatrixElement) obtained at various classification threshold (see EUnsupervisedSegmenter::SetClassificationThreshold).
The ROC points are strictly ordered by decreasing threshold order meaning the true positive rate and false positive rate (see EConfusionMatrixElement) are sorted in increasing order.
| index | The index of the classifier to use. If the index is equal to '-1', the index corresponding to EUnsupervisedSegmenter::SetClassificationThreshold will be used. |
Each ROC point corresponds to a different classifier (see EDeepLearningDefectDetectionMetrics::GetNumberOfClassifiers).
It means that the ROC curve is the perfect tool to choose a threshold depending on the false and true positive rate values that best suit your application.
| bool eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::IsDefectDetectionMetricsValid | ( | ) | const |
Whether the defect detection metrics are valid or not. Defect detection metrics are completely valid when the metrics has results for at least one good and one defective images. Some metrics might be valid with only defective or good results.
|
virtual |
Loads the defect detection metrics. The given ESerializer must have been created for reading.
| path | The file path. |
Reimplemented in eVision::EasyDeepLearning::ESupervisedSegmenterMetrics, and eVision::EasyDeepLearning::EUnsupervisedSegmenterMetrics.
|
virtual |
Loads the defect detection metrics. The given ESerializer must have been created for reading.
| serializer | The serializer. |
Reimplemented in eVision::EasyDeepLearning::ESupervisedSegmenterMetrics, and eVision::EasyDeepLearning::EUnsupervisedSegmenterMetrics.
| eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics & eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::operator= | ( | const eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics & | other | ) |
Copy operator.
| other | Other object |
|
virtual |
Saves the defect detection metrics. The given ESerializer must have been created for writing.
| path | The file path. |
Reimplemented in eVision::EasyDeepLearning::ESupervisedSegmenterMetrics, and eVision::EasyDeepLearning::EUnsupervisedSegmenterMetrics.
|
virtual |
Saves the defect detection metrics. The given ESerializer must have been created for writing.
| serializer | The ESerializer object that is written to. |
Reimplemented in eVision::EasyDeepLearning::ESupervisedSegmenterMetrics, and eVision::EasyDeepLearning::EUnsupervisedSegmenterMetrics.
| void eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::SetClassificationThreshold | ( | float | threshold | ) |
Classification threshold.
By default, the classification threshold will be equal to the classification threshold of the last unsupervised segmenter used to produce the results that compose this metric.
Some metrics such as EDeepLearningDefectDetectionMetrics::GetROCPoint, EDeepLearningDefectDetectionMetrics::GetAccuracy or EDeepLearningDefectDetectionMetrics::GetConfusion depends on the classification threshold. By default, these methods will returns the metric corresponding to the classification threshold.
Modifying the classification threshold in this class doesn't modify the classification threshold of the unsupervised segmenter.
| CppWrapperInternal::WrappedClass eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics::wrappedClass_ |