Collection of metrics used to evaluate the state of a ESupervisedSegmenter tool. More...
#include <eVision/EasyDeepLearning/EasySegment/ESupervisedSegmenterMetrics.h>
Public Member Functions | |
| ESupervisedSegmenterMetrics () | |
| Constructs an ESupervisedSegmenterMetrics object. | |
| ESupervisedSegmenterMetrics (const eVision::EasyDeepLearning::ESupervisedSegmenterMetrics &other) | |
| Copy constructor of an ESupervisedSegmenterMetrics object. | |
| virtual | ~ESupervisedSegmenterMetrics () throw () |
| eVision::EasyDeepLearning::ESupervisedSegmenterMetrics & | operator= (const eVision::EasyDeepLearning::ESupervisedSegmenterMetrics &other) |
| Assignment operator. | |
| bool | operator== (const eVision::EasyDeepLearning::ESupervisedSegmenterMetrics &other) const |
| Equality operator. | |
| virtual void | Load (const std::string &path) |
| Loads an supervised segmentation metric. The given ESerializer must have been created for reading. | |
| virtual void | Load (eVision::ESerializer *serializer) |
| Loads an supervised segmentation metric. The given ESerializer must have been created for reading. | |
| virtual void | Save (const std::string &path) |
| Saves an supervised segmentation metric. The given ESerializer must have been created for writing. | |
| virtual void | Save (eVision::ESerializer *serializer) |
| Saves an supervised segmentation metric. The given ESerializer must have been created for writing. | |
| bool | IsValid () const |
| Indicates whether the object contains at least one result. | |
| float | GetWeightedPixelAccuracy () const |
| Weighted accuracy. | |
| float | GetBalancedPixelAccuracy () const |
| Balanced accuracy. | |
| float | GetPixelAccuracy () const |
| Accuracy. | |
| float | GetPixelLabelAccuracy (const std::string &label) const |
| Pixel label accuracy. | |
| float | GetWeightedError () const |
| Weighted error. | |
| float | GetBalancedError () const |
| Balanced error. | |
| float | GetError () const |
| Error. | |
| float | GetLabelError (const std::string &label) const |
| Label error. | |
| uint64_t | GetPixelConfusion (const std::string &trueLabel, const std::string &predictedLabel) const |
| Pixel-wise confusion between the given true and predicted labels. | |
| float | GetNormalizedPixelConfusion (const std::string &trueLabel, const std::string &predictedLabel) const |
| Pixel-wise normalized confusion between the given true and predicted labels. | |
| float | GetWeightedIntersectionOverUnion () const |
| Weighted Intersection over Union (IoU). | |
| float | GetBalancedIntersectionOverUnion () const |
| Balanced Intersection over Union (IoU). | |
| float | GetIntersectionOverUnion (const std::string &label) const |
| Intersection over union. | |
| uint64_t | GetGroundtruthBlobConfusion (const std::string &groundtruthLabel, const std::string &predictedLabel) const |
| Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label. | |
| uint64_t | GetGroundtruthBlobConfusion (int groundtruthLabelIndex, int predictedLabelIndex) const |
| Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label. | |
| uint64_t | GetPredictedBlobConfusion (const std::string &groundtruthLabel, const std::string &predictedLabel) const |
| Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label. | |
| uint64_t | GetPredictedBlobConfusion (int groundtruthLabelIndex, int predictedLabelIndex) const |
| Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label. | |
| float | GetBlobDetectionAveragePrecision () const |
| Average precision for blob detection. | |
| float | GetBlobDetectionBestFScore () const |
| Best F1-Score of the segmenter for blob detection. | |
| float | GetBlobDetectionBestFScoreThreshold () const |
| Classification threshold that achieves the best F1-Score for blob detection. | |
| float | GetBlobDetectionRecall () const |
| Recall for blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics::GetClassificationThreshold). | |
| float | GetBlobDetectionPrecision () const |
| Precision for blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics::GetClassificationThreshold). | |
| float | GetBlobDetectionFScore () const |
| F1-Score for defective blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics::GetClassificationThreshold). | |
| int | GetNumLabels () const |
| Number of labels recognized by the segmenter that produced the results aggregated in the metrics. | |
| std::string | GetLabel (int labelIdx) const |
| Label recognized by the segmenter that produced the results aggregated in the metrics. | |
| ESupervisedSegmenterMetrics (CppWrapperInternal::InternalConstructor) | |
| Public Member Functions inherited from eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics | |
| EDeepLearningDefectDetectionMetrics () | |
| Constructs an empty EDeepLearningDefectDetectionMetrics object. | |
| EDeepLearningDefectDetectionMetrics (const eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics &other) | |
| Copy constructor. | |
| virtual | ~EDeepLearningDefectDetectionMetrics () throw () |
| eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics & | operator= (const eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics &other) |
| Copy operator. | |
| bool | IsDefectDetectionMetricsValid () const |
| Whether the defect detection metrics are valid or not. Defect detection metrics are completely valid when the metrics has results for at least one good and one defective images. Some metrics might be valid with only defective or good results. | |
| int | GetNumDefectiveSample () const |
| Number of defective sample added to the metric. | |
| int | GetNumGoodSample () const |
| Number of good sample added to the metric. | |
| float | GetAccuracy (int index) const |
| The accuracy of the segmenter. | |
| float | GetAccuracy () const |
| The accuracy of the segmenter. | |
| float | GetBestAccuracy () const |
| Best achievable accuracy. | |
| float | GetBestBalancedAccuracy () const |
| Best achievable balanced accuracy. | |
| float | GetBestWeightedAccuracy (float goodWeight, float badWeight) const |
| Best achievable weighted accuracy. | |
| float | GetBestAccuracyClassificationThreshold () const |
| Classification threshold giving the best achievable accuracy (see EDeepLearningDefectDetectionMetrics::GetBestAccuracy). | |
| float | GetBestBalancedAccuracyClassificationThreshold () const |
| Classification threshold giving the best achievable balanced accuracy (see EDeepLearningDefectDetectionMetrics::GetBestBalancedAccuracy). | |
| float | GetBestWeightedAccuracyClassificationThreshold (float goodWeight, float badWeight) const |
| Classification threshold giving the best achievable weighted accuracy (see EDeepLearningDefectDetectionMetrics::GetBestWeightedAccuracy). | |
| uint32_t | GetConfusion (eVision::EasyDeepLearning::EConfusionMatrixElement element, int index) const |
| Confusion value of one label with another. | |
| uint32_t | GetConfusion (eVision::EasyDeepLearning::EConfusionMatrixElement element) const |
| Confusion value of one label with another. | |
| float | GetAreaUnderROCCurve () const |
| The area under ROC curve (AUC) of the classifier (see EDeepLearningDefectDetectionMetrics::GetROCPoint). It's value is between 0 and 1. | |
| float | GetAveragePrecision () const |
| Average precision. | |
| float | GetBestFScore () const |
| Best F1-Score. | |
| float | GetBestFScoreThreshold () const |
| Classification threshold that yields the best F1-Score. | |
| float | GetRecall (int index) const |
| Recall for the current threshold. | |
| float | GetRecall () const |
| Recall for the current threshold. | |
| float | GetPrecision (int index) const |
| Precision for the current threshold. | |
| float | GetPrecision () const |
| Precision for the current threshold. | |
| float | GetFScore (int index) const |
| The F1-Score for the current threshold. | |
| float | GetFScore () const |
| The F1-Score for the current threshold. | |
| int | GetPrecisionRecallCurveIndex () const |
| Index in the precision/recall curve for the current EDeepLearningDefectDetectionMetrics::GetClassificationThreshold. | |
| int | GetNumPrecisionRecallCurvePoint () const |
| Number of point in the precision/recall curve. | |
| eVision::EasyDeepLearning::EROCPoint | GetPrecisionRecallCurvePoint (int index) const |
| Get a point on the precision/recall curve. | |
| eVision::EasyDeepLearning::EROCPoint | GetPrecisionRecallCurvePoint () const |
| Get a point on the precision/recall curve. | |
| int | GetNumberOfClassifiers () const |
| Number of different possible classifiers. | |
| eVision::EasyDeepLearning::EROCPoint | GetROCPoint (int index) const |
| ROC (Receiver Operating Characteristic) point. | |
| eVision::EasyDeepLearning::EROCPoint | GetROCPoint () const |
| ROC (Receiver Operating Characteristic) point. | |
| float | GetClassificationThreshold () const |
| Classification threshold. | |
| void | SetClassificationThreshold (float threshold) |
| Classification threshold. | |
| EDeepLearningDefectDetectionMetrics (CppWrapperInternal::InternalConstructor) | |
Additional Inherited Members | |
| Public Attributes inherited from eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics | |
| CppWrapperInternal::WrappedClass | wrappedClass_ |
Collection of metrics used to evaluate the state of a ESupervisedSegmenter tool.
A metric is a value summarizing the quality of a collection of supervised segmentation results (see ESupervisedSegmenterResult) with respect to their ground truth. The ESupervisedSegmenterMetrics contains three types of metrics:
The pixel metrics are the error (see ESupervisedSegmenterMetrics::GetError), the pixel accuracy (see ESupervisedSegmenterMetrics::GetPixelAccuracy), the pixel confusion (see ESupervisedSegmenterMetrics::GetPixelConfusion), the pixel per-label accuracy (see ESupervisedSegmenterMetrics::GetPixelLabelAccuracy) and the intersection over union (see ESupervisedSegmenterMetrics::GetIntersectionOverUnion).
The blob based metrics are two confusion matrices (ESupervisedSegmenterMetrics::GetGroundtruthBlobConfusion and ESupervisedSegmenterMetrics::GetPredictedBlobConfusion), and various defect detection metrics (ESupervisedSegmenterMetrics::GetBlobDetectionRecall, ESupervisedSegmenterMetrics::GetBlobDetectionPrecision, and ESupervisedSegmenterMetrics::GetBlobDetectionFScore). Note that the metrics for defective blob detection are different from the metrics for defective image detection because, in the case of blob detection, we have no "good" ground truth results.
See EDeepLearningDefectDetectionMetrics for a description of the defect detection metrics. These metrics are available when EDeepLearningDefectDetectionMetrics::IsDefectDetectionMetricsValid is true, i.e. when both images that are entirely background and images with non-background pixels have been added to the metrics.
Most metrics depends upon the EDeepLearningDefectDetectionMetrics::GetClassificationThreshold.
License(s): BasicTypes
| eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::ESupervisedSegmenterMetrics | ( | ) |
Constructs an ESupervisedSegmenterMetrics object.
| eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::ESupervisedSegmenterMetrics | ( | const eVision::EasyDeepLearning::ESupervisedSegmenterMetrics & | other | ) |
Copy constructor of an ESupervisedSegmenterMetrics object.
| other | Reference to the ESupervisedSegmenterMetrics object that should be copied |
|
virtual | ||||||||||||
| eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::ESupervisedSegmenterMetrics | ( | CppWrapperInternal::InternalConstructor | ) |
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBalancedError | ( | ) | const |
Balanced error.
The balanced error is the weighted error (ESupervisedSegmenterMetrics::GetWeightedError) where each label is given an equal weight.
The error is also called the cross-entropy loss.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBalancedIntersectionOverUnion | ( | ) | const |
Balanced Intersection over Union (IoU).
The balanced intersection over union is the weighted intersection over union (ESupervisedSegmenterMetrics::GetWeightedIntersectionOverUnion) where each label is given equal weight.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBalancedPixelAccuracy | ( | ) | const |
Balanced accuracy.
The balanced accuracy is the weighted accuracy (ESupervisedSegmenterMetrics::GetWeightedPixelAccuracy) where each label is given an equal weight.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBlobDetectionAveragePrecision | ( | ) | const |
Average precision for blob detection.
Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label. The average precision for blob detection is the average of the precision (see ESupervisedSegmenterMetrics::GetBlobDetectionPrecision) over all the possible classification threshold values. As such, the average precision does not depend on a specific classification threshold.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBlobDetectionBestFScore | ( | ) | const |
Best F1-Score of the segmenter for blob detection.
See ESupervisedSegmenterMetrics::GetBlobDetectionBestFScoreThreshold for the corresponding threshold.
See EDeepLearningDefectDetectionMetrics::GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBlobDetectionBestFScoreThreshold | ( | ) | const |
Classification threshold that achieves the best F1-Score for blob detection.
See ESupervisedSegmenterMetrics::GetBlobDetectionBestFScore for the corresponding F1-Score.
See EDeepLearningDefectDetectionMetrics::GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBlobDetectionFScore | ( | ) | const |
F1-Score for defective blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics::GetClassificationThreshold).
See ESupervisedSegmenterMetrics::GetBlobDetectionBestFScore for the corresponding F1-Score.
See EDeepLearningDefectDetectionMetrics::GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBlobDetectionPrecision | ( | ) | const |
Precision for blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics::GetClassificationThreshold).
The precision is the proportion of detected defective blobs that match ground truth defective blobs.
Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetBlobDetectionRecall | ( | ) | const |
Recall for blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics::GetClassificationThreshold).
The recall is the proportion of ground truth defective blobs that are matched to predicted defective blobs.
Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetError | ( | ) | const |
Error.
The error is also called the cross-entropy loss.
| uint64_t eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetGroundtruthBlobConfusion | ( | const std::string & | groundtruthLabel, |
| const std::string & | predictedLabel ) const |
Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.
See ESupervisedSegmenterMetrics::GetPredictedBlobConfusion for the number of predicted blobs that match to these ground truth blobs.
| groundtruthLabel | Ground truth label |
| predictedLabel | Predicted label |
A ground truth blob is matched to the subset of predicted blobs from the same predicted label that has the best intersection over union with the ground truth blob. Otherwise, the ground truth blob is matched to background.
| uint64_t eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetGroundtruthBlobConfusion | ( | int | groundtruthLabelIndex, |
| int | predictedLabelIndex ) const |
Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.
See ESupervisedSegmenterMetrics::GetPredictedBlobConfusion for the number of predicted blobs that match to these ground truth blobs.
| groundtruthLabelIndex | Ground truth label index |
| predictedLabelIndex | Predicted label index |
A ground truth blob is matched to the subset of predicted blobs from the same predicted label that has the best intersection over union with the ground truth blob. Otherwise, the ground truth blob is matched to background.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetIntersectionOverUnion | ( | const std::string & | label | ) | const |
Intersection over union.
The intersection over union if the ratio between the number of correctly classified pixels from the given label to the total number of pixels that belongs to that label or are predicted as being from that label.
Assuming that the given label is the positive class, the intersection over union is expressed as IOU = TP / (TP + FP + FN) where TP is the number of true positive, FP the number of false positive (pixels that belongs to label but are not predicted as label) and FN the number of false negative (pixels predicted as label but that do not belong to label).
| label | Label |
The intersection over union is also called the Jaccard index.
| std::string eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetLabel | ( | int | labelIdx | ) | const |
Label recognized by the segmenter that produced the results aggregated in the metrics.
| labelIdx | Index of the label between 0 and ESupervisedSegmenterMetrics::GetNumLabels - 1. |
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetLabelError | ( | const std::string & | label | ) | const |
Label error.
| label | Label |
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetNormalizedPixelConfusion | ( | const std::string & | trueLabel, |
| const std::string & | predictedLabel ) const |
Pixel-wise normalized confusion between the given true and predicted labels.
The normalized confusion is the ratio of the number of pixels belonging to the 'trueLabel' that are classified as 'predictedLabel' to the total number of pixels belonging to the 'trueLabel'.
| trueLabel | True label |
| predictedLabel | Predicted label |
| int eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetNumLabels | ( | ) | const |
Number of labels recognized by the segmenter that produced the results aggregated in the metrics.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetPixelAccuracy | ( | ) | const |
Accuracy.
The accuracy is the ratio of the number of correctly classified pixels to the total number of pixels.
The accuracy will be skewed towards the most represented labels in the dataset. For example, if your dataset has 1% of defective pixels, the accuracy will mostly reflect the results on the 99% of good pixels and will say nothing about the capability of the model to detect the defect labels.
| uint64_t eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetPixelConfusion | ( | const std::string & | trueLabel, |
| const std::string & | predictedLabel ) const |
Pixel-wise confusion between the given true and predicted labels.
The confusion is the number of pixels belonging to the 'trueLabel' that are classified as 'predictedLabel'.
| trueLabel | True label |
| predictedLabel | Predicted label |
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetPixelLabelAccuracy | ( | const std::string & | label | ) | const |
Pixel label accuracy.
The label accuracy is the ratio of the number of correctly classified pixels of the given class to the total number of ground truth pixels from that class. If there are no ground truth pixels from the requested label, the method returns '-1'.
| label | Label |
| uint64_t eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetPredictedBlobConfusion | ( | const std::string & | groundtruthLabel, |
| const std::string & | predictedLabel ) const |
Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.
See ESupervisedSegmenterMetrics::GetGroundtruthBlobConfusion for the number of ground truth blobs that match to these predicted blobs.
| groundtruthLabel | Ground truth label |
| predictedLabel | Predicted label |
A predicted blob may be counted several times in the confusion matrix for predicted blobs:
| uint64_t eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetPredictedBlobConfusion | ( | int | groundtruthLabelIndex, |
| int | predictedLabelIndex ) const |
Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.
See ESupervisedSegmenterMetrics::GetGroundtruthBlobConfusion for the number of ground truth blobs that match to these predicted blobs.
| groundtruthLabelIndex | Ground truth label index |
| predictedLabelIndex | Predicted label index |
A predicted blob may be counted several times in the confusion matrix for predicted blobs:
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetWeightedError | ( | ) | const |
Weighted error.
The weighted error is the weighted average of each label error (ESupervisedSegmenterMetrics::GetLabelError) with respect to the dataset label weights.
The error is also called the cross-entropy loss.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetWeightedIntersectionOverUnion | ( | ) | const |
Weighted Intersection over Union (IoU).
The weighted intersection over union is the weighted averaged of each label intersection over union (see ESupervisedSegmenterMetrics::GetIntersectionOverUnion) with respect to the dataset label weights.
| float eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::GetWeightedPixelAccuracy | ( | ) | const |
Weighted accuracy.
The weighted accuracy is the weighted average of each label accuracy (ESupervisedSegmenterMetrics::GetPixelLabelAccuracy) with respect to the dataset label weights.
| bool eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::IsValid | ( | ) | const |
Indicates whether the object contains at least one result.
|
virtual |
Loads an supervised segmentation metric. The given ESerializer must have been created for reading.
| path | The file path. |
Reimplemented from eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics.
|
virtual |
Loads an supervised segmentation metric. The given ESerializer must have been created for reading.
| serializer | The serializer. |
Reimplemented from eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics.
| eVision::EasyDeepLearning::ESupervisedSegmenterMetrics & eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::operator= | ( | const eVision::EasyDeepLearning::ESupervisedSegmenterMetrics & | other | ) |
Assignment operator.
| other | Reference to the ESupervisedSegmenterMetrics object used for the assignment |
| bool eVision::EasyDeepLearning::ESupervisedSegmenterMetrics::operator== | ( | const eVision::EasyDeepLearning::ESupervisedSegmenterMetrics & | other | ) | const |
Equality operator.
| other | Reference to the ESupervisedSegmenterMetrics object |
|
virtual |
Saves an supervised segmentation metric. The given ESerializer must have been created for writing.
| path | The file path. |
Reimplemented from eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics.
|
virtual |
Saves an supervised segmentation metric. The given ESerializer must have been created for writing.
| serializer | The serializer. |
Reimplemented from eVision::EasyDeepLearning::EDeepLearningDefectDetectionMetrics.