Collection of metrics used to evaluate the state of a ESupervisedSegmenter tool. More...

Inheritance diagram for eVision.EasyDeepLearning.ESupervisedSegmenterMetrics:

[legend]

Collaboration diagram for eVision.EasyDeepLearning.ESupervisedSegmenterMetrics:

[legend]

Public Member Functions
	ESupervisedSegmenterMetrics ()
	Constructs an ESupervisedSegmenterMetrics object.
	ESupervisedSegmenterMetrics (eVision.EasyDeepLearning.ESupervisedSegmenterMetrics other)
	Copy constructor of an ESupervisedSegmenterMetrics object.
override void	Load (string path)
	Loads an supervised segmentation metric. The given ESerializer must have been created for reading.
override void	Load (eVision.ESerializer serializer)
	Loads an supervised segmentation metric. The given ESerializer must have been created for reading.
override void	Save (string path)
	Saves an supervised segmentation metric. The given ESerializer must have been created for writing.
override void	Save (eVision.ESerializer serializer)
	Saves an supervised segmentation metric. The given ESerializer must have been created for writing.
bool	IsValid ()
	Indicates whether the object contains at least one result.
float	GetPixelLabelAccuracy (string label)
	Pixel label accuracy.
float	GetLabelError (string label)
	Label error.
ulong	GetPixelConfusion (string trueLabel, string predictedLabel)
	Pixel-wise confusion between the given true and predicted labels.
float	GetNormalizedPixelConfusion (string trueLabel, string predictedLabel)
	Pixel-wise normalized confusion between the given true and predicted labels.
float	GetIntersectionOverUnion (string label)
	Intersection over union.
ulong	GetGroundtruthBlobConfusion (string groundtruthLabel, string predictedLabel)
	Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.
ulong	GetGroundtruthBlobConfusion (int groundtruthLabelIndex, int predictedLabelIndex)
	Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.
ulong	GetPredictedBlobConfusion (string groundtruthLabel, string predictedLabel)
	Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.
ulong	GetPredictedBlobConfusion (int groundtruthLabelIndex, int predictedLabelIndex)
	Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.
string	GetLabel (int labelIdx)
	Label recognized by the segmenter that produced the results aggregated in the metrics.
Public Member Functions inherited from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics
	EDeepLearningDefectDetectionMetrics ()
	Constructs an empty EDeepLearningDefectDetectionMetrics object.
	EDeepLearningDefectDetectionMetrics (eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics other)
	Copy constructor.
bool	IsDefectDetectionMetricsValid ()
	Whether the defect detection metrics are valid or not. Defect detection metrics are completely valid when the metrics has results for at least one good and one defective images. Some metrics might be valid with only defective or good results.
float	GetAccuracy (int index)
	The accuracy of the segmenter.
float	GetAccuracy ()
	The accuracy of the segmenter.
float	GetBestWeightedAccuracy (float goodWeight, float badWeight)
	Best achievable weighted accuracy.
float	GetBestWeightedAccuracyClassificationThreshold (float goodWeight, float badWeight)
	Classification threshold giving the best achievable weighted accuracy (see EDeepLearningDefectDetectionMetrics.GetBestWeightedAccuracy).
uint	GetConfusion (eVision.EasyDeepLearning.EConfusionMatrixElement element, int index)
	Confusion value of one label with another.
uint	GetConfusion (eVision.EasyDeepLearning.EConfusionMatrixElement element)
	Confusion value of one label with another.
float	GetRecall (int index)
	Recall for the current threshold.
float	GetRecall ()
	Recall for the current threshold.
float	GetPrecision (int index)
	Precision for the current threshold.
float	GetPrecision ()
	Precision for the current threshold.
float	GetFScore (int index)
	The F1-Score for the current threshold.
float	GetFScore ()
	The F1-Score for the current threshold.
eVision.EasyDeepLearning.EROCPoint	GetPrecisionRecallCurvePoint (int index)
	Get a point on the precision/recall curve.
eVision.EasyDeepLearning.EROCPoint	GetPrecisionRecallCurvePoint ()
	Get a point on the precision/recall curve.
eVision.EasyDeepLearning.EROCPoint	GetROCPoint (int index)
	ROC (Receiver Operating Characteristic) point.
eVision.EasyDeepLearning.EROCPoint	GetROCPoint ()
	ROC (Receiver Operating Characteristic) point.

Properties
float	WeightedPixelAccuracy `[get]`
	Weighted accuracy.
float	BalancedPixelAccuracy `[get]`
	Balanced accuracy.
float	PixelAccuracy `[get]`
	Accuracy.
float	WeightedError `[get]`
	Weighted error.
float	BalancedError `[get]`
	Balanced error.
float	Error `[get]`
	Error.
float	WeightedIntersectionOverUnion `[get]`
	Weighted Intersection over Union (IoU).
float	BalancedIntersectionOverUnion `[get]`
	Balanced Intersection over Union (IoU).
float	BlobDetectionAveragePrecision `[get]`
	Average precision for blob detection.
float	BlobDetectionBestFScore `[get]`
	Best F1-Score of the segmenter for blob detection.
float	BlobDetectionBestFScoreThreshold `[get]`
	Classification threshold that achieves the best F1-Score for blob detection.
float	BlobDetectionRecall `[get]`
	Recall for blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
float	BlobDetectionPrecision `[get]`
	Precision for blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
float	BlobDetectionFScore `[get]`
	F1-Score for defective blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics.ClassificationThreshold).
int	NumLabels `[get]`
	Number of labels recognized by the segmenter that produced the results aggregated in the metrics.
Properties inherited from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics
int	NumDefectiveSample `[get]`
	Number of defective sample added to the metric.
int	NumGoodSample `[get]`
	Number of good sample added to the metric.
float	BestAccuracy `[get]`
	Best achievable accuracy.
float	BestBalancedAccuracy `[get]`
	Best achievable balanced accuracy.
float	BestAccuracyClassificationThreshold `[get]`
	Classification threshold giving the best achievable accuracy (see EDeepLearningDefectDetectionMetrics.BestAccuracy).
float	BestBalancedAccuracyClassificationThreshold `[get]`
	Classification threshold giving the best achievable balanced accuracy (see EDeepLearningDefectDetectionMetrics.BestBalancedAccuracy).
float	AreaUnderROCCurve `[get]`
	The area under ROC curve (AUC) of the classifier (see EDeepLearningDefectDetectionMetrics.GetROCPoint). It's value is between 0 and 1.
float	AveragePrecision `[get]`
	Average precision.
float	BestFScore `[get]`
	Best F1-Score.
float	BestFScoreThreshold `[get]`
	Classification threshold that yields the best F1-Score.
int	PrecisionRecallCurveIndex `[get]`
	Index in the precision/recall curve for the current EDeepLearningDefectDetectionMetrics.ClassificationThreshold.
int	NumPrecisionRecallCurvePoint `[get]`
	Number of point in the precision/recall curve.
int	NumberOfClassifiers `[get]`
	Number of different possible classifiers.
float	ClassificationThreshold `[get, set]`
	Classification threshold.

Detailed Description

Collection of metrics used to evaluate the state of a ESupervisedSegmenter tool.

A metric is a value summarizing the quality of a collection of supervised segmentation results (see ESupervisedSegmenterResult) with respect to their ground truth. The ESupervisedSegmenterMetrics contains three types of metrics:

pixel based metrics that are related to the quality of the segmentation masks
blob based metrics that are related to the ability of the supervised segmentation tool to detect foreground blobs
defect detection metrics that are related to the ability of the supervised segmentation tool to differentiate between images that contains foreground pixels (defective images) and images that are entirely background (good images).

The pixel metrics are the error (see ESupervisedSegmenterMetrics.Error), the pixel accuracy (see ESupervisedSegmenterMetrics.PixelAccuracy), the pixel confusion (see ESupervisedSegmenterMetrics.GetPixelConfusion), the pixel per-label accuracy (see ESupervisedSegmenterMetrics.GetPixelLabelAccuracy) and the intersection over union (see ESupervisedSegmenterMetrics.GetIntersectionOverUnion).

The blob based metrics are two confusion matrices (ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion and ESupervisedSegmenterMetrics.GetPredictedBlobConfusion), and various defect detection metrics (ESupervisedSegmenterMetrics.BlobDetectionRecall, ESupervisedSegmenterMetrics.BlobDetectionPrecision, and ESupervisedSegmenterMetrics.BlobDetectionFScore). Note that the metrics for defective blob detection are different from the metrics for defective image detection because, in the case of blob detection, we have no "good" ground truth results.

See EDeepLearningDefectDetectionMetrics for a description of the defect detection metrics. These metrics are available when EDeepLearningDefectDetectionMetrics.IsDefectDetectionMetricsValid is true, i.e. when both images that are entirely background and images with non-background pixels have been added to the metrics.

Most metrics depends upon the EDeepLearningDefectDetectionMetrics.ClassificationThreshold.

License(s): BasicTypes

Constructor & Destructor Documentation

◆ ESupervisedSegmenterMetrics() [1/2]

eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.ESupervisedSegmenterMetrics ( )

Constructs an ESupervisedSegmenterMetrics object.

◆ ESupervisedSegmenterMetrics() [2/2]

eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.ESupervisedSegmenterMetrics ( eVision.EasyDeepLearning.ESupervisedSegmenterMetrics other )

Copy constructor of an ESupervisedSegmenterMetrics object.

Parameters

other Reference to the ESupervisedSegmenterMetrics object that should be copied

Member Function Documentation

◆ GetGroundtruthBlobConfusion() [1/2]

ulong eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion	(	int	groundtruthLabelIndex,
		int	predictedLabelIndex )

Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.

See ESupervisedSegmenterMetrics.GetPredictedBlobConfusion for the number of predicted blobs that match to these ground truth blobs.

Parameters

groundtruthLabelIndex	Ground truth label index
predictedLabelIndex	Predicted label index

A ground truth blob is matched to the subset of predicted blobs from the same predicted label that has the best intersection over union with the ground truth blob. Otherwise, the ground truth blob is matched to background.

◆ GetGroundtruthBlobConfusion() [2/2]

ulong eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion	(	string	groundtruthLabel,
		string	predictedLabel )

Number of ground truth blobs of the given ground truth label that best match with blobs of the given predicted label.

See ESupervisedSegmenterMetrics.GetPredictedBlobConfusion for the number of predicted blobs that match to these ground truth blobs.

Parameters

groundtruthLabel	Ground truth label
predictedLabel	Predicted label

A ground truth blob is matched to the subset of predicted blobs from the same predicted label that has the best intersection over union with the ground truth blob. Otherwise, the ground truth blob is matched to background.

◆ GetIntersectionOverUnion()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetIntersectionOverUnion ( string label )

Intersection over union.

The intersection over union if the ratio between the number of correctly classified pixels from the given label to the total number of pixels that belongs to that label or are predicted as being from that label.

Assuming that the given label is the positive class, the intersection over union is expressed as IOU = TP / (TP + FP + FN) where TP is the number of true positive, FP the number of false positive (pixels that belongs to label but are not predicted as label) and FN the number of false negative (pixels predicted as label but that do not belong to label).

Parameters

label Label

The intersection over union is also called the Jaccard index.

◆ GetLabel()

string eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetLabel ( int labelIdx )

Label recognized by the segmenter that produced the results aggregated in the metrics.

Parameters

labelIdx Index of the label between 0 and ESupervisedSegmenterMetrics.NumLabels - 1.

◆ GetLabelError()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetLabelError ( string label )

Label error.

Parameters

label Label

◆ GetNormalizedPixelConfusion()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetNormalizedPixelConfusion	(	string	trueLabel,
		string	predictedLabel )

Pixel-wise normalized confusion between the given true and predicted labels.

The normalized confusion is the ratio of the number of pixels belonging to the 'trueLabel' that are classified as 'predictedLabel' to the total number of pixels belonging to the 'trueLabel'.

Parameters

trueLabel	True label
predictedLabel	Predicted label

◆ GetPixelConfusion()

ulong eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelConfusion	(	string	trueLabel,
		string	predictedLabel )

Pixel-wise confusion between the given true and predicted labels.

The confusion is the number of pixels belonging to the 'trueLabel' that are classified as 'predictedLabel'.

Parameters

trueLabel	True label
predictedLabel	Predicted label

◆ GetPixelLabelAccuracy()

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPixelLabelAccuracy ( string label )

Pixel label accuracy.

The label accuracy is the ratio of the number of correctly classified pixels of the given class to the total number of ground truth pixels from that class. If there are no ground truth pixels from the requested label, the method returns '-1'.

Parameters

label Label

◆ GetPredictedBlobConfusion() [1/2]

ulong eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion	(	int	groundtruthLabelIndex,
		int	predictedLabelIndex )

Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.

See ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion for the number of ground truth blobs that match to these predicted blobs.

Parameters

groundtruthLabelIndex	Ground truth label index
predictedLabelIndex	Predicted label index

A predicted blob may be counted several times in the confusion matrix for predicted blobs:

Once if the more than 50% of the blob intersects ground truth background.
Once for each non-background label the predicted blob has an intersection with.

◆ GetPredictedBlobConfusion() [2/2]

ulong eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.GetPredictedBlobConfusion	(	string	groundtruthLabel,
		string	predictedLabel )

Number of predicted blobs of the given predicted label that match to blobs of the given ground truth label.

See ESupervisedSegmenterMetrics.GetGroundtruthBlobConfusion for the number of ground truth blobs that match to these predicted blobs.

Parameters

groundtruthLabel	Ground truth label
predictedLabel	Predicted label

A predicted blob may be counted several times in the confusion matrix for predicted blobs:

Once if the more than 50% of the blob intersects ground truth background.
Once for each non-background label the predicted blob has an intersection with.

◆ IsValid()

bool eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.IsValid ( )

Indicates whether the object contains at least one result.

◆ Load() [1/2]

override void eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Load ( eVision.ESerializer serializer )

virtual

Loads an supervised segmentation metric. The given ESerializer must have been created for reading.

Parameters

serializer The serializer.

Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.

◆ Load() [2/2]

override void eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Load ( string path )

virtual

Loads an supervised segmentation metric. The given ESerializer must have been created for reading.

Parameters

path	The file path.

Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.

◆ Save() [1/2]

override void eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Save ( eVision.ESerializer serializer )

virtual

Saves an supervised segmentation metric. The given ESerializer must have been created for writing.

Parameters

serializer The serializer.

Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.

◆ Save() [2/2]

override void eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Save ( string path )

virtual

Saves an supervised segmentation metric. The given ESerializer must have been created for writing.

Parameters

path	The file path.

Reimplemented from eVision.EasyDeepLearning.EDeepLearningDefectDetectionMetrics.

Property Documentation

◆ BalancedError

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedError

get

Balanced error.

The balanced error is the weighted error (ESupervisedSegmenterMetrics.WeightedError) where each label is given an equal weight.

The error is also called the cross-entropy loss.

◆ BalancedIntersectionOverUnion

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedIntersectionOverUnion

get

Balanced Intersection over Union (IoU).

The balanced intersection over union is the weighted intersection over union (ESupervisedSegmenterMetrics.WeightedIntersectionOverUnion) where each label is given equal weight.

◆ BalancedPixelAccuracy

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BalancedPixelAccuracy

get

Balanced accuracy.

The balanced accuracy is the weighted accuracy (ESupervisedSegmenterMetrics.WeightedPixelAccuracy) where each label is given an equal weight.

◆ BlobDetectionAveragePrecision

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionAveragePrecision

get

Average precision for blob detection.

Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label. The average precision for blob detection is the average of the precision (see ESupervisedSegmenterMetrics.BlobDetectionPrecision) over all the possible classification threshold values. As such, the average precision does not depend on a specific classification threshold.

◆ BlobDetectionBestFScore

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScore

get

Best F1-Score of the segmenter for blob detection.

See ESupervisedSegmenterMetrics.BlobDetectionBestFScoreThreshold for the corresponding threshold.

See EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ BlobDetectionBestFScoreThreshold

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionBestFScoreThreshold

get

Classification threshold that achieves the best F1-Score for blob detection.

See ESupervisedSegmenterMetrics.BlobDetectionBestFScore for the corresponding F1-Score.

See EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ BlobDetectionFScore

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionFScore

get

F1-Score for defective blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics.ClassificationThreshold).

See ESupervisedSegmenterMetrics.BlobDetectionBestFScore for the corresponding F1-Score.

See EDeepLearningDefectDetectionMetrics.GetFScore for a definition of the F1-Score. Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ BlobDetectionPrecision

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionPrecision

get

Precision for blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics.ClassificationThreshold).

The precision is the proportion of detected defective blobs that match ground truth defective blobs.

Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ BlobDetectionRecall

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.BlobDetectionRecall

get

Recall for blob detection given the current classification threshold (see EDeepLearningDefectDetectionMetrics.ClassificationThreshold).

The recall is the proportion of ground truth defective blobs that are matched to predicted defective blobs.

Blob detection is the ability of the segmenter to correctly detect foreground ground truth blobs, regardless of their specific label.

◆ Error

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.Error

get

Error.

The error is also called the cross-entropy loss.

◆ NumLabels

int eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.NumLabels

get

Number of labels recognized by the segmenter that produced the results aggregated in the metrics.

◆ PixelAccuracy

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.PixelAccuracy

get

Accuracy.

The accuracy is the ratio of the number of correctly classified pixels to the total number of pixels.

The accuracy will be skewed towards the most represented labels in the dataset. For example, if your dataset has 1% of defective pixels, the accuracy will mostly reflect the results on the 99% of good pixels and will say nothing about the capability of the model to detect the defect labels.

◆ WeightedError

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedError

get

Weighted error.

The weighted error is the weighted average of each label error (ESupervisedSegmenterMetrics.GetLabelError) with respect to the dataset label weights.

The error is also called the cross-entropy loss.

◆ WeightedIntersectionOverUnion

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedIntersectionOverUnion

get

Weighted Intersection over Union (IoU).

The weighted intersection over union is the weighted averaged of each label intersection over union (see ESupervisedSegmenterMetrics.GetIntersectionOverUnion) with respect to the dataset label weights.

◆ WeightedPixelAccuracy

float eVision.EasyDeepLearning.ESupervisedSegmenterMetrics.WeightedPixelAccuracy

get

Weighted accuracy.

The weighted accuracy is the weighted average of each label accuracy (ESupervisedSegmenterMetrics.GetPixelLabelAccuracy) with respect to the dataset label weights.

Public Member Functions

Properties

Detailed Description

Constructor & Destructor Documentation

◆ ESupervisedSegmenterMetrics() [1/2]

◆ ESupervisedSegmenterMetrics() [2/2]

Member Function Documentation

◆ GetGroundtruthBlobConfusion() [1/2]

◆ GetGroundtruthBlobConfusion() [2/2]

◆ GetIntersectionOverUnion()

◆ GetLabel()

◆ GetLabelError()

◆ GetNormalizedPixelConfusion()

◆ GetPixelConfusion()

◆ GetPixelLabelAccuracy()

◆ GetPredictedBlobConfusion() [1/2]

◆ GetPredictedBlobConfusion() [2/2]

◆ IsValid()

◆ Load() [1/2]

◆ Load() [2/2]

◆ Save() [1/2]

◆ Save() [2/2]

Property Documentation

◆ BalancedError

◆ BalancedIntersectionOverUnion

◆ BalancedPixelAccuracy

◆ BlobDetectionAveragePrecision

◆ BlobDetectionBestFScore

◆ BlobDetectionBestFScoreThreshold

◆ BlobDetectionFScore

◆ BlobDetectionPrecision

◆ BlobDetectionRecall

◆ Error

◆ NumLabels

◆ PixelAccuracy

◆ WeightedError

◆ WeightedIntersectionOverUnion

◆ WeightedPixelAccuracy