Usage of metrics

A metric is a function that is used to judge the performance of your model. Metric functions are to be supplied in the metrics parameter when a model is compiled.

A metric function is similar to an objective function, except that the results from evaluating a metric are not used when training the model.

You can either pass the name of an existing metric, or pass a Theano/TensorFlow symbolic function (see Custom metrics).

Arguments

• y_true: True labels. Theano/TensorFlow tensor.
• y_pred: Predictions. Theano/TensorFlow tensor of the same shape as y_true.

Returns

Single tensor value representing the mean of the output array across all datapoints.

---

Available metrics

binary_accuracy

binary_accuracy(y_true, y_pred)

Calculates the mean accuracy rate across all predictions for binary classification problems.

---

categorical_accuracy

categorical_accuracy(y_true, y_pred)

Calculates the mean accuracy rate across all predictions for multiclass classification problems.

---

sparse_categorical_accuracy

sparse_categorical_accuracy(y_true, y_pred)

Same as categorical_accuracy, but useful when the predictions are for sparse targets.

---

top_k_categorical_accuracy

top_k_categorical_accuracy(y_true, y_pred, k=5)

Calculates the top-k categorical accuracy rate, i.e. success when the target class is within the top-k predictions provided.

---

mean_squared_error

mean_squared_error(y_true, y_pred)

Calculates the mean squared error (mse) rate between predicted and target values.

---

mean_absolute_error

mean_absolute_error(y_true, y_pred)

Calculates the mean absolute error (mae) rate between predicted and target values.

---

mean_absolute_percentage_error

mean_absolute_percentage_error(y_true, y_pred)

Calculates the mean absolute percentage error (mape) rate between predicted and target values.

---

mean_squared_logarithmic_error

mean_squared_logarithmic_error(y_true, y_pred)

Calculates the mean squared logarithmic error (msle) rate between predicted and target values.

---

hinge

hinge(y_true, y_pred)

Calculates the hinge loss, which is defined as max(1 - y_true * y_pred, 0).

---

squared_hinge

squared_hinge(y_true, y_pred)

Calculates the squared value of the hinge loss.

---

categorical_crossentropy

categorical_crossentropy(y_true, y_pred)

Calculates the cross-entropy value for multiclass classification problems. Note: Expects a binary class matrix instead of a vector of scalar classes.

---

sparse_categorical_crossentropy

sparse_categorical_crossentropy(y_true, y_pred)

Calculates the cross-entropy value for multiclass classification problems with sparse targets. Note: Expects an array of integer classes. Labels shape must have the same number of dimensions as output shape. If you get a shape error, add a length-1 dimension to labels.

---

binary_crossentropy

binary_crossentropy(y_true, y_pred)

Calculates the cross-entropy value for binary classification problems.

---

kullback_leibler_divergence

kullback_leibler_divergence(y_true, y_pred)

Calculates the Kullback-Leibler (KL) divergence between prediction and target values.

---

poisson

poisson(y_true, y_pred)

Calculates the poisson function over prediction and target values.

---

cosine_proximity

cosine_proximity(y_true, y_pred)

Calculates the cosine similarity between the prediction and target values.

---

matthews_correlation

matthews_correlation(y_true, y_pred)

Calculates the Matthews correlation coefficient measure for quality of binary classification problems.

---

precision

precision(y_true, y_pred)

Calculates the precision, a metric for multi-label classification of how many selected items are relevant.

---

recall

recall(y_true, y_pred)

Calculates the recall, a metric for multi-label classification of how many relevant items are selected.

---

fbeta_score

fbeta_score(y_true, y_pred, beta=1)

Calculates the F score, the weighted harmonic mean of precision and recall.

This is useful for multi-label classification, where input samples can be classified as sets of labels. By only using accuracy (precision) a model would achieve a perfect score by simply assigning every class to every input. In order to avoid this, a metric should penalize incorrect class assignments as well (recall). The F-beta score (ranged from 0.0 to 1.0) computes this, as a weighted mean of the proportion of correct class assignments vs. the proportion of incorrect class assignments.

With beta = 1, this is equivalent to a F-measure. With beta < 1, assigning correct classes becomes more important, and with beta > 1 the metric is instead weighted towards penalizing incorrect class assignments.

---

fmeasure

fmeasure(y_true, y_pred)

Calculates the f-measure, the harmonic mean of precision and recall.

---

Custom metrics

Custom metrics can be defined and passed via the compilation step. The function would need to take (y_true, y_pred) as arguments and return either a single tensor value or a dict metric_name -> metric_value.

# for custom metrics
import keras.backend as K

def mean_pred(y_true, y_pred):
    return K.mean(y_pred)

def false_rates(y_true, y_pred):
    false_neg = ...
    false_pos = ...
    return {
        'false_neg': false_neg,
        'false_pos': false_pos,
    }

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy', mean_pred, false_rates])