Metrics

Machine learning evaluation metrics, implemented in Python, R, Haskell, and MATLAB / Octave.

Metrics provides implementations of various supervised machine learning evaluation metrics in the following languages:

  • Python easy_install ml_metrics
  • R install.packages("Metrics") from the R prompt
  • Haskell cabal install Metrics
  • MATLAB / Octave (clone the repo & run setup from the MATLAB command line)

For more detailed installation instructions, see the README for each implementation.

EVALUATION METRICS

Evaluation MetricPythonRHaskellMATLAB / Octave
Absolute Error (AE)✓✓✓✓
Average Precision at K (APK, AP@K)✓✓✓✓
Area Under the ROC (AUC)✓✓✓✓
Classification Error (CE)✓✓✓✓
F1 Score (F1) ✓
Gini ✓
Levenshtein✓ ✓✓
Log Loss (LL)✓✓✓✓
Mean Log Loss (LogLoss)✓✓✓✓
Mean Absolute Error (MAE)✓✓✓✓
Mean Average Precision at K (MAPK, MAP@K)✓✓✓✓
Mean Quadratic Weighted Kappa✓✓ ✓
Mean Squared Error (MSE)✓✓✓✓
Mean Squared Log Error (MSLE)✓✓✓✓
Normalized Gini ✓
Quadratic Weighted Kappa✓✓ ✓
Relative Absolute Error (RAE) ✓
Root Mean Squared Error (RMSE)✓✓✓✓
Relative Squared Error (RSE) ✓
Root Relative Squared Error (RRSE) ✓
Root Mean Squared Log Error (RMSLE)✓✓✓✓
Squared Error (SE)✓✓✓✓
Squared Log Error (SLE)✓✓✓✓

TO IMPLEMENT

  • F1 score
  • Multiclass log loss
  • Lift
  • Average Precision for binary classification
  • precision / recall break-even point
  • cross-entropy
  • True Pos / False Pos / True Neg / False Neg rates
  • precision / recall / sensitivity / specificity
  • mutual information

HIGHER LEVEL TRANSFORMATIONS TO HANDLE

  • GroupBy / Reduce
  • Weight individual samples or groups

PROPERTIES METRICS CAN HAVE

(Nonexhaustive and to be added in the future)

  • Min or Max (optimize through minimization or maximization)
  • Binary Classification
    • Scores predicted class labels
    • Scores predicted ranking (most likely to least likely for being in one class)
    • Scores predicted probabilities
  • Multiclass Classification
    • Scores predicted class labels
    • Scores predicted probabilities
  • Regression
  • Discrete Rater Comparison (confusion matrix)

GitHub