Continuous - Scikit-Learn Continuous Metrics

Continuous - Mean Squared Error

• sklearn.metrics.mean_squared_error

• Most common loss function in statistics
• Sensitive to outliers

Continuous - Mean Absolute Error

• sklearn.metrics.mean_absolute_error

• Not as sensetive to outliers
• Some prefer this over $MSE$

Continuous - Coefficient of Determination ($R^2$)

• sklearn.metrics.r2_score

• Might look familiar to correlation coefficient
• Depending on the case: $r^2 = R^2$
• [0, 1]
• Considered a Goodness of fit statistic: measure of how well the predictions approximate real data
• One of the common outputs when running a linear/logistic regression
• Not a good measure when comparing for overfitting

$R^2$ - Issues

Continuous - Adjusted Coeff. of Determination (Adj. $R^2$)

• Where $p$ is number of features and $n$ number of data points
• Also one of the common outputs when running a linear/logistic regression
  • Recall statsmodels library: diabetes dataset

Continuous - Coefficient of Determination $R^2$

Continuous - Explained Variance

• sklearn.metrics.explained_variance_score

• Remember PCA?

Continuous - Residual Sum of Squares

• Careful
  • Any change to dataset
  • Comparing models with models
• Residuals are important

Categorical - Scikit-Learn Classification Metrics

Categorical - Accuracy

• sklearn.metrics.accuracy_score

• Where $1()$ is the indicator function
• Simple accuracy calculation

Confusion Matrix

Confusion Matrix

• Classification - Categorical Responses
• Basically - Right vs Wrong
• Two dimensions
  • Rows represent predictions
  • Columns represent actuals
• Can be done for > 2 categorical predictions
  • Table for each category

Confusion Matrix - Multiple Categories

Confusion Matrix - Category A

Confusion Matrix - Category B

Confusion Matrix - Category C

Confusion Matrix - Category D

Confusion Matrix - Calculating

• sklearn.metrics.confusion_matrix
  • Works with multiple classifications
• sklearn.metrics.multilabel_confusion_matrix
  • Builds out individual matricies

Confusion Matrix - Metrics

Accuracy (ACC) $$ ACC = \dfrac{\color{green}{TP} + \color{pink}{TN}}{\color{green}{TP} + \color{pink}{TN} + \color{blue}{FP} + \color{red}{FN}} $$

Confusion Matrix - Metrics

Confusion Matrix - Common Measures

• Precision (PPV)
  $ PPV = \dfrac{\sum{\color{green}{TP}}}{\sum{\color{orange}{PP}}} $
  • Proportions of positive results that are true positive
  • sklearn.metrics.precision_score
• Recall (TPR)
  $ TPR = \dfrac{\sum{\color{green}{TP}}}{\sum{\color{purple}{CP}}} $
  • Measures the proportion of positives that are correctly identified
  • sklearn.metrics.recall_score
• F1 Score
  $ 2 \cdot \dfrac{PPV \cdot TPR}{PPV + TPR}$
  • Measure of a models accuracy
  • sklearn.metrics.f1_score

In Summary

• Many different evaluation techniques
• Continuous
  • Accuracy
  • Mean Squared Error / Mean Absolute Error
  • $R^2$
• Residuals
• Categorical
  • Confusion Matrix
    • Tons of metrics
• Reciever Operator Characteristic
  • Area Under the ROC
• Model calibration
  • Many methods

Mid-Term Due October 28th

• Finish up your mid-term
• I will download all repos at midnight on Friday
• No late assignments accepted
  • You will get a zero