Movement and Networks I

HES 505 Fall 2023: Session 25

Matt Williamson

Revisiting Deviance

Pseudo- \(R^2\)

\[ \begin{equation} R^2_L = \frac{D_{null} - D_{fitted}}{D_{null}} \end{equation} \]

\[ \begin{equation} \begin{aligned} R^2_{CS} &= 1 - \left( \frac{L_0}{L_M} \right)^{(2/n)}\\ &= 1 - \exp^{2(ln(L_0)-ln(L_M))/n} \end{aligned} \end{equation} \]

Cohen’s Likelihood Ratio

logistic.rich <- glm(y ~ MeanAnnTemp + PrecipWetQuarter + PrecipDryQuarter, 
                     family=binomial(link="logit"),
                     data=pts.df[,2:8])

logistic.null <- glm(y ~ 1, 
                     family=binomial(link="logit"),
                     data=pts.df[,2:8])
with(logistic.rich, 
     null.deviance - deviance)/with(logistic.rich,
                                    null.deviance)
[1] 0.4495966
1 - exp(2*(logLik(logistic.null)[1] - logLik(logistic.rich)[1])/nobs(logistic.rich))
[1] 0.4308873

So what is deviance???

  • Saturated model (i.e., perfect model)

  • One parameter for each observation

  • Null model

  • Only 1 parameter (for the intercept)

  • Your model

  • 1 parameter for each covariate

Sub-sampling Methods

  • Split data into training and testing

  • Testing set needs to be large enough for results to be statistically meaningful

  • Test set should be representative of the data as a whole

  • Validation data used to tune parameters (not always)

Subsampling your data with caret

pts.df$y <- as.factor(ifelse(pts.df$y == 1, "Yes", "No"))
library(caret)
Train <- createDataPartition(pts.df$y, p=0.6, list=FALSE)

training <- pts.df[ Train, ]
testing <- pts.df[ -Train, ]

Misclassification

  • Confusion matrices compare actual values to predictions
  • True Positive (TN) - This is correctly classified as the class if interest / target.
  • True Negative (TN) - This is correctly classified as not a class of interest / target.
  • False Positive (FP) - This is wrongly classified as the class of interest / target.
  • False Negative (FN) - This is wrongly classified as not a class of interest / target.

Confusion Matrices in R

train.log <- glm(y ~ ., 
                 family="binomial", 
                 data=training[,2:8])

predicted.log <- predict(train.log, 
                         newdata=testing[,2:8], 
                         type="response")

pred <- as.factor(
  ifelse(predicted.log > 0.5, 
                         "Yes",
                         "No"))
confusionMatrix(testing$y, pred)
Confusion Matrix and Statistics

          Reference
Prediction No Yes
       No  24   3
       Yes  3   9
                                          
               Accuracy : 0.8462          
                 95% CI : (0.6947, 0.9414)
    No Information Rate : 0.6923          
    P-Value [Acc > NIR] : 0.02323         
                                          
                  Kappa : 0.6389          
                                          
 Mcnemar's Test P-Value : 1.00000         
                                          
            Sensitivity : 0.8889          
            Specificity : 0.7500          
         Pos Pred Value : 0.8889          
         Neg Pred Value : 0.7500          
             Prevalence : 0.6923          
         Detection Rate : 0.6154          
   Detection Prevalence : 0.6923          
      Balanced Accuracy : 0.8194          
                                          
       'Positive' Class : No

Confusion Matrices

\[ \begin{equation} \begin{aligned} Accuracy &= \frac{TP + TN}{TP + TN + FP + FN}\\ Sensitivity &= \frac{TP}{TP + FN}\\ Specificity &= \frac{TN}{FP + TN}\\ Precision &= \frac{TP}{TP + FP}\\ Recall &= \frac{TP}{TP + FN} \end{aligned} \end{equation} \]

Depends upon threshold!!

Confusion Matrices in R

library(tree)
tree.model <- tree(y ~ . , training[,2:8])
predict.tree <- predict(tree.model, newdata=testing[,2:8], type="class")
confusionMatrix(testing$y, predict.tree)
Confusion Matrix and Statistics

          Reference
Prediction No Yes
       No  23   4
       Yes  2  10
                                          
               Accuracy : 0.8462          
                 95% CI : (0.6947, 0.9414)
    No Information Rate : 0.641           
    P-Value [Acc > NIR] : 0.004181        
                                          
                  Kappa : 0.6549          
                                          
 Mcnemar's Test P-Value : 0.683091        
                                          
            Sensitivity : 0.9200          
            Specificity : 0.7143          
         Pos Pred Value : 0.8519          
         Neg Pred Value : 0.8333          
             Prevalence : 0.6410          
         Detection Rate : 0.5897          
   Detection Prevalence : 0.6923          
      Balanced Accuracy : 0.8171          
                                          
       'Positive' Class : No

Confusion Matrices in R

library(randomForest)
class.model <- y ~ .
rf <- randomForest(class.model, data=training[,2:8])
predict.rf <- predict(rf, newdata=testing[,2:8], type="class")
confusionMatrix(testing$y, predict.rf)
Confusion Matrix and Statistics

          Reference
Prediction No Yes
       No  26   1
       Yes  5   7
                                          
               Accuracy : 0.8462          
                 95% CI : (0.6947, 0.9414)
    No Information Rate : 0.7949          
    P-Value [Acc > NIR] : 0.2851          
                                          
                  Kappa : 0.602           
                                          
 Mcnemar's Test P-Value : 0.2207          
                                          
            Sensitivity : 0.8387          
            Specificity : 0.8750          
         Pos Pred Value : 0.9630          
         Neg Pred Value : 0.5833          
             Prevalence : 0.7949          
         Detection Rate : 0.6667          
   Detection Prevalence : 0.6923          
      Balanced Accuracy : 0.8569          
                                          
       'Positive' Class : No

Threshold-Free Methods

  • Receiver Operating Characteristic Curves

  • Illustrates discrimination of binary classifier as the threshold is varied

  • Area Under the Curve (AUC) provides an estimate of classification ability

Criticisms of ROC/AUC

  • Treats false positives and false negatives equally

  • Undervalues models that predict across smaller geographies

  • Focus on discrimination and not calibration

  • New methods for presence-only data

ROC in R (using pROC)

  • Generate predictions (note the difference for tree and rf)
library(pROC)
train.log <- glm(y ~ ., 
                 family="binomial", 
                 data=training[,2:8])

predicted.log <- predict(train.log, 
                         newdata=testing[,2:8], 
                         type="response")

predict.tree <- predict(tree.model, newdata=testing[,2:8], type="vector")[,2]

predict.rf <- predict(rf, newdata=testing[,2:8], type="prob")[,2]

ROC in R (using pROC)

plot(roc(testing$y, predicted.log), print.auc=TRUE)

plot(roc(testing$y, predict.tree), print.auc=TRUE, print.auc.y = 0.45, col="green", add=TRUE)

plot(roc(testing$y, predict.rf), print.auc=TRUE, print.auc.y = 0.4, col="blue", add=TRUE)

Cross-validation

  • Often want to make sure that fit/accuracy not a function of partition choice

  • Cross-validation allows resampling of data (multiple times)

  • K-fold - Data are split into K datasets of ~ equal size, model fit to \((K-1)(\frac{n}{K})\) observations to predict heldout set

  • Leave One Out (LOO) - Model fit to n-1 observations to predict the held out observation

Crossvalidation in R using caret

fitControl <- trainControl(method = "repeatedcv",
                           number = 10,
                           repeats = 10,
                           classProbs = TRUE,
                           summaryFunction = twoClassSummary)

log.model <- train(y ~., data = pts.df[,2:8],
               method = "glm",
               trControl = fitControl)
pred.log <- predict(log.model, newdata = testing[,2:8], type="prob")[,2]

tree.model <- train(y ~., data = pts.df[,2:8],
               method = "rpart",
               trControl = fitControl)

pred.tree <- predict(tree.model, newdata=testing[,2:8], type="prob")[,2]

rf.model <- train(y ~., data = pts.df[,2:8],
               method = "rf",
               trControl = fitControl)
pred.rf <- predict(rf.model, newdata=testing[,2:8], type="prob")[,2]

Crossvalidation in R using caret

plot(roc(testing$y, pred.log), print.auc=TRUE)

plot(roc(testing$y, pred.tree), print.auc=TRUE, print.auc.y = 0.45, col="green", add=TRUE)

plot(roc(testing$y, pred.rf), print.auc=TRUE, print.auc.y = 0.4, col="blue", add=TRUE)

Crossvalidation in R using caret

Spatial predictions

best.rf <- rf.model$finalModel
best.glm <- log.model$finalModel

rf.spatial <- terra::predict(pred.stack.scl, best.rf, type="prob")


glm.spatial <- terra::predict(pred.stack.scl, best.glm,type="response" )

Spatial predictions

On to networks!

What is a network?

  • A collection of connected objects

  • Tend to described in terms of nodes (the objects) and edges (the connections)

  • Analyzed using algorithms from graph theory

Types of networks

  • (Un)directed

  • Weighted

  • Muli-type

Describing networks for analysis

Common measures

  • Graph-level: density, diameter, distance

  • Component-level: density, distribution

  • Node-level: centrality, degree-distribution

Common questions

  • What are the shortest paths across the network?

  • Where are the most important locations for maintaining the network?

  • How does the loss of a node alter the subsequent configuration of the network?

  • How do we translate typical movement paths into network structures?