Quantifying uncertainty 👢

# Quantifying uncertainty <br> 👢
### Dr. Çetinkaya-Rundel

---

layout: true
  
<div class="my-footer">
<span>
Dr. Mine Çetinkaya-Rundel -
<a href="https://introds.org" target="_blank">introds.org
</a>
</span>
</div>

---

## Announcements

- Check presentation schedule and let me know if there are any concerns.
- We'll start presentations at 12pm (not 12:10pm)!

---

# Inference

---

- Statistical inference is the process of using sample data to make conclusions 
about the underlying population the sample came from.
- Similar to tasting a spoonful of soup while cooking to make an inference about the entire pot.

---

## Estimation

So far we have done lots of estimation (mean, median, slope, etc.), i.e.
- used data from samples to calculate sample statistics
- which can then be used as estimates for population parameters

---

<br>

.pull-left[
<img src="img/spear.png" width="400" style="display: block; margin: auto;" />
]
.pull-right[
<img src="img/net.png" width="400" style="display: block; margin: auto;" />
]

---

.question[
If you want to estimate a population parameter, do you prefer to report a range 
of values the parameter might be in, or a single value?
]

<br>

- If we report a point estimate, we probably won’t hit the exact population 
parameter.
- If we report a range of plausible values we have a good shot at capturing 
the parameter.

---

.footnote[
Source: [General election poll tracker: How do the parties compare?](https://www.bbc.co.uk/news/uk-politics-49798197), 18 Nov 2019.
]

---

# Confidence intervals

---

## Confidence intervals

A plausible range of values for the population parameter is a **confidence interval**.

--
- In order to construct a confidence interval we need to quantify the variability 
of our sample statistic.

--
- For example, if we want to construct a confidence interval for a population mean, 
we need to come up with a plausible range of values around our observsed sample mean.

--
- This range will depend on how precise and how accurate our sample mean is as an 
estimate of the population mean.

--
- Quantifying this requires a measurement of how much we would expect the sample 
mean to vary from sample to sample.

---

.question[
Suppose we split the class in half down the middle of the classroom and ask 
each student their heights. Then, we calculate the mean height of students 
on each side of the classroom. Would you expect these two means to be exactly 
equal, close but not equal, or wildly different?
]

.question[
Suppose you randomly sample 50 students and 5 of them are left handed. If you 
were to take another random sample of 50 students, how many would you expect to 
be left handed? Would you be surprised if only 3 of them were left handed? Would 
you be surprised if 40 of them were left handed?
]

---

## Quantifying the variability of a sample statistic

We can quantify the variability of sample statistics using

- simulation: via bootstrapping (today)

- theory: via Central Limit Theorem (in Stat 2!)

---

# Bootstrapping

---

## Bootstrapping

- The term **bootstrapping** comes from the phrase "pulling oneself up by one’s 
bootstraps", which is a metaphor for accomplishing an impossible task without 
any outside help.
- In this case the impossible task is estimating a population parameter, and we’ll 
accomplish it using data from only the given sample.
- Note that this notion of saying something about a population parameter using 
only information from an observed sample is the crux of statistical inference, 
it is not limited to bootstrapping.

---

## Rent in Edinburgh

---

## Sample

Fifteen 3 BR flats in Edinburgh were randomly selected on rightmove.co.uk.

```r
library(tidyverse)
edi_3br <- read_csv2("data/edi-3br.csv") # ; separated
```

```
## # A tibble: 15 x 4
##    flat_id  rent title                      address                        
##    <chr>   <dbl> <chr>                      <chr>                          
##  1 flat_01   825 3 bedroom apartment to re… Burnhead Grove, Edinburgh, Mid…
##  2 flat_02  2400 3 bedroom flat to rent     Simpson Loan, Quartermile, Edi…
##  3 flat_03  1900 3 bedroom flat to rent     FETTES ROW, NEW TOWN, EH3 6SE  
##  4 flat_04  1500 3 bedroom apartment to re… Eyre Crescent, Edinburgh, Midl…
##  5 flat_05  3250 3 bedroom flat to rent     Walker Street, Edinburgh       
##  6 flat_06  2145 3 bedroom flat to rent     George Street, City Centre, Ed…
##  7 flat_07  1500 3 bedroom flat to rent     Waverley Place , Edinburgh EH7…
##  8 flat_08  1950 3 bedroom flat to rent     Drumsheugh Place, Edinburgh    
##  9 flat_09  1725 3 bedroom flat to rent     Crighton Place, Leith, Edinbur…
## 10 flat_10  2995 3 bedroom flat to rent     Simpson Loan, Meadows, Edinbur…
## 11 flat_11  1400 3 bedroom flat to rent     42, Learmonth Court, Edinburgh…
## 12 flat_12  1995 3 bedroom apartment to re… Chester Street, Edinburgh, Mid…
## 13 flat_13  1250 3 bedroom duplex to rent   Elmwood Terrace, Lochend, Edin…
## 14 flat_14  1995 3 bedroom apartment to re… Great King Street, Edinburgh, …
## 15 flat_15  1600 3 bedroom ground floor fl… Roseneath Terrace,Edinburgh,EH9
```
]

---

## Observed sample

---

## Observed sample

Sample mean ≈ £1895 😱

<br>

---

## Bootstrap population

Generated assuming there are more flats like the ones in the observed sample... Population mean = ❓

<br>

---

## Bootstrapping scheme

1. Take a bootstrap sample - a random sample taken **with replacement** from the 
original sample, of the same size as the original sample.
2. Calculate the bootstrap statistic - a statistic such as mean, median, 
proportion, slope, etc. computed on the bootstrap samples.
3. Repeat steps (1) and (2) many times to create a bootstrap distribution - 
a distribution of bootstrap statistics.
4. Calculate the bounds of the XX% confidence interval as the middle XX% 
of the bootstrap distribution.

---

## Let's bootstrap

---

# Bootstrapping in R

---

## Two ways

1. Using `for` loops
2. Using the **infer** package

---

## Bootstrapping with `for` loops

```r
set.seed(9014)

boot_df <- tibble(
  replicate = 1:100,
  stat = rep(NA, 100)
  )

for (i in 1:100){
   boot_df$stat[i] <- edi_3br %>% 
     sample_n(15, replace = TRUE) %>% 
     summarise(stat = mean(rent)) %>% 
     pull()
}
```

---

## Bootstrap results

```r
ggplot(boot_df, aes(x = stat)) +
  geom_histogram(binwidth = 100)
```

```r
boot_df %>%
  summarise(
    lower = quantile(stat, 0.025),
    upper = quantile(stat, 0.975),
    )
```
]
]
.pull-right[

```
## # A tibble: 1 x 2
##   lower upper
##   <dbl> <dbl>
## 1 1594. 2292.
```
]

---

## infer `$\in$` tidymodels

.pull-left[
<img src="img/infer-part-of-tidymodels.png" width="350" />
]
.pull-right[
The objective of `infer` is to perform statistical inference using an expressive statistical grammar that coheres with the `tidyverse` design framework.

```r
library(infer)
```
]

---

## Generate bootstrap means

```r
edi_3br %>%
  # specify the variable of interest
  specify(response = rent)
```

---

## Generate bootstrap means

```r
edi_3br %>%
  # specify the variable of interest
  specify(response = rent)
  # generate 15000 bootstrap samples
  generate(reps = 15000, type = "bootstrap")
```

---

## Generate bootstrap means

```r
edi_3br %>%
  # specify the variable of interest
  specify(response = rent)
  # generate 15000 bootstrap samples
  generate(reps = 15000, type = "bootstrap")
  # calculate the mean of each bootstrap sample
  calculate(stat = "mean")
```

---

## Generate bootstrap means

```r
# save resulting bootstrap distribution
boot_df <- edi_3br %>%
  # specify the variable of interest
  specify(response = rent) %>% 
  # generate 15000 bootstrap samples
  generate(reps = 15000, type = "bootstrap") %>% 
  # calculate the mean of each bootstrap sample
  calculate(stat = "mean")
```

---

## The bootstrap sample

```r
boot_df
```

```
## # A tibble: 15,000 x 2
##    replicate  stat
##        <int> <dbl>
##  1         1 1793.
##  2         2 1938.
##  3         3 2175 
##  4         4 2159.
##  5         5 2084 
##  6         6 1761 
##  7         7 1787.
##  8         8 1817.
##  9         9 1963.
## 10        10 1800.
## # … with 14,990 more rows
```

---

## Visualize the bootstrap distribution

```r
ggplot(data = boot_df, mapping = aes(x = stat)) +
  geom_histogram(binwidth = 100) +
  labs(title = "Bootstrap distribution of means")
```

---

## Calculate the confidence interval

A 95% confidence interval is bounded by the middle 95% of the bootstrap distribution.

```r
boot_df %>%
  summarize(lower = quantile(stat, 0.025),
            upper = quantile(stat, 0.975))
```

```
## # A tibble: 1 x 2
##   lower upper
##   <dbl> <dbl>
## 1 1603. 2213.
```

---

## Visualize the confidence interval

---

## Interpret the confidence interval

.question[
The 95% confidence interval for the mean rent of three bedroom flats in 
Edinburgh was calculated as (1603, 2213). Which of the 
following is the correct interpretation of this interval?

<br>

**(a)** 95% of the time the mean rent of three bedroom flats in this sample is between £1603 and £2213.

**(b)** 95% of all three bedroom flats in Edinburgh have rents between £1603 and £2213.

**(c)** We are 95% confident that the mean rent of all three bedroom flats is between £1603 and £2213.

**(d)** We are 95% confident that the mean rent of three bedroom flats in this sample is between £1603 and £2213.
]

---

# Accuracy vs. precision

---

## Confidence level

**We are 95% confident that ...**

- Suppose we took many samples from the original population and built a 95% confidence interval based on each sample.
- Then about 95% of those intervals would contain the true population parameter.

---

## Commonly used confidence levels

---

## Precision vs. accuracy

.question[
If we want to be very certain that we capture the population parameter, should 
we use a wider interval or a narrower interval? What drawbacks are associated 
with using a wider interval?
]

![garfield](img/garfield.png)

---

## Changing confidence level

.question[
How would you modify the following code to calculate a 90% confidence interval? 
How would you modify it for a 99% confidence interval?
]

```r
edi_3br %>%
  specify(response = rent) %>% 
  generate(reps = 15000, type = "bootstrap") %>% 
  calculate(stat = "mean") %>%
  summarize(lower = quantile(stat, 0.025),
            upper = quantile(stat, 0.975))
```

---

## Recap

- Sample statistic `$\ne$` population parameter, but if the sample is good, it can be a good estimate.
- We report the estimate with a confidence interval, and the width of this interval depends on the variability of sample statistics from different samples from the population.
- Since we can't continue sampling from the population, we bootstrap from the one sample we have to estimate sampling variability.
- We can do this for any sample statistic:
  - For a mean: `calculate(stat = "mean")`
  - For a median: `calculate(stat = "median")`
  - Learn about calculating bootstrap intervals for other statistics 
  in your homework.