Provide instant feedback

# Provide instant feedback
## <br><br> What they forgot to teach you<br>about <em>teaching</em> R
### <a href="https://wtf-teach.netlify.app/">wtf-teach.netlify.app</a>

---

layout: true
  
<div class="my-footer">
<span>
<a href="http://wtf-teach.netlify.app/" target="_blank">wtf-teach.netlify.app</a>
</span>
</div>

---

# Why auto feedback?

---

.pull-left[
.center[
.huge[👉] <br>
.large[**Nudging**] <br>
students towards the right answer, especially in formative assessments
]
]

---

.sample-question[
Suppose 10 means from a simulated sampling distribution is stored in a vector called `means`.

```r
means
```

```
##  [1] -1.21  0.28  1.08 -2.35  0.43  0.51 -0.57 -0.55 -0.56 -0.89
```
What is the value of the first mean?
]
--
<br>
.pull-left-wide[
.sample-answer[

```r
mean[1]
```

```
## Error in mean[1]: object of type 'closure' is not subsettable
```
]
]
--
.pull-right-narrow[
<img src="images/student.png" width="100%" style="display: block; margin: auto 0 auto auto;" />
]

---

## Nudging

<br>

Providing helpful feedback can help them nudge them towards success:

```r
mean[1]
```

---

.sample-question[
Visualise the relationship between city and highway mileage of cars from the `mpg` dataset, conditional on year of manufacture.
]

--
<br>
.sample-answer[
There is a strong, positive, linear relationship between the city and highway mileage of cars. Year does not seem to be related to either variable.

```r
ggplot(mpg,aes(x = hwy, y = cty, fill=year)) +geom_point()+geom_smooth()
```

<img src="05-feedback_files/figure-html/unnamed-chunk-9-1.png" width="30%" style="display: block; margin: auto;" />
]

---

.pull-left-wide[
.sample-feedback[
- You mention a linear relationship, however your plot uses a loess fit to visualise the relationship between city and highway mileage. Also, the plot displays the uncertainty around the fit, but you haven't addressed it in your narrative.
- Year should be mapped to the `color` aesthetic, not `fill`. 
- Plot styling: Use informative axis labels, noting units of measurement. Also, give an informative title to your plot.
- Code styling: Use consistent spacing around operators (e.g ` = `) and line breaks after `+` in each layer of your ggplot.
]
]
.pull-right-narrow[
<img src="images/teacher.png" width="100%" style="display: block; margin: auto;" />
]

---

.pull-left[
.center[
.huge[👉] <br>
.large[**Nudging**] <br>
students towards the right answer, especially in formative assessments
]
]
.pull-right[
.center[
.huge[➕] <br>
.large[**Scaling**] <br>
up efficiency of grading faster than (human) resources
]
]

---

## Scaling

- Students turning in their work as R Markdown documents makes collecting submissions including code and narrative straightforward.

- Providing feedback on both the code and narrative is not scalable unless (human) resources dedicated to your course grow proportionally with enrolments.

---

# How auto feedback?

---

---

## 📦 learnr

- **learnr** is an R package that makes it easy to create interactive tutorials from R Markdown documents.

- Tutorials can include:
  - Narrative, figures, illustrations, and equations
  - Code exercises (R code chunks that users can edit and execute directly)
  - Multiple choice questions
  - Videos (YouTube, Vimeo)
  - Interactive Shiny components

- learnr is on CRAN

```r
install.packages("learnr")
```
  
---

## 📦 gradethis

- Companion to the learnr package, **gradethis** provides multiple methods to grade learnr exercises:
  - `grade_this()`: Grade result of exercise code
  - `grade_this_code()`: Grade code of exercise

- gradethis is not yet on CRAN

```r
devtools::install_github("rstudio-education/gradethis")
```

---

## Demo

[**`[code]`**](https://github.com/rstudio-education/wtf-teach-rstudio-global/tree/main/materials/05-feedback/penguins/penguins.Rmd)
]

---

# learnr

---

## YAML

Start with a YAML, just like in R Markdown:

```
---
title: "Palmer penguins"
output: 
  learnr::tutorial:
    progressive: true
    allow_skip: true
    css: "css/font-size.css"
runtime: shiny_prerendered
---
```

- `runtime: shiny_prerendered`
- `progressive: true` for "Continue" buttons between subsections
- `alow_skip: true` to allow skipping exercises

---

## Customization

- You can change the style of your learnr tutorial
- You might, at a minimum, implement a couple customizations for accessibility:
  - Increase font size in the narrative, using a [CSS file](https://github.com/rstudio-education/wtf-teach-rstudio-global/tree/main/materials/05-feedback/penguins/css/font-size.css) that lives in a directory called `css/` and loaded in the YAML with

```
css: "css/font-size.css"
```

- Increase font size in code boxes, using a [JS file](https://github.com/rstudio-education/wtf-teach-rstudio-global/tree/main/materials/05-feedback/penguins/js/exercise-font-size.js) that lives in a directory called `js/` and loaded with 
  
```
<script language="JavaScript" src="js/exercise-font-size.js"></script>
```
  
---

## Narrative

- R Markdown style section and subsection headings with `##`, `###`, etc.
- Text, figures, illustrations, and equations
- Videos: supported services include YouTube and Vimeo

```
### Learning goals

-   Practice developing exercises with learnr.
-   Practice writing code checking tests with gradethis.

### Getting help

If you have any questions call for help in Zoom!
```

---

## Multiple choice questions

```r
quiz(
  question("What position is the letter A in the english alphabet?",
    answer("8"),
    answer("14"),
    answer("1", correct = TRUE),
    answer("23"),
    incorrect = "See [here](https://en.wikipedia.org/wiki/English_alphabet) and try again.",
    allow_retry = TRUE
  ),

question("Where are you right now? (select ALL that apply)",
    answer("Planet Earth", correct = TRUE),
    answer("Pluto"),
    answer("At a computing device", correct = TRUE),
    answer("In the Milky Way", correct = TRUE),
    incorrect = paste0("Incorrect. You're on Earth, ",
                       "in the Milky Way, at a computer.")
  )
)
```

---

## Text entry questions

```r
question_text(
  "Please enter the word 'C0rrect' below:",
  answer("correct", message = "Don't forget to capitalize"),
  answer("c0rrect", message = "Don't forget to capitalize"),
  answer("Correct", message = "Is it really an 'o'?"),
  answer("C0rrect ", message = "Make sure you do not have a trailing space"),
  answer("C0rrect", correct = TRUE),
  allow_retry = TRUE,
  trim = FALSE
)
```

---

## Code exercises - rendered

---

## Code exercises - code

---

# gradethis

---

## Code exercises - checking

---

## Checking the result - `grade_this()`

- Use a code chunk with the same label, suffixed with `-check`
- `.result` refers to the resulting output
- Think about ways things can go wrong and write test cases for them
- Write a "catch all" fail statement for everything else

---

## Checking code - `grade_this_code()`

---

## Keep in mind!

- Exercise chunks run in independent sessions, they don't actually work like R Markdown chunks (i.e. they don't remember what happened before)

- Use setup chunks to make the tutorial experience feel more like a data analysis story

- Leverage this feature to write robust code and checks

---

# Sharing

---

## Sharing with students

- You could share the R Markdown document (and all accompanying files) but that’s probably not what you want to do...

- Deploy on
  - shinyapps.io
  - RStudio Connect (free for academic use, requires setup)

- Road less travelled: [distribute as a package](https://education.rstudio.com/blog/2020/09/delivering-learnr-tutorials-in-a-package/)

- See the [publishing instructions](https://rstudio.github.io/learnr/publishing.html) on the learnr website for step-by-step instructions

---

# Recording data

---

## Recording attempts

- A "good enough" solution for formative exercises: embed a 
Google/Microsoft/etc. Form at the end and ask students to "submit" their work.

- This only records that the student reached the end of the tutorial and not how (or even if) they answered any of the questions or exercises.

- **Tip:** Add a free-text question to the form asking students to reflect on the exercises they just completed - you can then analyse the free-text data to gain insights into what students are struggling with -- "minute paper".

---

## Recording solutions

- The [`learnrhash`](https://rundel.github.io/learnrhash/) package builds on the previous method by providing a way for students to submit their answers by generating a text "hash" which can be copy and pasted into the web form.

```r
devtools::install_github("rundel/learnrhash")
```

- This package is being used to create the submission tool at the end of the sample tutorial.

.footnote[
.small[
See also the [**submitr**](https://github.com/dtkaplan/submitr) package by Danny Kaplan for a different approach to recording event data in learnr tutorials.
]
]

---

# Closing thoughts

---

## Best practices for automated feedback

- Measure twice, cut once (verify the correctness of your tests)

- Use rounding & type coercion to write robust tests

- Don't give automated feedback on everything, asking narrative questions that can't be auto checked but gets the student thinking and writing has pedagogical benefits

- Consider peer feedback where automated feedback is not feasible (e.g. interpretation, narrative) but scalability is an issue

---

Build a tutorial where students build develop their analysis in exercise code 
chunks (that are not checked) and only multiple choice questions are used for 
assessment.

---

.large[
.light-blue[
.hand[Q: I've built simple tutorials already. How do I make the jump to code checking and providing automated feedback that is actually useful?]
]
]

- Replicate an existing tutorial with checks (e.g. [this one](https://minecr.shinyapps.io/feedback-at-scale/)) and make incremental changes
- Read the [Testing](https://r-pkgs.org/tests.html#test-tests) chapter in R Packages (Wickham and Bryan) 
- Also read the [Metaprogramming](https://adv-r.hadley.nz/metaprogramming.html) section in Advanced R (Wickham)

---

- First, chances are you're not using these live, but you might be...
- If so, make sure to max out your instances and instance size on shinyapps.io.
- Essential reading:
  - [Publishing learnr Tutorials on shinyapps.io](https://cran.r-project.org/web/packages/learnr/vignettes/shinyapps-publishing.html) by Angela Li
  - [Teach R with learnr: a powerful tool for remote teaching](https://education.rstudio.com/blog/2020/05/learnr-for-remote/) by Allison Horst

---

* [dsbox](https://rstudio-education.github.io/dsbox/tutorials/index.html) - Tutorials accompanying [Data Science in a Box](https://datasciencebox.org/)

* [RStudio Cloud Primers](https://rstudio.cloud/learn/primers) - these are all learnr tutorials and their code is available on [github](https://github.com/rstudio-education/primers)

* [tutorialgRaficosFN](https://github.com/yabellini/tutorialgRaficosFN) by Yanina Saibene Bellini