```{r setup,warning=F,message=F}
library(tidyverse)
library(quanteda)
library(quanteda.textplots)
library(quanteda.textstats)
```


# Creating Objects

```{r}
first_debate <- read.csv("../data/us_election_2020_1st_presidential_debate.csv",
  stringsAsFactors = F,encoding="UTF-8")

# optional / bonus : speaker variable with regular expressions
first_debate <- first_debate %>%
  mutate(speaker=str_extract(speaker,"[A-z]*$")) 

debate_corp <- corpus(first_debate)
# optional : renaming
docnames(debate_corp) <- paste0(1:nrow(first_debate),"_",
                                first_debate$speaker)
```

## Tokenization

```{r}
tokens(debate_corp,"word")
tokens(debate_corp,"sentence")
tokens(debate_corp,"character")
debate_toks <- tokens(debate_corp)
```

## document-feature-matrix

```{r}
debate_dfm <- dfm(debate_toks)
```

# Descriptive analysis 

- where are terms used

```{r}
kwic(debate_toks, "crosstalk") %>%
  # restrict to first 15 occurances
  head(15) %>%
  textplot_xray()
```

- in which context are terms used? - `window` gives the number of features before or after

```{r}
kwic(debate_toks, "country",window=4)
```

- readability

```{r}
textstat_readability(debate_corp) %>% head()
```

- keyness statistics
    - `dfm_group()` is a helper function that groups all texts with the same docvar, here: speaker; this way we can compare all texts by a speaker to the other texts

```{r}
keyness <- dfm_group(debate_dfm,speaker) %>%
 textstat_keyness("Biden")

head(keyness)

keyness %>%
 textplot_keyness()
```

# Exercises

You can do the following tasks either on the same corpus or another corpus you find interesting.

- e.g. [Parlspeech](https://dataverse.harvard.edu/dataverse/ParlSpeech)
- e.g. [EUSpeech](https://dataverse.harvard.edu/dataverse/euspeech) / [2020 Version](https://osf.io/c79ha/)
- [Collection of example datasets](https://www.ics.uci.edu/~smyth/courses/cs175/text_data_sets.html)

## Exercise: Keywords in context

Using the commands above, check in which context the concepts of women and men are used.

*Tip: You can use the question mark as replacement of a single letter or the asteriks sign as a replacement of multiple letters, e.g. to include singular and plural.*


```{r}
kwic(debate_toks,"wom?n")
kwic(debate_toks,"m?n")
```

## Exercise: Keyness statistics

Using the commands above, find out what are the most typical words for one of actors in your dataset. If you are still using the debates, look at a different actor than before and try not to copy the code.


```{r}
keyness <- dfm_group(debate_dfm,speaker) %>%
 textstat_keyness("Trump")

head(keyness)

keyness %>%
 textplot_keyness()
```

## Exercise: Readability

Using the commands above, calculate readability scores for the speakers. You can check the documentation of `textstat_readability()` to see different metrics.

Merge them to the original data - you might have to create a document variable to do so. Then use your data wrangling skills to calculate who is on average the most readable.

```{r}
# calculate readability
readability_stats <- textstat_readability(debate_corp)

# merging
first_debate$document <- paste0(1:nrow(first_debate),"_",
                                first_debate$speaker)
readability_stats <- left_join(first_debate,readability_stats)

# analyze
readability_stats %>% 
  group_by(speaker) %>%
  select(Flesch) %>%
  summarize_all(mean)
```
