```{r setup}
library(tidyverse)
library(quanteda)
library(quanteda.textstats)
```



```{r}
first_debate <- read.csv("../data/us_election_2020_1st_presidential_debate.csv",
  stringsAsFactors = F,encoding="UTF-8")

# optional / bonus : speaker variable with regular expressions
first_debate <- first_debate %>%
  mutate(speaker=str_extract(speaker,"[A-z]*$")) 

debate_corp <- corpus(first_debate)
# optional : renaming
docnames(debate_corp) <- paste0(1:nrow(first_debate),"_",
                                first_debate$speaker)
debate_toks <- tokens(debate_corp)

debate_dfm <- dfm(debate_toks)

```

## Loading and applying a dictionary

```{r}
# read in dictionary in yaml-format
newsmap_dict <- dictionary(file = "english.yml",
                           format = "YAML")
newsmap_dict
# apply dictionary at dfm level
dict_dfm_results <- dfm_lookup(debate_dfm,newsmap_dict)
# snippet
dict_dfm_results[650:655,111:113]

# apply dictionary at tokens level
dict_toks_results <- tokens_lookup(debate_toks,newsmap_dict)
# snippet
dict_toks_results[650:655]
```


Dictionary results with `textstat_frequency()`

```{r}
dict_dfm_results %>% textstat_frequency()
```

## Differences between tokens and dfm

We can apply dictionaries to tokens and dfms. How are the results different?


```{r}
newsmap_dict <- dictionary(file = "english.yml",
                           format = "YAML")
debate_dfm %>% dfm_lookup(newsmap_dict) %>% 
  textstat_frequency() %>% head(4)

debate_toks %>% tokens_lookup(newsmap_dict) %>% 
  dfm() %>% 
  textstat_frequency() %>% head(4)

```

To understand, have a look at the dictionary key and what it selects in specific texts:

```{r}






```


# Exercises

You can do the following tasks either on the same corpus or another corpus you find interesting.

- e.g. [Parlspeech](https://dataverse.harvard.edu/dataverse/ParlSpeech)
- e.g. [EUSpeech](https://dataverse.harvard.edu/dataverse/euspeech) / [2020 Version](https://osf.io/c79ha/)
- [Collection of example datasets](https://www.ics.uci.edu/~smyth/courses/cs175/text_data_sets.html)

You can keep using the newsmap dictionary or use a different dictionary

- built into quanteda, you find 2015 Lexicoder Sentiment Dictionary in quanteda dictionary format by typing `data_dictionary_LSD2015`
- several dictionaries in wordstat format are available [here](https://provalisresearch.com/products/content-analysis-software/wordstat-dictionary/), most for free
- [Moral foundations dictionary](https://moralfoundations.org/other-materials/)
- with a bit more effort, you could translate any wordlist / thesaurus you find into a dictionary, e.g.: http://gender-decoder.katmatfield.com/about#masculine

## Exercise: Grouping

You can also apply dictionaries by group, using docvars to form these groups. For that, you can either group the dfm before applying the dictionary with `dfm_group()` or specify the group inside the `textstat_frequency()` command.

**&rarr; E.g. How do the candidates differ in geographic focus?**

*Or, a different grouping variable if you chose to use your own data*

```{r}




```

## Exercise: Levels

As you may have seen when having a look at the newsmap dictionary, this is dictionary with multiple levels - continents, regions and countries.

While the dictionary normally gives you results for all levels, you can also apply each level separately by specifying `levels=1` (or a different level) inside the `dfm_lookup()` command.

Apply the first level of the newsmap dictionary - if you now work with a different dictionary, try out if this nevertheless works or if your dictionary only has a single level.


```{r}



```


## Exercise: Weighting

`dfm_weight()` allows you to weigh your dfm by different characteristics. You can consult the documentation to see the different options.

Try relative frequency and a binary weighting (all non-zeros coded as 1) - How do the results for textstat_frequency differ?

```{r}



```

# Exercise: Dictionary Creation

I've asked you to think about a concept you want to measure and what words might be indicators for this concept.

Check the documentation of the `dictionary()` function for the syntax of specifying your own dictionary and create it. Apply it to the presidential debate or your dataset.

If you do not have an idea for your own concept: Try to create a topic-dictionary (e.g. economy, covid, foreign relations, ...) and apply it to the presidential debate.

```{r}





```

# Exercise: Wrangling

*This requires a better understanding of R - feel free to skip the exercises if you're a beginner*

We can use our dictionary to classify the statements with a bit of wrangling. For that, it is often easier to transform the results into a `data.frame` and work with them outside of quanteda.

Use `convert()` to convert the results of applying the dictionary into a dataframe and bind them to the original data

```{r}


```

Now, we need to find a decision rule: 

- e.g. assign each statement to the class with most hits
- e.g. assign only statements to a class that only have hits from one dictionary key

There are several ways to implement this and they are a question of *data management*, not text analysis - one is using case_when() inside a mutate() function.

```{r}





```



# Transfer: EUI Theses

In preparation for this class, I downloaded all thesis abstracts listed in cadmus. You can download the data from the course webpage. 

Have a look at how thesis abstracts of the different departments consider different continents. Caution: For some theses, cadmus has no abstract. You may want to filter those out. And maybe you want to plot the results?

```{r}

```


# Other Options

Another idea you might consider is a [targeted dictionary analysis](https://tutorials.quanteda.io/advanced-operations/targeted-dictionary-analysis/) that applies the dictionary only to words surrounding a concept - e.g. to find sentiment about a topic or the way media talk about male and female politicians.