```{r setup}
library(tidyverse)
library(quanteda)
```



```{r}
first_debate <- read.csv("../data/us_election_2020_1st_presidential_debate.csv",
  stringsAsFactors = F,encoding="UTF-8")

# optional / bonus : speaker variable with regular expressions
first_debate <- first_debate %>%
  mutate(speaker=str_extract(speaker,"[A-z]*$")) 

debate_corp <- corpus(first_debate)
# optional : renaming
docnames(debate_corp) <- paste0(1:nrow(first_debate),"_",
                                first_debate$speaker)
debate_toks <- tokens(debate_corp)

debate_dfm <- dfm(debate_toks)

```


# Pre-processing

Pre-processing has three different aspects

- restricting feature definitions
- removing uninformative features
- uniting features

## The structure of your dfm

First, some tools to understand what your dfm / tokens object currently looks like. There are specific commands to get 

- the number of tokens

```{r}
ntoken(debate_toks) %>% head()
```

- the number of features in a dfm

```{r}
nfeat(debate_dfm)
```

- the most frequent features

```{r}
topfeatures(debate_dfm)
```

- the number of documents

```{r}
ndoc(debate_dfm)
```

- Names of features as a vector

```{r}
 
featnames(debate_dfm) %>% head()
```

- frequency of features as a vector

```{r}
featfreq(debate_dfm) %>% head()
```

You can use many of these commands to check the results of your pre-processing.

## Restricting feature definitions

Example: Restrict features by removing punctuation, numbers and symbols

```{r}
# check feature number
nfeat(debate_dfm)

# restrict feature definition
debate_toks <- tokens(debate_corp,  remove_punct=T,
  remove_numbers=T, remove_symbols=T)
debate_dfm <- dfm(debate_toks)

# check new feature number
nfeat(debate_dfm)
```

## Removing uninformative features

Now try yourself!

- check stopwords in your language, e.g. with `stopwords("en")` - if you think it is useful, remove them

```{r}



```

`dfm_trim()` is an alternative that selects words based on their frequency.

- try the different options of `dfm_trim()`
    - remove features that occur less than 10 times in total
    - remove features occuring in less than 2 documents
    - remove features occuring more than 50 times

*Tip: If you specify verbose=T in your command, you see how much you remove.*


```{r}




```



## Uniting features

Stem the features of your dfm. How many features did you loose with that?

```{r}




```


# Transformations

We have already used some commands to transform dfms for pre-processing. 

- `dfm_subset()` : Selection based on docvars
- `dfm_group()` : grouping of documents based on docvars
- `dfm_select()`: Selection of features
    - `dfm_trim()`: Selection of features based on frequency
- `dfm_weight()` & `dfm_tfidf()` : weighting the feature counts
- `dfm_lookup()` : Looking up dictionaries


## Grouping

First, group your dataset by speaker. How did the number of documents change?

```{r}




```

## Subsetting

Subset your documents. If you still use the debate dataset, you are a bit limited but you could drop statements by Chris Wallace. Otherwise, try to drop certain years. Again, check the number of documents before and after.

```{r}




```

## Weighting

Finally, weigh your dfm: `dfm_weight()` allows you to weigh your dfm by different characteristics. You can consult the documentation to see the different options.

See how the `topfeatures()` change when you use binary weighting and relative frequency.

```{r}




```

A frequently applied weight is [*term-frequency-inverse-document-frequency*](https://en.wikipedia.org/wiki/Tf%E2%80%93idf). It is also implemented in quanteda with `dfm_tfidf()`. Look at the results of this weighting for your first documents. Is this more or less meaningful to you?

```{r}




``` 