```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = T,
                      eval=T)
library(rvest)
library(dplyr)
library(magrittr)
```

## define start page

We look for briefing statements from the White House

```{r}
# replace with your URL
startpage<-"https://www.whitehouse.gov/briefing-room/"
```

## generate urls of results pages

```{r}
searchresults<-read_html(startpage)
# choose last page of search results

last_page<-html_elements(searchresults,".page-numbers") %>% tail(1) %>% html_attr("href")

last_page

# extract number
library(stringr)
last_page <- str_extract(last_page,"[0-9]+")
last_page <- as.numeric(last_page)

# create resulturls
resulturls <- paste0(startpage,"page/", 1:last_page)

length(resulturls)
```

## Scrape Links to all statements

In the long exercise file, we discussed two options: merging the titles and URLs inside the loop or putting them in a list and extracting them afterwards. I do the second here because it is more robust to interruptions.

Additionally, note that I artificially restricted us to the first 10 * 10 results to speed the process up a bit.

```{r}
titles <- links <- dates <- list()


#for (i in 1:length(resulturls)){
for (i in 1:10){
  page<-read_html(resulturls[i])
  titles[[i]]<-html_elements(page,".news-item__title") %>% html_text()
  links[[i]]<-html_elements(page,".news-item__title") %>%
    html_attr("href")
  dates[[i]] <- html_elements(page,"time") %>% html_text()
}

results_df <- data.frame(titles=unlist(titles),
                         links=unlist(links),
                         dates=unlist(dates),
                         stringsAsFactors = F)
```

## Download statements

Now, we download all speeches for which we collected the URLs. I've also built in a time measure so you can get a feeling how long the scraping takes.

```{r}
start <- Sys.time()

speechtext <- rep("",length(results_df$links))
for (i in 1:length(results_df$links)){
  speechtext[i] <- read_html(results_df$links[i]) %>%
    html_elements(".body-content .row") %>%
    html_text()
}
results_df$text <- speechtext

end <- Sys.time()
end-start
```



## Inspect results

As an example: check statements 80-83.

```{r}
results_df[80:83,]

```

Can you analyze this data, using any of the methods we've used in the class?