```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = T,
                      eval=T)
```



# 'Parsing HTML'

We learn reading web data into R. As cosmopolitan political scientists, we want to learn something about US politics from [The Presidency Project](https://www.presidency.ucsb.edu/) of UC Santa Barbara.
Visit the page and search for a keyword or president you are interested in. Then, copy the URL and assign it to a new object.

```{r,eval=F}
browseURL("https://www.presidency.ucsb.edu/")
```

Now, let's try to scrape this page

- **First**, we need to load the package we'll use for most of our scraping. It is called *rvest*. Please load it - for example with the `library()` command.
- **Next**, read in (sometimes called: parse) the webpage of which you copied the URL. To tell R to read the webpage, we can use the function `read_html()` on the url object you just created - assign the result to a new object
- **Then**, call the object to which you assigned the page

```{r}
library(rvest)
url <- "https://www.presidency.ucsb.edu/advanced-search?field-keywords=crisis&items_per_page=25"
searchresults <- read_html(url)
searchresults
```


# Tables

We start with something satisfying: Out of the box, `rvest` converts tables into a list of data frames when calling `html_table()` on a page. Use this on the webpage you parsed.

```{r}
html_table(searchresults)
```

If you assign the result to an object, the object will be a list.
You can extract specific tables from this list by subsetting the list (that is, putting the number of the table you want in two squared brackets). Or, if you want to proceed in a piping-chain, you can use the command `extract2()` from the `magrittr` package, adding the number of the table in brackets (the command name is no typo - `extract` without the 2 works for vectors, `extract2()` works for lists).

**Try both variants for extracting the results table.**


```{r}
tables <- html_table(searchresults)

tables[[1]]
library(magrittr)
extract2(tables,1)
```


## Further Exercise: Tables


Check this [list of Eiffel Tower replicas and derivatives](https://en.wikipedia.org/wiki/Eiffel_Tower_replicas_and_derivatives). Scrape the table and count which country has the most Eiffel tower replicas.

```{r}







```

The [Watergate scandal](https://en.wikipedia.org/wiki/Watergate_scandal) still shapes how we think of political scandals today and has become an inspiration for the names of many other scandals.

Visit the [list of scandals that end with 'gate'](https://en.wikipedia.org/wiki/List_of_%22-gate%22_scandals). Scrape the table that contains scandals from the domain of politics. Plot the distribution of these scandals over time - which year was a high point in political scandals ending in -gate?

If you want, compare this to another domain of scandals - did the use of 'gate' for scandals follow a similar pattern?

```{r}












```

# Extracting elements

Now we got tables - but how do we get the rest of the webpage? To make it more interesting, select a speech from your results (we will learn how to do this automatically but for now, you can do this by visiting the URL in your browser again) on which we will practice scraping the text of a page.

The function `read_html()` parses the html code, similar to what our browser does. Still, it gives us the entire document including the HTML commands. Since we do not want the formatting of the webpage, we can use the function html_text() to extract the Webpage text.

Try it out: Parse the URL of the speech you are interested in and apply `html_text()` to it.

```{r}


```

Did you find the speech you saw in your browser? Admittedly, this still looks very messy. 
Maybe you are thinking: If only, there would be a way to tell R to just get the text of the speech! Luckily there is.

# Selecting parts of pages

## CSS Selectors

The `html_elements()` command allows us to select specific 'elements' of the HTML Code. One example would be the HTML tags we learned about and their respective content. You can have a look at the documentation of the `html_elements()` command for more information.

We will get into  CSS selectors in a bit. For now, we will focus on two of the most important selectors:

- in their most basic form, so-called type-selectors selectors work on *types of HTML tags* - so if you write the name of a tag (without the brackets), the CSS selector will select all elements with that tag. For example, "h1" will select everything inside a `<h1>` tag
    - **Try it out with some of the HTML tags that you remember.** If you don't remember any, try [headings](https://www.w3schools.com/html/html_headings.asp)
- a very useful selector is the star-symbol - it just selects *all* tags in the page (so we also call it a universal selector)
    - **Try the universal selector on the webpage.**
- try to **combine this with the `html_text()` command** to extract just the text rather than the entire element with its formatting

```{r}















```


## Further Exercise: CSS Selectors


We will learn how to select parts of specific webpages soon. However, since webpages are built with a similar architecture, often, we get quite far by selecting generic elements.

Visit the [webpage of the guardian](https://www.theguardian.com/) and extract the headlines of different levels until you do not find anything anymore. Can you identify how the different levels are used in the webpage by looking at the original page? Remember that since this is a new webpage, you will have to  combine the commands we have learned so far, namely `read_html()`, `html_elements()` and `html_text()`.

```{r}






```


## The Grammar of CSS Selectors

Now that we practiced the `html_elements()` command a bit, we move beyond just selecting all headlines, all links etc. towards specific headlines, links or elements more generally.

Basically, HTML Tags describe the formatting and structure of a webpage. CSS selectors are a type of *grammar* or *pattern-description* that helps us select specific parts of that structure depending on the look, positioning etc. For this we use the classes and attributes that further define HTML tags.

CSS selectors follow a specific grammar that you do not need to memorize but might want to look up when you are scraping.   Below, you find an overview of this grammar.  

### Basic selectors

<table border=1 width="100%">
<tr>
<td width="25%">*</td>
<td width="15%">universal selector</td>
<td width="70%">Matches everything.</td>
</tr>
<tr>
<td>element</td>
<td>type selector</td>
<td>Matches an element</td>
</tr>
<tr>
<td>&lsqb;attribute&rsqb;</td>
<td>attribute selector</td>
<td>Matches elements containing a given attribute</td>
</tr>
<tr>
<td>&lsqb;attribute=value&rsqb;</td>
<td>attribute selector</td>
<td>Matches elements containing a given attribute with a given value</td>
</tr>
<tr>
<td>.class</td>
<td>class selector</td>
<td>Matches the value of a class attribute</td>
</tr>
<tr>
<td>#id</td>
<td>ID selector</td>
<td>Matches the value of an id attribute</td>
</tr>
</table>

### More complex attribute selectors

<table border=1 width="100%">
<tr>
<td>&lsqb;attribute*=value&rsqb;</td>
<td>Matches elements with an attribute that contains a given value</td>
<td>a&lsqb;href*="pressrelease"&rsqb;</td>
</tr>
<tr>
<td>&lsqb;attribute^="value"&rsqb;</td>
<td>Matches elements with an attribute that starts with a given value</td>
<td>a&lsqb;href*="/press/"&rsqb;</td>
</tr>
<tr>
<td>&lsqb;attribute&dollar;="value"&rsqb;</td>
<td>Matches elements with an attribute that ends with a given value</td>
<td>&lsqb;href$=".pdf"&rsqb;</td>
</tr>
</table>

### Combining CSS Selectors

There are several ways to combine CSS Selectors:

<table border=1 width="100%">
<tr><td>element,element     </td>

<td>Selects all &lt;>div&gt; elements and all &lt;>p&gt; elements</td> <td>div, p   </td></tr>
<tr><td>element element     </td>

<td>Selects all &lt;>p&gt; elements inside &lt;>div&gt; elements</td> <td>div p     </td></tr>
<tr><td>element>element     </td>

<td>Selects all &lt;>p&gt; elements where the parent is a &lt;>div&gt; element</td> <td>div > p     </td></tr>
<tr><td>element+element     </td>

<td>Selects all &lt;>p&gt; elements that are placed immediately after &lt;>div&gt; elements</td><td>div + p     </td> </tr>
<tr><td>element1~element2   </td>

<td>Selects every &lt;ul&gt; element that are preceded by a &lt;p&gt; element</td> <td>p ~ ul   </td></tr>
</table>

If you want to practice CSS Selectors, the [w3schools](https://www.w3schools.com/cssref/trysel.asp) has a test playground where you can try out lots of more complex selectors and read up on them.


If you want to practice the logic of CSS Selectors in a fun way, play with the [CSS Diner](https://flukeout.github.io/) where you can also learn about different selector structures.



### Using SelectorGadget

While understanding HTML helps, we often do not need to engage with the code because there are lots of tools to help us. That is: We can get some help at doing many of the things we just learned painfully!

For example, SelectorGadget is a JavaScript bookmarklet that allows you to interactively figure out what css selector you need to extract parts of the page. If you have not heard of selectorgadget, check its [webpage](https://selectorgadget.com/).

We will try to use SelectorGadget now. [If you have Chrome, you can just install SelectorGadget in your browser](https://chrome.google.com/webstore/detail/selectorgadget/mhjhnkcfbdhnjickkkdbjoemdmbfginb). If you have a different browser, drag this link into your [bookmark bar and click on it when needed](javascript:(function(){var%20s=document.createElement('div');s.innerHTML='Loading...';s.style.color='black';s.style.padding='20px';s.style.position='fixed';s.style.zIndex='9999';s.style.fontSize='3.0em';s.style.border='2px%20solid%20black';s.style.right='40px';s.style.top='40px';s.setAttribute('class','selector_gadget_loading');s.style.background='white';document.body.appendChild(s);s=document.createElement('script');s.setAttribute('type','text/javascript');s.setAttribute('src','https://dv0akt2986vzh.cloudfront.net/unstable/lib/selectorgadget.js');document.body.appendChild(s);})();).
Now, use it to select the text on the speech webpage you have used.

1. Click on the element you want to select. SelectorGadget will make a first guess at what css selector you want and mark all similar elements. It's likely to be a bad guess since it only has one example to learn from, but it's a start. Elements that match the selector will be highlighted in yellow.
2. Click on elements that shouldn't be selected. They will turn red.  Click on elements that *should* be selected but are not so far. They will turn green.
3. Iterate until only the elements you want are selected.  SelectorGadget is not perfect and sometimes will not be able to find a useful css selector. Sometimes starting from a different element helps, sometimes having a look at the HTML code and thinking of the grammar we learned above will help you.



## Applying CSS Selectors

Now, we try to use these CSS Selectors with the `html_elements()` command. This is a bit of repetition from before but it aids the memory: Parse the page, use the CSS selector to select only the text of the speech assign it to a new object `selected`. Then, inspect the results by calling the object!


```{r}



```

This already looks more structured - but we should get rid of the HTML tags. Apply the `html_text()` command we used before to the elements which we selected in the last step. This way, we get just the text from the elements we selected.

```{r}



```


# Extracting links 


So, you have learned how to use some of the basic functions of the `rvest` package: `read_html()`, `html_elements()`, `html_table()` and `html_text()`.
But most of the time, we are not just interested in a single page but multiple pages from the same domain, e.g. all newspaper reports by a specific newspaper or all speeches by a politician. So we need another step: We have to learn to follow links without actually opening the browser and clicking on the link and copying the new path.

Now, we will learn two things:

- to extract links from webpages
- ways to automate following these links


## Extracting links from webpages

To extract links, we need another command. Remember that hyperlinks are an *attribute* of the text? This means, they are not directly visible on the page but bare an attribute of the elements we see. Because of that, the `rvest` command to get these links is called `html_attr()`. We can use it to extract different types of attributes, so you will have to tell rvest the attribute that we are interested in is a link. Remember what links looked like?

`This is text <a href="https://www.presidency.ucsb.edu/">with a link</a>.`

*href* stands for hyperreference and signifies the webpage the link leads to. You can specify `name="href"` inside the `html_attr()` command to extract the link. For example:

`html_attr(object,"href")`

However, this will only work on individual HTML tags not on entire pages (since the link is an attribute of the specific tag, not the page), so we will use `html_elements()` again. Please try three things on the original search result overview page you stored:

- first, you can extract potential links from all HTML tags by using the **universal selector** (`html_elements("*")`)
- second, extract all links from **a-tags** - is there any difference?
- look up the selector of the **articles** and extract the links from those tags


```{r}





```


Do you notice something about the links? They are missing a part. That is because they are relative links within the directory structure of the webpage. To 'repair' them, we need to add the **base url** of the webpage. This is typically just the url of the webpage we originally scraped from.

For adding the base url, we can use the function `paste()` that pastes together two character vectors. I recommend using `paste0()` which pastes the vectors together without inserting separators like white space between the vectors. If you have never used paste, try it out:

```{r}
paste("a","b")
paste0("a","b")
```

Now, completing the paths of the URLs we scraped should not be a problem for you. Re-use the code you used to extract the links of the tags, assign it to an object called `url` and add the base url (https://www.presidency.ucsb.edu) in front of it.

*Watch out for the slashes between the base url and the address of your page - having none or too many slashes is a typical problem!*


```{r}




```

# Automation

Now, we will learn how to follow multiple links and automate this process in different ways.

You will see that in the end, webscraping is a function of programming in R. The more you learn to use loops, functions and apply commands, the easier the scraping will be. In the end, scraping is just a small step in the whole process of getting data.

So far, we already learned to collect all the links we are interested in and learned the basic commands of rvest. Now, there are multiple ways to proceed:

- you can use a `for()`-loop that loops over the vector of links and scrapes each of them
- You can write a function that scrapes the content of all the links
    - you can put the function into your loop
    - you can `apply()` the function to a vector - this is the fastest variant but takes some getting used to

For today, we will start with the easiest variant and just create a `for`-loop. Later, we will also use `apply()` but there are good reasons why you will often return to simple loops.

We again start by scraping the results of your query. I recommend to first write down a few lines of code as if you would just want to scrape the first link:

- extract the links to the pages for each speech
- parse the first of the speech pages
- extract the speech elements on this page and just keep the text

This is also a good exercise to see to which extent you remember what we have learned so far. You can then think about re-writing the code in the next step.


```{r}










```

Now, we to analyze the code: which part will vary when you try to repeat this multiple times?
If you have figured this out, it is time to try it!

```{r}






```




# Collecting results from multiple pages

So far, we are still rather limited in our data collection: We have just collected the speeches from the first page.

Now, we write a loop that runs through all pages and collects

- the table of results
- the links to each speech.

### How does the page url change?


It would be tedious to extract the links to all further pages - especially because not all of them are listed on the first page. Instead, we need to find a **logic behind the sequence of pages**. Click through the first few pages of results - what changes about the url as you do so?

While we are looking at the URL: Generally, we try to collect as much data per page as possible. Do you find where the URL says the number of results per pages? Change the URL to include a higher number and copy-paste your code to scrape the table and the links - do you find more links or do you need to adapt more?

```{r}









```


### Looping through pages

To do this not for a fixed number of pages but all pages, we need to find out the number of results pages. We can simply do this by returning to the page and checking what is the number of the last page.

Or, we try to automate this by extracting the link to the last page using SelectorGadget. We can then use [regular expressions](https://www.regular-expressions.info/) to automatically extract the number of the last page. I've pre-written this, given we do not cover Regular Expressions in this course but you can check this Cheat Sheet if you want:

[Regular Expressions Cheat sheet](https://evoldyn.gitlab.io/evomics-2018/ref-sheets/R_strings.pdf) 

**Advice: Try to set the number of results per page as high as you can to reduce the number of pages you have to visit**. We do this here by setting the items per page to 100.


```{r,error=T}
# replace with your URL
url<-"https://www.presidency.ucsb.edu/advanced-search?field-keywords=crisis&items_per_page=100"

searchresults<-read_html(url)
# choose last page of search results

last_page<-html_elements(page,".pager-last a") %>% html_attr("href")
library(stringr)
last_page <- str_extract(last_page,"[0-9]*$")
last_page <- as.numeric(last_page)
```

Now, try to create the URLs for all pages. A hint: you can often also reach the first page by specifying `page=0` (in this case) or `page=1` (in those cases where the second page has an address that includes the number 2).

```{r}

```



Now, we can write a loop that goes through all pages and collects the results tables and links.


For this, we need to create empty objects for the tables and URLs we want to collect. We can either progressively bind the tables together each time we loop through the code or store them in a list and combine them afterwards - i have pre-written code for both.


```{r,error=T}

# OPTION 1: we merge the tables and URLs inside the loop

tables<-NULL
URLs<-NULL



for (i in 1:length(resulturls)) {
  
  page<-read_html(resulturls[i])
  tables<-bind_rows(tables,
                   html_table(page))
  URLs<-c(URLs,
          html_elements(page,".views-field-title a") %>% 
          html_attr("href"))
}

tables$URL<-URLs



# OPTION 2: we put them in a list and afterwards extract and merge them

empty_list<- list()
empty_list2 <- list()


for (i in 1:length(resulturls)){
  page<-read_html(paste0(base,i))
  empty_list[[i]]<-html_table(page) %>% extract2(1)
  empty_list2[[i]]<-html_elements(page,".views-field-title a") %>%
    html_attr("href")
}
full_table <- do.call("rbind",empty_list)
all_links <- unlist(empty_list2)

```

Based on this, you're ready to collect the text of all speeches!

You can try this with two levels of difficulty:

If you prefer to repeat what we've done to make sure you've understood each part, **open a new R file and try to write the scraper from scratch!** [05_singlefile.rmd](http://theresagessler.eu/eui_cta/material/05_singlefile.rmd)

If you think you can adapt this to a new webpage, **write a similar scraper for [White House Briefing Statements](https://www.whitehouse.gov/briefings-statements/) or a similar page you want to scrape.** [05_scraping_briefings.rmd](http://theresagessler.eu/eui_cta/material/05_scraping_briefings.rmd)

Can you analyze this data, using any of the methods we've used in the class?

(And if you want to adapt this to download pdf files - as some of you suggest, I uploaded another example script that I've used for European Central Bank Texts in a different class: [example_imf.rmd](http://theresagessler.eu/eui_cta/material/example_imf.rmd) )