class: center, middle, inverse, title-slide # Webscraping
🕸 ### Dr. Çetinkaya-Rundel --- layout: true <div class="my-footer"> <span> Dr. Mine Çetinkaya-Rundel - <a href="https://introds.org" target="_blank">introds.org </a> </span> </div> --- ## Week 6 - Web scraping, and automating it via the use of functions and iteration (which we'll revisit many times over the rest of the semester) - Team peer evaluations - due Thursday 17:00 (look for an email from TEAMMATES) - HW and OQ scores are in gradebook - For OQ: 1 means you did it, 0 means you didn't - For HW: See repo issues for actual feedback - Workshop tomorrow - Take a seat where everyone can work together and have access to a computer. If you know you'll be a few minutes late, let your teammates know so they can save a spot for you. .question[ .large[ Any questions? ] ] --- class: center, middle # Scraping the web --- ## Scraping the web: what? why? - Increasing amount of data is available on the web -- - These data are provided in an unstructured format: you can always copy&paste, but it's time-consuming and prone to errors -- - Web scraping is the process of extracting this information automatically and transform it into a structured dataset -- - Two different scenarios: - Screen scraping: extract data from source code of website, with html parser (easy) or regular expression matching (less easy). - Web APIs (application programming interface): website offers a set of structured http requests that return JSON or XML files. --- class: center, middle # Web Scraping with rvest --- ## Hypertext Markup Language - Most of the data on the web is still largely available as HTML - It is structured (hierarchical / tree based), but it''s often not available in a form useful for analysis (flat / tidy). ```html <html> <head> <title>This is a title</title> </head> <body> <p align="center">Hello world!</p> </body> </html> ``` --- ## rvest .pull-left[ - The **rvest** package makes basic processing and manipulation of HTML data straight forward - It's designed to work with pipelines built with `%>%` ] .pull-right[ <img src="img/rvest.png" width="230" style="display: block; margin: auto 0 auto auto;" /> ] --- ## Core rvest functions - `read_html` - Read HTML data from a url or character string - `html_node ` - Select a specified node from HTML document - `html_nodes` - Select specified nodes from HTML document - `html_table` - Parse an HTML table into a data frame - `html_text` - Extract tag pairs' content - `html_name` - Extract tags' names - `html_attrs` - Extract all of each tag's attributes - `html_attr` - Extract tags' attribute value by name --- ## SelectorGadget .pull-left[ - Open source tool that eases CSS selector generation and discovery - Easiest to use with the [Chrome Extension](https://chrome.google.com/webstore/detail/selectorgadget/mhjhnkcfbdhnjickkkdbjoemdmbfginb) - Find out more on the [SelectorGadget vignette](https://cran.r-project.org/web/packages/rvest/vignettes/selectorgadget.html) ] .pull-right[ <img src="img/selector-gadget.png" width="456" /> ] --- ## Using the SelectorGadget .pull-left[ - Click on the app logo next to the search bar - A box will open in the bottom right of the website ] .pull-right[ <img src="img/selector-gadget.gif" height="250" style="display: block; margin: auto;" /> ] -- - Click on a page element (it will turn green), SelectorGadget will generate a minimal CSS selector for that element, and will highlight (yellow) everything that is matched by the selector -- - Click on a highlighted element to remove it from the selector (red), or click on an unhighlighted element to add it to the selector -- - Through this process of selection and rejection, SelectorGadget helps you come up with the appropriate CSS selector for your needs --- class: center, middle # Top 250 movies on IMDB --- ## Top 250 movies on IMDB Take a look at the source code, look for the tag `table` tag: <br> http://www.imdb.com/chart/top ![imdb_top](img/imdb_top_250.png) --- ## First check if you're allowed! ```r library(robotstxt) paths_allowed("http://www.imdb.com") ``` ``` ## www.imdb.com No encoding supplied: defaulting to UTF-8. ``` ``` ## [1] TRUE ``` vs. e.g. ```r paths_allowed("http://www.facebook.com") ``` ``` ## www.facebook.com ``` ``` ## [1] FALSE ``` --- ## <i class="fas fa-laptop"></i> Hands on - IMDB 250 - Go to [rstudio.cloud](https://rstudio.cloud/spaces/34062/projects) - Start the assignment called *Hands on - Webscraping* - Open `01-imdb-250movies.R` - Follow along, and fill in the blanks if you like --- ## Select and format pieces .small[ ```r page <- read_html("http://www.imdb.com/chart/top") titles <- page %>% html_nodes(".titleColumn a") %>% html_text() years <- page %>% html_nodes(".secondaryInfo") %>% html_text() %>% str_replace("\\(", "") %>% # remove ( str_replace("\\)", "") %>% # remove ) as.numeric() scores <- page %>% html_nodes("#main strong") %>% html_text() %>% as.numeric() imdb_top_250 <- tibble( title = titles, year = years, score = scores ) ``` ] ---
--- ## Clean up / enhance May or may not be a lot of work depending on how messy the data are - See if you like what you got: ```r glimpse(imdb_top_250) ``` ``` ## Observations: 250 ## Variables: 3 ## $ title <chr> "The Shawshank Redemption", "The Godfather", "The Godfathe… ## $ year <dbl> 1994, 1972, 1974, 2008, 1957, 1993, 2003, 1994, 1966, 1999… ## $ score <dbl> 9.2, 9.1, 9.0, 9.0, 8.9, 8.9, 8.9, 8.9, 8.8, 8.8, 8.8, 8.8… ``` - Add a variable for rank ```r imdb_top_250 <- imdb_top_250 %>% mutate(rank = 1:nrow(imdb_top_250)) ``` ---
--- ## Analyze .question[ How would you go about answering this question: Which 1995 movies made the list? ] -- ```r imdb_top_250 %>% filter(year == 1995) ``` ``` ## # A tibble: 8 x 4 ## title year score rank ## <chr> <dbl> <dbl> <int> ## 1 Seven 1995 8.6 21 ## 2 The Usual Suspects 1995 8.5 33 ## 3 Braveheart 1995 8.3 77 ## 4 Toy Story 1995 8.3 82 ## 5 Heat 1995 8.2 121 ## 6 Casino 1995 8.2 139 ## 7 Before Sunrise 1995 8.1 194 ## 8 La Haine 1995 8 230 ``` --- ## Analyze .question[ How would you go about answering this question: Which years have the most movies on the list? ] -- ```r imdb_top_250 %>% group_by(year) %>% summarise(total = n()) %>% arrange(desc(total)) %>% head(5) ``` ``` ## # A tibble: 5 x 2 ## year total ## <dbl> <int> ## 1 1995 8 ## 2 1957 7 ## 3 2004 7 ## 4 2014 7 ## 5 2000 6 ``` --- ## Visualize .question[ How would you go about creating this visualization: Visualize the average yearly score for movies that made it on the top 250 list over time. ] -- .small[ <img src="w6_d1-webscraping_files/figure-html/unnamed-chunk-14-1.png" width="1500" /> ] --- ## Potential challenges - Unreliable formatting at the source - Data broken into many pages - ... .question[ Compare the display of information at [gumtree.com/cars-vans-motorbikes/edinburgh](https://www.gumtree.com/cars-vans-motorbikes/edinburgh) to the list on the IMDB top 250 list. What challenges can you foresee in scraping a list of the available apartments? ] --- ## <i class="fas fa-laptop"></i> Hands on - IMDB TV Shows - Continue the assignment in [rstudio.cloud](https://rstudio.cloud/spaces/34062/projects): *Hands on - Webscraping* - Open `02-imdb-tvshows.R` - Scrape the names, scores, and years of most popular TV shows on IMDB: [www.imdb.com/chart/tvmeter](http://www.imdb.com/chart/tvmeter) - Create a data frame called `tvshows` with four variables: `rank`, `name`, `score`, `year` - Examine each of the **first three** TV shows to also obtain .midi[ - Genre - Runtime - How many episodes so far - First five plot keywords ] - Add this information to the `tvshows` data frame you created earlier