rbtl - Data Science Lifecycle

Lars Schöbitz

Global Health Engineering - ETH Zurich

2022-04-29

Welcome back!

You got your data!

via GIPHY

What’s happening next?

Today

1. Classroom tools
2. Data Science Lifecycle
3. R basics: Functions, Arguments, Objects, Operators
4. Live Coding Exercise
5. Pair Programming Exercise
6. Homework and Project Report

Learning Objectives

1. Learners can import their data from a CSV file to a team repository on GitHub
2. Learners can list the six elements of the data science lifecycle
3. Learners know three different ways of getting support in solving coding problems online

Classroom tools

Live Coding Exercises

• Instructor writes and narrates code out loud
• Intstructor explains elements and principles that are relevant
• Code is displayed on projector screen
• Learners join by writing and executing the same code
• Learners “code-along” with the instructor

Pair Programming Exercises

• Two learners work together on one computer
• One person (the driver) does the typing
• The other person (the navigator) offers comments and suggestions
• Roles get switched

Taking Notes Together

• Questions in shared online document: https://docs.google.com/document/d/1B_fGhU2-p7GdMjDdRq73JXAhAM7VU2JDu6t1foSQ0og/edit

Sticky Notes

• Use as status flags
• Orange: Exercise completed
• Pink: Problem, need support

Data Science Lifecycle

Deep End

via GIPHY

1

Add notes

2

1. First you must import your data into R. This typically means that you take data stored:

• in a file
• in a database,
• or web application, and load it into R.

2. If you can get your data into R, you can do data science on it!

3

1. Once you’ve imported your data, it is a good idea to tidy it.

2. This means to get it into a structure that lets you focus on the questions about your data and not the format it needs to be in

3. This is the step where we spend by far the biggest amount of our time.

4. It has been estimated that we spend about 80% of our time with cleaning and tidying data, so that we can use it fo our analyses.

5. In my courses you will learn how to structure your data in a tidy format as you collect it, and before you import it. A highly underestimated skill.

4

1. Once your data is tidy, a common first step is to transform it.

2. This includes

• narrowing in on what interests you (like all people without toilets in one district, or all data from last year)
• creating new variables from existing (like toilet density, as in number of people per toilet)
• calculating summary statistics (like counts or the mean)

Together, tidying and transforming are called wrangling, because getting your data in a form that’s natural to work with often feels like a fight!

5

1. Once you have tidy data with the variables you need, there are two main engines of knowledge generation:

• visualisation
• and modelling.

2. These have complementary strengths and weaknesses so any real analysis will iterate between them many times.

3. Visualisation is a fundamentally human activity. A good visualisation will show you things that you did not expect, or raise new questions about the data.

6

1. Models are complementary tools to visualisation. Once you have made your questions sufficiently precise, you can use a model to answer them.

2. In my courses, they play a very small part, but I am confident that you will be able to enhance your knowledge on modelling once you understand the basic principles of using the R lanuage.

7

1. The last step of data science is communication, an absolutely critical part of any data analysis project.

2. It doesn’t matter how well your models and visualisation have led you to understand the data, unless you can also communicate your results to others.

3. And this is a part where R has become really strong.

R

Packages

base R

sqrt(49)
sum(1, 2)

• Functions come with R

R Packages

library(dplyr)

• Installed once in the Console: install.packages("dplyr")
• Loaded per script

Packages

• Main way that reusable code is shared in R
• Combination of code, data, and documentation
• R has a rich ecosystem of packages for data manipulation & analysis

Functions & Arguments

library(dplyr)

filter(.data = gapminder, 
       year == 2007)

• Function: filter()
• Argument: .data =
• Arguments following: year == 2007 What do do with the data

Objects

library(dplyr)

gapminder_yr_2007 <- filter(.data = gapminder, 
       year == 2007)

• Function: filter()
• Argument: .data =
• Arguments following: year == 2007 What do do with the data
• Object: gapminder_yr_2007

Operators

library(dplyr)

gapminder_yr_2007 <- gapminder %>% 
  filter(year == 2007) 

• Function: filter()
• Argument: .data =
• Arguments following: year == 2007 What do do with the data
• Object: gapminder_yr_2007
• Assignment operator: <-
• Pipe operator: %<%

Rules

Rules of dplyr functions:

• First argument is always a data frame
• Subsequent arguments say what to do with that data frame
• Always return a data frame
  
• Don’t modify in place

Live Coding Exercise

ae-10-data-science-lifecycle

1. Head over to the GitHub Organisation for the course.
2. Find the repo for week 10 that has your GitHub username.
3. Clone the repo with your username to the RStudio Cloud.
4. Open the file: ae-10a-lifecycle.qmd
5. Use your Sticky Notes to let me know when you are ready.

Break

15:00

Photo by Blake Wisz

Solving coding problems

Tipps for search engines

• Use describe verbs that describe what you want to do
• Be specific
• Add R to the search query
• Add the name of the R package name to the search query
• Scroll through the top 5 results (don’t just pick the first)

Example: “How to remove a legend from a plot in R ggplot2”

Stack Overflow

What is it?

• The biggest support network for (coding) problems
• Can be initimidating at first
• Upvote system

Workflow

• First, briefly read the question that was posted
• Then, read the answer marked as “correct”
• Then, read one or two more answers with high votes
• Then, check out the “Linked” posts
• Always give credit for the solution

Give credit

Give credit

Give credit

ggplot(data = global_waste_data_kg_year,
       mapping = aes(x = income_id, 
                     y = capita_kg_year,
                     color = income_id)) +
  ## Remove legend ref: https://stackoverflow.com/a/35622358/6816220
  theme(legend.position = "none")

Other sources for help

• RStudio Community Forum: https://community.rstudio.com/
• Our rbtl Slack channel
• Documentation websites: https://dplyr.tidyverse.org/
• Twitter community: #rstats

Artwork by @allison_horst

Minimal reproducible example (reprex)

• Needed when asking questions online
• We will practice this in another class
• Good support information: https://www.tidyverse.org/help/#reprex

Artwork by @allison_horst

Break

10:00

Photo by Blake Wisz

Pair Programming

ae-10-data-science-lifecycle

1. Team up in pairs
2. Decide who writes code, and who supports without writing code
3. Open the file: ae-10b-lifecycle.qmd
4. Work through the exercises
5. Use your sticky notes to indicate if you need support

30:00

Homework Assignment

Teams on GitHub

Teams on GitHub

Teams on GitHub

Research Project Template

• Report template
• Complete list of items for grading (also as open Google Sheet)
• Information on intentions for publishing

Discuss and decide

• Data: Who will upload which data as part of the homework?
• Results & Discussion:
  • Who writes in file:
    • 03-01-results.qmd?
    • 03-02-results.qmd?
    • 03-03-results.qmd?

03:00

Submission

• All details in assignment week 10
• Due: Tuesday, 4th May at 23:59 (2 points)

Thanks! 🌻

Slides created via revealjs and Quarto: https://quarto.org/docs/presentations/revealjs/ Access slides as PDF on GitHub

All material is licensed under Creative Commons Attribution Share Alike 4.0 International.