How to Process, Analyze and Visualize Data

Data science! Big data! Statistics! Infographics! Buzzword!

News

We have added links to the lectures for each day in the course outline.

Course Outline

This is a lab-oriented course where you will learn the basics to

• Take raw data (e.g., emails, logs)
• Extract meaningful information
• Use statistical tools
• Make visualizations

Each day will start with a 30 minute lecture followed by 2.5 hours of lab. Students will work through labs and exercises during class. We will post links to them prior to lecture.

Please make sure to complete Day 0 before the first class!!

• Day 0: Class organization and programming environment setup. Setup for the course before you arrive. Email us with questions!

• Day 1: An end-to-end example getting you from a dataset found online to several plots of campaign contributions. Lecture

• Day 2: Lots of visualization examples, and practice going from data to chart. Lecture

• Day 3: Statistics basics, including T-Tests, Linear Regression, and statistical significance. We'll use campaign finance and per-county health rankings. Lecture

• Day 4: Text processing on a large text corpus (the Enron email dataset) using tf-idf and cosine similarity. Lecture

• Day 5: Scaling up to process large datasets using Hadoop/MapReduce on a larger copy of the Enron dataset. Lecture

• Day 6: You tell us! Get into groups or work on your own to analyze a dataset of your choosing, and tell us a story!

Sponsors

We are grateful to our sponsors for speaking to students about their real-world experience and for covering our expenses.