Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 8 of 8

Data Quality & Validation

In Lesson 7 you compressed six of Maya's bakery columns into two clean PCA components. Every method in this course, the histogram in Lesson 1, the correlation in Lesson 2, the outlier rules in Lesson 3, the PCA in Lesson 7, quietly assumed one thing: that the numbers it was handed were real. This lesson questions that assumption before you ever plot a thing.

Maya has stopped typing her numbers in by hand. Her till now emails her a spreadsheet every Monday, bakery_week.csv, and she wants to run the same analysis on it automatically each week. The trouble is that an automatic export is a promise, not a fact: a stray dollar sign, a sensor that misfires, a day logged twice, and your tidy pipeline produces a confident, wrong answer.

Below is the export that just landed: ten rows for two weeks of trading. It looks fine at a glance. In fact it hides five different problems, one for each check you are about to learn, and any one of them would poison a mean, a correlation or a PCA. By the end of this lesson you will catch every one in seconds, every week, without looking.

By the end of this lesson you will be able to:

  • Run a repeatable five-point quality pass on a fresh export before trusting it
  • Spot and fix wrong column types (text that should be a number)
  • Catch impossible values with range and cross-field consistency checks, and tell an impossible value apart from a genuine outlier
  • Verify uniqueness and key integrity, finding duplicate rows and a broken primary key
  • Codify every check as a reusable rule that re-runs itself on next week's export

Prerequisites: you can run R and load a package with library(). It helps to have met the EDA framework (Lesson 1), correlation (Lesson 2), and especially outliers and the outlier-versus-error distinction (Lesson 3, the $905 festival day). Every new term is defined as it appears.

What we are checking

Data quality is "fitness for use"

Data quality is not about perfection, it is about fitness for use: is this table good enough to answer the question you are about to ask of it? A festival day of $1,250 is unusual but perfectly real and usable; a temperature reading of 203 degrees is not a hot day, it is a broken thermometer, and it will wreck any average it touches.

The good news is that almost every real defect falls into one of five buckets. Walk a fresh export through these five checks, in order, and you will catch the overwhelming majority of problems before they reach your analysis:

  • Types: is each column the kind of thing it should be (a number stored as a number, not as text)?
  • Ranges: is every value physically possible (no 203-degree day, no negative count)?
  • Completeness: are the values that should be there actually present, or are some missing?
  • Consistency: do related columns agree with each other (you cannot ring up more sales than people who walked in)?
  • Keys: is each row a distinct thing, one row per day, with no duplicates?
Key Insight
The whole point of this lesson is the last word in each bucket: a check. A check is just a yes/no question you can write as code. Once it is code, it costs nothing to run it again on next week's export, and the week after that. You do the thinking once; the computer does the watching forever.