Lecture: ( by Vikash Das, youtube ) →

https://youtu.be/PPrPuxqWc7E?si=BICC-JeTCeYdMROJ

Notes:

First, let’s understand → WHY?

Why do we need Data Versioning?

An ML project runs as a pipeline of steps: Data Ingestion → Data Preprocessing → Feature Engineering → Feature Selection → Model Training → Model Evaluation.

• Each step is a component - usually its own Python file/module (a class with methods + attributes). You import them all into a Main.py that runs the whole flow.
• Each step produces an artifact - its output. E.g. Data Ingestion's artifact is the data pulled from the source (like S3), Preprocessing's artifact is the cleaned data that feeds Feature Engineering; and so on.

The pipeline is not static → things keep changing:

• The data source might change (a new file in S3) → the ingestion artifact changes.
• You run experiments → e.g. a different way to remove outliers in preprocessing, or a tweak in feature engineering → those outputs change too.
• Any such change ripples downstream: different features → different model → different evaluation metrics.

So the real question becomes: if a change makes things worse, how do you roll back to a previous version of the data? That's exactly what data versioning gives you. And it isn't only for tabular CSVs → you can version JSON, dictionaries, model outputs, etc., and even compare versions to make better choices.

Next →

What is DVC?

Just like Git versions your code, DVC (Data Version Control) versions your data.