Get Started: Data Pipelines
Versioning large data files and directories for data science is great, but not enough. How is data filtered, transformed, or used to train ML models? DVC introduces a mechanism to capture data pipelines — series of data processes that produce a final result.
DVC pipelines and their data can also be easily versioned (using Git). This allows you to better organize projects, and reproduce your workflow and results later — exactly as they were built originally! For example, you could capture a simple ETL workflow, organize a data science project, or build a detailed machine learning pipeline.
Watch and learn, or follow along with the code example below!
Pipeline stages
Use dvc run to create stages. These represent processes (source code tracked
with Git) that form the steps of a pipeline. Stages also connect code to its
data input and output. Let's transform a Python script into a
stage:
$ dvc run -n prepare \
-p prepare.seed,prepare.split \
-d src/prepare.py -d data/data.xml \
-o data/prepared \
python src/prepare.py data/data.xmlA dvc.yaml file is generated. It includes information about the command we ran
(python src/prepare.py), its dependencies, and
outputs.
There's no need to use dvc add for DVC to track stage outputs (data/prepared
in this case); dvc run already took care of this. You only need to run
dvc push if you want to save them to
remote storage,
(usually along with git commit to version the stage file itself).
Dependency graphs (DAGs)
By using dvc run multiple times, and specifying outputs of a
stage as dependencies of another one, we can describe a sequence of
commands that gets to a desired result. This is what we call a data pipeline
or dependency graph.
Let's create a second stage chained to the outputs of prepare, to perform
feature extraction:
$ dvc run -n featurize \
-p featurize.max_features,featurize.ngrams \
-d src/featurization.py -d data/prepared \
-o data/features \
python src/featurization.py data/prepared data/featuresThe dvc.yaml file is updated automatically and should include two stages now.
This should be a good point to commit the changes with Git. These include
.gitignore, dvc.lock, and dvc.yaml — which describe our pipeline.
Reproduce
The whole point of creating this dvc.yaml pipeline file is an ability to
reproduce the pipeline:
$ dvc reproDVC pipelines (dvc.yaml file, dvc run, and dvc repro commands) solve a few
important problems:
- Automation - run a sequence of steps in a "smart" way that makes iterating on your project faster. DVC automatically determines which parts of a project need to be run, and it caches "runs" and their results, to avoid unnecessary re-runs.
- Reproducibility -
dvc.yamlanddvc.lockfiles describe what data to use and which commands will generate the pipeline results (such as an ML model). Storing these files in Git makes it easy to version and share. - Continuous Delivery and Continuous Integration (CI/CD) for ML - describing projects in way that it can be reproduced (built) is the first necessary step before introducing CI/CD systems. See our sister project, CML for some examples.
Visualize
Having built our pipeline, we need a good way to understand its structure. Seeing a graph of connected stage files would help. DVC lets you do just that, without leaving the terminal!
$ dvc dag
+---------+
| prepare |
+---------+
*
*
*
+-----------+
| featurize |
+-----------+
** **
** *
* **
+-------+ *
| train | **
+-------+ *
** **
** **
* *
+----------+
| evaluate |
+----------+Refer to
dvc dagto explore other ways this command can visualize a pipeline.