Managing External Data

There are cases when data is so large, or its processing is organized in such a way, that its preferable to avoid moving it from its original location. For example data on a network attached storage (NAS), processing data on HDFS, running Dask via SSH, or for a script that streams data from S3 to process it.

External outputs and external dependencies provide ways to track data outside of the project.

How external outputs work

External outputs are considered part of the (extended) DVC project: DVC will track them for versioning, detecting when they change (reported by dvc status, for example).

To use existing files or directories in an external location as stage outputs, give their remote URLs or external paths to dvc add, or put them in dvc.yaml (deps field). Use the same format as the url of certain dvc remote types. Currently, the following protocols are supported:

• Amazon S3
• Google Cloud Storage
• SSH
• HDFS
• Local files and directories outside the workspace

External outputs require an external cache in the same external/remote file.

Note that remote storage is a different feature, and that external outputs are not pushed or pulled from/to DVC remotes.

⚠️ Avoid using the same DVC remote used for dvc push, dvc pull, etc. for external outputs, because it may cause data collisions: the hash of an external output could collide with that of a local file with different content.

Examples

Let's take a look at the following operations on all the supported location types:

1. Adding a dvc remote in the same location as the desired outputs, and configure it as external cache with dvc config.
2. Tracking existing data on the external location using dvc add (--external option needed). This produces a .dvc file with an external URL or path in its outs field.
3. Creating a simple stage with dvc run (--external option needed) that moves a local file to the external location. This produces an external output in dvc.yaml.

$ dvc remote add s3cache s3://mybucket/cache
$ dvc config cache.s3 s3cache

$ dvc add --external s3://mybucket/existing-data

$ dvc run -d data.txt \
          --external \
          -o s3://mybucket/data.txt \
          aws s3 cp data.txt s3://mybucket/data.txt

$ dvc remote add gscache gs://mybucket/cache
$ dvc config cache.gs gscache

$ dvc add --external gs://mybucket/existing-data

$ dvc run -d data.txt \
          --external \
          -o gs://mybucket/data.txt \
          gsutil cp data.txt gs://mybucket/data.txt

$ dvc remote add sshcache ssh://user@example.com/cache
$ dvc config cache.ssh sshcache

$ dvc add --external ssh://user@example.com/existing-data

$ dvc run -d data.txt \
          --external \
          -o ssh://user@example.com/data.txt \
          scp data.txt user@example.com:/data.txt

$ dvc remote add hdfscache hdfs://user@example.com/cache
$ dvc config cache.hdfs hdfscache

$ dvc add --external hdfs://user@example.com/existing-data

$ dvc run -d data.txt \
          --external \
          -o hdfs://user@example.com/data.txt \
          hdfs fs -copyFromLocal \
                  data.txt \
                  hdfs://user@example.com/data.txt

$ dvc remote add webhdfscache webhdfs://user@example.com/cache
$ dvc config cache.webhdfs webhdfscache

$ dvc add --external webhdfs://user@example.com/existing-data

$ dvc run -d data.txt \
          --external \
          -o webhdfs://user@example.com/data.txt \
          curl --upload-file data.txt \
              "http://user@example.com:50075/webhdfs/v1/data.txt?op=CREATE"

$ dvc add --external /home/shared/existing-data

$ dvc run -d data.txt \
          --external \
          -o /home/shared/data.txt \
          cp data.txt /home/shared/data.txt