dvc list not showing all files tracked by DVC?

[dwinski]

I am confused why when I run "dvc list --dvc-only" from the command line I don't see all the files with paths/hashes in the dvc.lock file that were originally listed in my project's dvc.yaml file. Instead, I only see the files listed as outputs of each stage of the pipeline.

To be clear, here is the command I am running from the root project folder where the dvc and git repos where initialized:

> dvc list --dvc-only .

[dberenbaum]

Hi, do you mean that you don't see dependencies? DVC does not cache those dependencies. It only computes their hash values for pipelines, so they are not considered tracked by DVC unless you also dvc add those dependencies.

[dwinski]

Ok, thanks. To be clear, does dvc only (by default) track the outputs listed in the dvc yaml file and not the dependencies? In the first stage of my pipelines I often have a data file as a dependency that could change so I've been wanting dvc to track (store) versions of it too. In general, I've been confused about the purpose of manually using "dvc add" when I am already using the dvc.yaml and dvc.lock files and keeping these files versioned with git. Is there any documentation showing best practices for different pipeline workflows?

[dberenbaum]

Correct. A typical workflow is to dvc add the raw data, then create a pipeline using that raw data as a dependency of the first pipeline stage.

I'm leaving some links here to docs that discuss how to setup pipelines, but please add suggestions for how you think we could clarify better.

https://dvc.org/doc/start/data-management/data-pipelines
https://dvc.org/doc/user-guide/pipelines/defining-pipelines

[dwinski]

Ok, thanks. That helps. I'll review the documentation, but off hand I'd say it might be nice to have a "data" subfield of the dependencies in the dvc.yaml field that DVC would pick up on as indicating data to automatically cache. Then, as I understand it, I wouldn't necessarily need to manually "dvc add" anything. If that functionality or something similar already exists please let me know.

[dwinski]

@dberenbaum

I find this part of the documentation from https://dvc.org/doc/start/data-management/data-pipelines a little misleading:

"DVC uses the pipeline definition to automatically track the data used and produced by any stage, so there's no need to manually run dvc add for data/prepared!"

That makes it sound like the raw data used by the initial stage of the pipeline would also be stored/cached but per our discussion that's not the case unless you explicitly use dvc add.

[dwinski]

@dberenbaum
A couple more questions for you on handling changes to the raw data in the dvc pipeline workflow:

If the raw data changes do I need to manually use dvc add to generate a new .dvc file for it and then also manually use a git commit to track that .dvc file for it? If so, this seems tedious compared to ease of just worrying about keeping the dvc.lock file commited regularly to track/cache many other outputs of pipeline (correct me if I am misunderstanding anything). I'm almost thinking maybe it would be worthwhile to use a workaround where add a dummy first stage of pipeline that puts the raw data under the "outs" field in for that stage to allow it to be automatically tracked/cached.

Also, are there ever live DVC office hours where you can talk with DVC developers and users?

[dberenbaum]

Yes, you have the right idea. How are you updating that raw data? Making a stage to update it is a pretty typical workflow.

Unfortunately, we don't have any office hours scheduled.