Lineage (or Bundle Metadata)

Backward tracing bundle history

Bundle Metadata

Beyond human, processing, and UUId names, each bundle also tracks:

• Creation date

• Git repo, branch, and latest commit hash

• Task start and complete timestamps

• Bundle dependency set: the input bundles taken as input

It's that last set, that allows us to trace lineage.

Accessing lineage

Given the UUID of the output bundle, you can access lineage via the CLI.

dsdt lineage 564cfb28-14ac-412f-a09b-2a5bdb835f48
------ Lineage @ depth 0 -----
processing name: GetFeatures_2019_09_10____2016_12_01_8159b5bded
uuid: 564cfb28-14ac-412f-a09b-2a5bdb835f48
creation date: 2019-09-10 09:01:39.106816
code repo: https://github.intuit.com/data-science/iep-tt-pipeline
code hash: 2b07f8dd02bdea160481673fde6e5abb111a6ae5
code method:
git commit URL: https://github.intuit.com/data-science/iep-tt-pipeline/commit/2b07f8dd02bdea160481673fde6e5abb111a6ae5
code branch: HEAD
Start 1568131299.0983887 Stop 1568131299.100352 Duration 0.001963376998901367

	------ Lineage @ depth 1 -----
	processing name: GoogTrendTurbotaxLive_2019_09_10____2016_12_01_2fdd4e0539
	uuid: e57dd790-0d5b-434f-b4f8-65bbda2dfa7d
	creation date: 2019-09-10 09:01:15.001553
	code repo: https://github.intuit.com/data-science/iep-tt-pipeline
	code hash: 2b07f8dd02bdea160481673fde6e5abb111a6ae5
	code method:
	git commit URL: https://github.intuit.com/data-science/iep-tt-pipeline/commit/2b07f8dd02bdea160481673fde6e5abb111a6ae5
	code branch: HEAD
	Start 1568131273.2171564 Stop 1568131275.0006177 Duration 1.783461332321167

	------ Lineage @ depth 1 -----
	processing name: GoogTrendTurboTaxLive_2019_09_10____2016_12_01_3239f44071
	uuid: a359e040-f326-4df2-9c6e-7ab3769d36dd
	creation date: 2019-09-10 09:01:14.987869
	code repo: https://github.intuit.com/data-science/iep-tt-pipeline
	code hash: 2b07f8dd02bdea160481673fde6e5abb111a6ae5
	code method:
	git commit URL: https://github.intuit.com/data-science/iep-tt-pipeline/commit/2b07f8dd02bdea160481673fde6e5abb111a6ae5
	code branch: HEAD
	Start 1568131273.2121537 Stop 1568131274.986895 Duration 1.7747414112091064