Push/Pull using S3

Make sure you've set up your local machine to access your AWS account by setting up your credentials.

This section will teach you:

• Local and remote contexts

• Committing and pushing bundles to S3

• Pulling bundles from S3

• The difference between localized (w/ files) and non-localized (files remain on S3) bundles

Set up a remote for your local context:

• local context: A Disdat-managed directory on your dev box (typically at ~/.disdat/context) that holds bundles.

• remote context: A Disdat-managed directory on S3. A remote consists of the remote context as well as the s3://<bucket>/<optional_key> that your Disdat project uses to store all of its contexts.

1. Switch to our examples context: dsdt switch examples

2. Add the remote dsdt remote -f <remote context name> <s3 bucket/key where you want contexts to go> Note: The -f flag is used to force a context to be bound to a new remote.

3. List the contexts dsdt context

$dsdt remote -f examples s3://disdat-prod/
$dsdt context
*	examples	[examples@s3://disdat-prod/context]

Commit and push your bundle to S3:

$dsdt commit my.bundle
$dsdt push my.bundle
Pushed committed bundle my.bundle uuid c1f9085f-8bb5-4417-8b65-804a1ae7e451 to remote s3://disdat-prod/context

Yay! You're bundle is now safely on S3. Let's delete it and restore it!

Alternatively another user could bind their local context to this remote and pull the bundle as well.

$dsdt rm --all my.bundle
Disdat Context examples
On local branch examples
Removing old bundle HumanName[my.bundle] ProcName[BundleWrapperTask_my_bundle____e1908ea6e8] Timestamp[1578893445.5314891] Owner[kyocum] uuid[386f13cf-5b51-4237-b649-8549eff30004] lineage[[]] presentation[3]
Removing latest bundle HumanName[my.bundle] ProcName[BundleWrapperTask_my_bundle____e1908ea6e8] Timestamp[1578893802.796833] Owner[kyocum] uuid[c1f9085f-8bb5-4417-8b65-804a1ae7e451] lineage[[]] presentation[3]

And nothing remains in our local context:

$dsdt ls -v
NAME                	PROC_NAME           	OWNER   	DATE              	COMMITTED	UUID                                    	TAGS

Restore it by pulling all the remote bundles:

$dsdt pull
Fast Pull synchronizing with remote context examples@s3://cdo-disdat-prod/context
Fast pull fetching 2 objects...
Fast pull complete -- thread pool closed and joined.

Check to see our bundle made it back:

$dsdt ls -v
NAME                	PROC_NAME           	OWNER   	DATE              	COMMITTED	UUID                                    	TAGS
my.bundle           	BundleWrapperTask_my_bundle____e1908ea6e8	kyocum  	01-12-20 21:36:42 	True    	c1f9085f-8bb5-4417-8b65-804a1ae7e451

Almost all Disdat CLI commands allow a .* wild card. You could have said: dsdt ls -v my.bu.*

Our bundle is restored! Let's look at it:

$dsdt cat my.bundle
s3://cdo-disdat-prod/context/examples/objects/c1f9085f-8bb5-4417-8b65-804a1ae7e451/README.md

Oh no, the bundle contains an S3 path! When we made it, it contained a path to the file on local disk. This is because the bundle has not been localized. By default Disdat only moves meta-data when pushing/pulling to/from remotes. But if you want to look at the data directly, one must localize the bundle. That is, pull down copies of the files as well as the bundle's meta data.

Localize the bundle to get a local copy of the files it contains:

$dsdt pull -l my.bundle
Found remote bundle with human name my.bundle, uuid c1f9085f-8bb5-4417-8b65-804a1ae7e451 localizing ...
$dsdt cat my.bundle
/Users/kyocum/.disdat/context/examples/objects/c1f9085f-8bb5-4417-8b65-804a1ae7e451/README.md

Now we have created a local copy of the bundle on S3. That bundle on S3 is still there, by the way.