MSI Second Tier ("tier 2") Storage & S3 Bucket Interface🔗

Second tier storage is used for storing data and files that you don't need readily accessible on tier 1. Prior to summer 2024, each user was granted up to 120 TB of storage on tier 2. MSI has since updated their policy so that new users are limited to 5 GB, with their documentation stating that MSI groups are now limited to 120 TB. As such, whenever possible, please have a PI create a s3 bucket where a large amount of data is expected to be stored.

For a comprehensive introduction to s3, see this s3 tutorial video (recorded as part of the Data Processing Workshop hosted for undergraduate research assistants). It is encouraged to watch this tutorial prior to your first time using s3 commands. If you don't have access to the CDNI google drive to view the video, ask Kim or Luci to be added.

S3 Basics and Usage🔗

It's important to know that S3 is not actually a filesystem, but an object storage system, and S3cmd is an API tool for object storage. As a result, the logic you are likely more familiar with when interacting with folder structures on the command line doesn't necessarily translate to s3cmd. S3 buckets don't actually have real directory structures, so when referencing a specific file or folder on S3, you provide what may look like a file path, but is actually a key prefix used to iterate through objects with matching substrings. This means you cannot cd into a s3 bucket, but instead have to continuously ls a bucket to see what is inside each subfolder.

For example, to list the contents of s3://luci-scratch/ABCC, it is important to include a trailing / (i.e. s3://luci-scratch/ABCC/). Without a trailing /, s3cmd will iterate through all objects containing luci-scratch/ABCC*, which does not include the contents of the folder itself:

$ s3cmd ls s3://luci-scratch/ABCC
                    DIR  s3://luci-scratch/ABCC/

When the trailing / is present, s3cmd iteratates through the bucket objects to list objects containing luci-scratch/ABCC/* and returns the following:

$ s3cmd ls s3://luci-scratch/ABCC/
                    DIR  s3://luci-scratch/ABCC/code/
                    DIR  s3://luci-scratch/ABCC/data/
2024-09-09 21:25 DIROBJ  s3://luci-scratch/ABCC/
2024-09-09 21:42   183M  s3://luci-scratch/ABCC/subjectlist.csv

Note that this logic also means that wildcard (*) doesn't work for most s3 commands

S3cmd Usage🔗

Read s3cmd usage for full usage

s3info -u x500 - view your s3 credentials and tier2 disk usage
s3cmd du -H s3://bucket-name/ - check the size of a single bucket (when using commands like du or ls on large buckets, we recommend using the s5cmd client)
s3cmd ls - list all buckets you are owner on (buckets that you do not own, but have been given access to, will not appear in this list)
s3cmd rm s3://bucket-name/file - remove a file from a bucket
s3cmd rb s3://bucket-name - remove an entire bucket
s3cmd del --recursive s3://bucket-name/subfolder - remove a subfolder of a bucket

Wildcard logic doesn't generally work for s3cmd commands but it does work for the ls and del commands.

Making a new bucket🔗

s3cmd mb s3://bucket-name/ - create a new bucket You're welcome to create your own buckets for personal use, but if you were added to MSI after summer 2024, you only have 5GB of tier 2 space.

Attention

Buckets intended to store large amounts of data or data repositories should be created by a PI.

This is for two reasons: PIs and legacy users have a larger storage quota on tier2 (120TB) and if you leave UMN, others will lose access to your bucket once your UMN account is deactivated!!

Also note that, while this has not always been the case, MSI currently requires users to follow AmazonS3 naming standards, e.g. names cannot include underscores or uppercase letters - see the AmazonS3 bucket naming rules for full details.

Syncing data to and from s3 buckets🔗

s3cmd sync /path/to/file s3://bucket-name/ - sync a file to a bucket
s3cmd sync s3://bucket-name/file /path/to/folder/output/ - sync a file from a bucket
s3cmd sync s3://bucket-name/dir_name /destination/directory/path/ --recursive - sync a folder from a bucket
s3cmd sync dir_name s3://bucket-name/ --recursive - sync a folder to a bucket

  • If you want to clear the prior contents of the directory that you are syncing to, include the --delete-removed flag to remove files that are present in the destination path but aren't present in the source path. USE WITH CAUTION
  • If your job runs out of time before the data sync is complete, use s3cmd sync --recursive --no-check-md5 to restart the sync process where it left off (even if the sync says complete)

Or use get/put:
s3cmd get s3://path/to/file /path/to/where/you/want/file/ - pull down single file
s3cmd put path/to/file s3://path/to/where/you/want/file/ - place a single file into bucket

Using s5cmd🔗

When using commands like du or ls on large buckets, we recommend using s5cmd client, an alternative S3 client that performs much faster on MSI than s3cmd for certain operations (roughly 10x-100x faster for du and ls). Currently, s5cmd cannot resume interrupted file transfers (upload or download), so it is advisable not to use it over s3cmd for file transfers.

The usage is similar to s3cmd except that it requires the --endpoint-url argumement to specify MSI's S3 (Tier 2) instead of the default (AWS S3). For example: s5cmd --endpoint-url https://s3.msi.umn.edu du -H s3://MY_BUCKET/*

Setup Required to use s5cmd
1. Add to your .bashrc (1) the path on MSI where it's installed, (2) the code needed for path completion (which can also be viewed with s5cmd --install-completion), (3) export your S3 credentials:

# path to s5cmd installed on MSI
export PATH=$PATH:/projects/standard/faird/shared/code/external/utilities/s5cmd
# prepare autocompletion suggestions for s5cmd and save them to COMPREPLY array
_s5cmd_cli_bash_autocomplete() {
    if [[ "${COMP_WORDS[0]}" != "source" ]]; then
        COMPREPLY=()
        local opts cur cmd
        cur="${COMP_WORDS[COMP_CWORD]}"
        cmd="${COMP_LINE:0:$COMP_POINT}"
        [ "${COMP_LINE:COMP_POINT-1:$COMP_POINT}" == " " ] \
            && cmd="${cmd} ''" 
        opts=$($cmd --generate-bash-completion)
        while IFS='' read -r line;
        do
            COMPREPLY+=("$line");
        done \
        < <(compgen -o bashdefault -o default -o nospace -W "${opts}" -- "${cur}")
        return 0
    fi
}
# call the _s5cmd_cli_bash_autocomplete to complete s5cmd command. 
complete -o nospace -F _s5cmd_cli_bash_autocomplete s5cmd

# export S3 credentials from s3info
export AWS_SECRET_ACCESS_KEY=$(s3info | grep Secret | awk '{ print $NF }' )
export AWS_ACCESS_KEY_ID=$(s3info | grep Access | awk '{ print $NF }' )

Updating S3 Policy & Bucket Access🔗

To gain access to a bucket, ask the bucket owner or others with access granted. If you're unsure of who already has access, consult the tier2 tab of CDNI's data tracking spreadsheet. Note that none of the scripts or commands below will work if you don't have bucket access.

We generally use fairly permissive s3 policies and so typically anyone with access to a bucket (not the bucket owner alone) can grant access to a new user. MSI provides instructions for how to grant someone access to a bucket using an s3 policy file, but the simplest method is to use the set_s3policy command with the following steps:

  1. If you haven't already done so, add the line export PATH="/projects/standard/faird/shared/code/internal/utilities/s3policy_bin/:$PATH" to your ~/.bashrcand open a new terminal after saving your changes in order for your changes to take effect.
  2. Print list of current users with bucket access with the command
    /projects/standard/faird/shared/code/internal/utilities/MSI-utilities/s3_get_x500/get_x500.sh s3://bucket_name/
  3. Copy the FULL LIST (see NOTE below), append the x500 for the new user you wish to grant access (x500_3 in the example below), and feed into the set_s3policy command like so, excluding the s3:// prefix for the bucket name in the command:
    set_s3policy x500_1,x500_2,x500_3 bucket_name FULL
  4. Finally, update the list of users with access to the bucket in tier2 tab of CDNI's tracking spreadsheet

NOTE: You must copy the full list from get_x500.sh because every time you set a policy you need to list EVERYONE that needs access. You CANNOT just list that one person's x500: this will result in only that person having bucket access and access removal for all other x500s, including the bucket owner. get_x500.sh will also list your x500 since you need to have access to the bucket in order to run this command.

If you do not have access to the faird share, you can create your own get_x500.sh script with the following code: 

# Give full path to bucket (ex: s3://bucket-name/)
bucket=$1

x500=$( for uid in $(s3cmd info ${bucket} | grep -oE '([0-9]{5})'); do s3info -u ${uid} | grep 'Sharing' | cut -d \@ -f 1 | cut -d \: -f 2; done )

all_x500s=$(echo $x500 | sed 's| |,|g')

echo $all_x500s

Advanced S3 Policy usage🔗

While set_s3policy should be sufficient for most uses, for more fine-grained management of bucket permissions you can manually create a bucket policy JSON file and apply it with s3cmd setpolicy FILE s3://BUCKET

See the AWS S3 documentation for examples of bucket policy files.

S3 Configuration🔗

Certain S3cmd or S3api commands may require that you enter your S3 credentials (including a Access Key ID and Secret Access Key, which can be found with the command s3info). To configure your S3 credentials so that you don't have to manually enter them:

  • Create a file named .s3cfg-RANDOM in your home directory (can't just be .s3cfg, name it something identifiable)

  • Run s3cmd -c ~/.s3cfg-RANDOM --configure and provide your Access Key ID and Secret Access Key when prompted

You shouldn't have to make any changes to the file unless the s3 bucket is hosted by something different than s3.amazonaws.com. You can test your access through an ls command - the s3cmd -c ~/.s3cfg-RANDOM will have to be part of every command you run to access the buckets that require the credentials you provided.

S3 access with Cyberduck🔗

View your MSI S3 credentials (login required)

Cyberduck S3 documentation

Cyberduck is a cloud storage browser for Windows and macOS devices, which can be used to access s3 storage from your local machine. To access MSI Tier 2 storage in Cyberduck, connect to s3.msi.umn.edu using your s3 credentials, which can be retrieved from the link at the top of this subsection or by running s3info in your MSI terminal.

To log in for the first time, follow the following steps:

  • Click "Open Connection" in the top left corner of the screen

  • Select "Amazon s3" from the drop down menu

  • Type in s3.msi.umn.edu for the server

  • Enter you access and secret keys

  • Click "Connect"