This open source project comes from our customer community. It was developed by Peak Games to assist with their own S3 workflow, and includes features such as tab completion and built-in wild card support for files in S3 commands. Enjoy!

Background

Up until now, working on multiple objects on

The Tool

Since s5cmd already has a worker pool, wildcard operations can be accomplished using a single worker for the ListObjects call (which can match further wildcards), then let other workers do the actual processing. It also supports shell autocompletion for bash and zsh, so if you’d like to use it as a more conventional CLI tool, you can just hit TAB and let it autocomplete options, buckets, and paths/objects for you.

Installation

Install s5cmd on Mac OS X:

$ brew tap peakgames/s5cmd https://github.com/peakgames/s5cmd
$ brew install s5cmd

The tool is written in Go, other platforms can compile and install it using:

$ go get -u github.com/peakgames/s5cmd

Set up credentials just as you would for the awscli tool: Use the ~/.aws/credentials file or environment variables, or a combination of both. (If you’re running on EC2, roles are also supported.)

Usage

Commands are in “command [command options] argument1 [argument2]” format. s5cmd also takes options, which affect all commands run. To get the list of s5cmd options:

$ s5cmd -help

To get a list of available commands, run without arguments:

$ s5cmd

s5cmd in Action

Say we have a bucket named “reports-bkt”, and we have some files inside. First, let’s download one:

$ s5cmd get s3://reports-bkt/a/2018/03/14/reports_19_13716285583145.csv.gz
                     # Downloading reports_19_13716285583145.csv.gz...
2018/03/21 11:46:05 +OK "get s3://reports-bkt/a/2018/03/14/reports_19_13716285583145.csv.gz ./reports_19_13716285583145.csv.gz"

Now, let’s scan all of last month’s CSV reports and match each day’s report with a wildcard:

$ s5cmd du -g -h s3://reports-bkt/a/2018/02/*/reports*csv.gz
                            + 10.7M bytes in 367 objects: s3://reports-bkt/a/2018/02/*/reports*csv.gz [STANDARD]
2018/03/21 11:46:24 +OK "du s3://reports-bkt/a/2018/02/*/reports*csv.gz" (1)

Looks like there are 367 reports for the whole month, taking up about 10MB, all using standard storage. Let’s download them all:

$ s5cmd cp --parents s3://reports-bkt/a/2018/02/*/reports*csv.gz .

Using the --parents option, each day of the month will be downloaded to its own directory. (This option creates directory structure starting from the first wildcard specified.)

These examples are just the tip of the iceberg. Piping commands using a file or another command’s output is another option.

You might have noticed that our bucket has the usual “letter prefix” scheme. Let’s say you want to download all files for a given date, for all prefixes. This structure is something like:

- a/[yyyy]/[mm]/[dd]/object_unique_id.gz
- b/[yyyy]/[mm]/[dd]/object_unique_id.gz
- c/[yyyy]/[mm]/[dd]/object_unique_id.gz
... up to ...
- z/[yyyy]/[mm]/[dd]/object_unique_id.gz

If you have hundreds of days and billions of objects, specifying the wildcard at the first level won’t really work. Since you already know the range (letters a to z), you can generate commands for each of the prefixes. Invoke the tool just once, and let it do the work. Try this:

$ for X in {a..z}; do echo get -n s3://reports-bkt/${X}/2018/03/14/reports*csv.gz; done | s5cmd -f -
2018/03/21 11:48:03 # Stats: Total             379 281 ops/sec 1.350311978s

The first command will generate a bunch of “get” commands, then pass the commands to s5cmd to do the work. Notice that we’ve used the “-n” (no-clobber) option to prevent overwriting if the object names are not really unique – we can’t use the --parents option because the wildcard is not in the directory name. You can see how many operations were done (and how much time it took) by checking the stat counters.

Contributing

All contributions to the project are welcome, and managed using the issue tracker at github.com/peakgames/s5cmd. If you are going to submit a PR, we suggest you open an issue first to discuss it with the team.

This is a guest post from Peak Games, which leverages S3 as part of a comprehensive pipeline that distills data into knowledge, further enhancing user experience of their world-class mobile games.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.