Tcl Library Source Code

Documentation
Login
Bounty program for improvements to Tcl and certain Tcl packages.
Tcl 2019 Conference, Houston/TX, US, Nov 4-8
Send your abstracts to [email protected]
or submit via the online form by Sep 9.


[ Main Table Of Contents | Table Of Contents | Keyword Index | Categories | Modules | Applications ]

NAME

S3 - Amazon S3 Web Service Interface

Table Of Contents

SYNOPSIS

package require Tcl 8.5
package require S3 ?1.0.3?
package require sha1 1.0
package require md5 2.0
package require base64 2.3
package require xsxp 1.0

S3::Configure ?-reset boolean? ?-retries integer? ?-accesskeyid idstring? ?-secretaccesskey idstring? ?-service-access-point FQDN? ?-use-tls boolean? ?-default-compare always|never|exists|missing|newer|date|checksum|different? ?-default-separator string? ?-default-acl private|public-read|public-read-write|authenticated-read|keep|calc? ?-default-bucket bucketname?
S3::SuggestBucket ?name?
S3::REST dict
S3::ListAllMyBuckets ?-blocking boolean? ?-parse-xml xmlstring? ?-result-type REST|xml|pxml|dict|names|owner?
S3::PutBucket ?-bucket bucketname? ?-blocking boolean? ?-acl {}|private|public-read|public-read-write|authenticated-read?
S3::DeleteBucket ?-bucket bucketname? ?-blocking boolean?
S3::GetBucket ?-bucket bucketname? ?-blocking boolean? ?-parse-xml xmlstring? ?-max-count integer? ?-prefix prefixstring? ?-delimiter delimiterstring? ?-result-type REST|xml|pxml|names|dict?
S3::Put ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-file filename? ?-content contentstring? ?-acl private|public-read|public-read-write|authenticated-read|calc|keep? ?-content-type contenttypestring? ?-x-amz-meta-* metadatatext? ?-compare comparemode?
S3::Get ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-compare comparemode? ?-file filename? ?-content contentvarname? ?-timestamp aws|now? ?-headers headervarname?
S3::Head ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-dict dictvarname? ?-headers headersvarname? ?-status statusvarname?
S3::GetAcl ?-blocking boolean? ?-bucket bucketname? -resource resourcename ?-result-type REST|xml|pxml?
S3::PutAcl ?-blocking boolean? ?-bucket bucketname? -resource resourcename ?-acl new-acl?
S3::Delete ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-status statusvar?
S3::Push ?-bucket bucketname? -directory directoryname ?-prefix prefixstring? ?-compare comparemode? ?-x-amz-meta-* metastring? ?-acl aclcode? ?-delete boolean? ?-error throw|break|continue? ?-progress scriptprefix?
S3::Pull ?-bucket bucketname? -directory directoryname ?-prefix prefixstring? ?-blocking boolean? ?-compare comparemode? ?-delete boolean? ?-timestamp aws|now? ?-error throw|break|continue? ?-progress scriptprefix?
S3::Toss ?-bucket bucketname? -prefix prefixstring ?-blocking boolean? ?-error throw|break|continue? ?-progress scriptprefix?

DESCRIPTION

This package provides access to Amazon's Simple Storage Solution web service.

As a quick summary, Amazon Simple Storage Solution provides a for-fee web service allowing the storage of arbitrary data as "resources" within "buckets" online. See http://www.amazonaws.com/ for details on that system. Access to the service is via HTTP (SOAP or REST). Much of this documentation will not make sense if you're not familiar with the terms and functionality of the Amazon S3 service.

This package provides services for reading and writing the data items via the REST interface. It also provides some higher-level operations. Other packages in the same distribution provide for even more functionality.

Copyright 2006 Darren New. All Rights Reserved. NO WARRANTIES OF ANY TYPE ARE PROVIDED. COPYING OR USE INDEMNIFIES THE AUTHOR IN ALL WAYS. This software is licensed under essentially the same terms as Tcl. See LICENSE.txt for the terms.

ERROR REPORTING

The error reporting from this package makes use of $errorCode to provide more details on what happened than simply throwing an error. Any error caught by the S3 package (and we try to catch them all) will return with an $errorCode being a list having at least three elements. In all cases, the first element will be "S3". The second element will take on one of six values, with that element defining the value of the third and subsequent elements. S3::REST does not throw an error, but rather returns a dictionary with the keys "error", "errorInfo", and "errorCode" set. This allows for reliable background use. The possible second elements are these:

COMMANDS

This package provides several separate levels of complexity.

The goal is to have at least the bottom-most layers implemented in pure Tcl using only that which comes from widely-available sources, such as Tcllib.

LOW LEVEL COMMANDS

These commands do not require any packages not listed above. They talk directly to the service, or they are utility or configuration routines. Note that the "xsxp" package was written to support this package, so it should be available wherever you got this package.

HIGH LEVEL COMMANDS

The routines in this section all make use of one or more calls to S3::REST to do their work, then parse and manage the data in a convenient way. All these commands throw errors as described in ERROR REPORTING unless otherwise noted.

In all these commands, all arguments are presented as name/value pairs, in any order. All the argument names start with a hyphen.

There are a few options that are common to many of the commands, and those common options are documented here.

LIMITATIONS

USAGE SUGGESTIONS

To fetch a "directory" out of a bucket, make changes, and store it back:

file mkdir ./tempfiles
S3::Pull -bucket sample -prefix of/interest -directory ./tempfiles \
  -timestamp aws
do_my_process ./tempfiles other arguments
S3::Push -bucket sample -prefix of/interest -directory ./tempfiles \
  -compare newer -delete true

To delete files locally that were deleted off of S3 but not otherwise update files:

S3::Pull -bucket sample -prefix of/interest -directory ./myfiles \
  -compare never -delete true

FUTURE DEVELOPMENTS

The author intends to work on several additional projects related to this package, in addition to finishing the unfinished features.

First, a command-line program allowing browsing of buckets and transfer of files from shell scripts and command prompts is useful.

Second, a GUI-based program allowing visual manipulation of bucket and resource trees not unlike Windows Explorer would be useful.

Third, a command-line (and perhaps a GUI-based) program called "OddJob" that will use S3 to synchronize computation amongst multiple servers running OddJob. An S3 bucket will be set up with a number of scripts to run, and the OddJob program can be invoked on multiple machines to run scripts on all the machines, each moving on to the next unstarted task as it finishes each. This is still being designed, and it is intended primarily to be run on Amazon's Elastic Compute Cloud.

TLS Security Considerations

This package uses the TLS package to handle the security for https urls and other socket connections.

Policy decisions like the set of protocols to support and what ciphers to use are not the responsibility of TLS, nor of this package itself however. Such decisions are the responsibility of whichever application is using the package, and are likely influenced by the set of servers the application will talk to as well.

For example, in light of the recent POODLE attack discovered by Google many servers will disable support for the SSLv3 protocol. To handle this change the applications using TLS must be patched, and not this package, nor TLS itself. Such a patch may be as simple as generally activating tls1 support, as shown in the example below.

package require tls
tls::init -tls1 1 ;# forcibly activate support for the TLS1 protocol

... your own application code ...

Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category amazon-s3 of the Tcllib Trackers. Please also report any ideas for enhancements you may have for either package and/or documentation.

When proposing code changes, please provide unified diffs, i.e the output of diff -u.

Note further that attachments are strongly preferred over inlined patches. Attachments can be made by going to the Edit form of the ticket immediately after its creation, and then using the left-most button in the secondary navigation bar.

KEYWORDS

amazon, cloud, s3

CATEGORY

Networking

COPYRIGHT

2006,2008 Darren New. All Rights Reserved. See LICENSE.TXT for terms.