Using ODS-Briefcase to mediate access to data stored on Amazon Simple Storage Service (Amazon S3)

Introduction

Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

S3 file storage offers users a number of features:

• Write, read, and delete objects containing from 1 byte to 5 terabytes of data each. The number of objects you can store is unlimited.
• Each object is stored in a bucket and retrieved via a unique, developer-assigned key.
• Authentication mechanisms are provided to ensure that data is kept secure from unauthorized access. Objects can be made private or public, and rights can be granted to specific users.
• Options for secure data upload/download and encryption of data at rest are provided for additional data protection.
• Uses standards-based REST and SOAP interfaces designed to work with any Internet-development toolkit.

ODS Briefcase offers a new "Amazon S3" folder type which provides several additional features, once the ODS system is configured with a registered Amazon API key and secret pair:

• Specify credentials for authentication against S3, enabling retrieval of any or all of your folders and files, including any associated metadata
• Specify which Sponger Extractor Cartridges and Sponger Meta Cartridges should be run against your S3 data
• Save fetched data, metadata, and Sponger output, into a newly created S3-specific RDF Graph in the Virtuoso RDF Quad Store
• Specify other ODS Users or WebID-based ACLs which should be granted access to your S3 data through ODS Briefcase

Example

Here is a sample scenario showing the whole process, from getting the Amazon S3 Access Key ID and Secret Key, to retrieving all S3 content to ODS-Briefcase.

Prerequisites

• An Amazon Web Services (AWS) account, which has been signed up for Amazon S3
• Files and folders (data) stored in one of your S3 Buckets

Steps

1 - Get an Amazon S3 Access Key ID and Secret Key for use with ODS-Briefcase

1. Browse to the AWS "Security Credentials" page, and log in with your existing AWS user credentials:
   
   
   
   
2. Click Access Credentials:
   
   
   
   
3. Click Create a new Access Key to get a new Key & Secret pair for use with ODS-Briefcase:
   
   
   
   
4. Click Yes when prompted:
   
   
   
   
5. Your new Access Key ID will now be displayed. Click Show to see the corresponding Secret Access Key.
   
   
   
   
6. You will need both of these for the steps that follow.
   
   
   
   

2 - Access files stored on Amazon S3 through ODS-Briefcase

1. Browse to your Virtuoso + ODS instance, http://<cname>[:<port>]/ods, e.g., <http://example.com/ods/>.
   
   
   
   
2. Log in as a Virtuoso+ODS user with normal privileges, e.g., demo.
   
   
   
   
3. Click to the ODS-Briefcase application:
   
   
   
   
4. Click New Folder from the main Briefcase control strip:
   
   
   
   
5. In the New Folder form, specify a Folder Name, e.g., MyAmazonS3Folder, and set the Folder type to Amazon S3:
   
   
   
   
6. Click to the newly visible S3 Properties tab:
   
   
   
   
7. Set Bucket Name to a specific S3 Bucket Name to restrict ODS-Briefcase access to only this Bucket, or you can leave it empty to give ODS-Briefcase access to all your Buckets. Also enter the Access Key ID and Secret Key values you obtained earlier:
   
   
   
   
8. All RDF data extracted from and otherwise collected about your S3 files will be saved to a named graph, where the default name is constructed from the host ODS instance, the ODS username, and the Briefcase folder name -- http://<host>[:<port>]/DAV/home/<user-name>/<folder-name>#this -- so, in this example, the default graph would be named <http://example.com/ods/demo/MyAmazonS3Folder#this>. Expert users may wish to change this graph name, but the default is suitable for most users.
   
   
   
   
9. You can also turn on the Virtuoso Sponger, to process your S3 files more comprehensively. When the Sponger is enabled for your folder, you can specify which "Sponger Extractor Cartridges" and "Sponger Meta Cartridges" will be run over the files in S3. You can enable all cartridges, but this will introduce some processing overhead. It's generally best to enable only those cartridges which will be applicable to your files and their content. For instance, you may know that you'll only be storing Microsoft and Open Office documents here. Click Create to finish the folder creation process.
   
   
   
   
10. Your new S3-linked folder should now be displayed in your Briefcase resources list:
    
    
    
    
11. Click the folder name to see the resources ODS-Briefcase has found in your Amazon S3 Dataspace:
    
    
    
    

References

• Virtuoso WebDAV S3 support
• Amazon S3 Services
• Amazon Cloud Front Getting Started Guide
• How Linked Data & WebID Deliver Controlled Access to Resources in Google Drive, SkyDrive, DropBox, Box.Net, and Amazon S3
• Using ODS-Briefcase to mediate access to data stored on Box (a/k/a box.com or box.net)
• Using ODS-Briefcase to mediate access to data stored on DropBox
• Using ODS-Briefcase to mediate access to data stored on Google Drive
• Using ODS-Briefcase to mediate access to data stored on SkyDrive
• Manage different Briefcase type folders in Briefcase
• ODS-Briefcase
• Virtuoso Sponger
• OpenLink-supplied Virtuoso Sponger Cartridges
• Supported Meta Cartridges in Virtuoso
• Meta Cartridges Usage via REST Request
• Parameterized Examples of Meta Cartridge Usage via REST Request

CategoryVirtuoso CategoryDAV CategoryTutorial CategoryDocumentation CategoryBriefcase CategoryODS CategoryOdsBriefcase