S3 Plugin

S3-plugin is a high performance PyTorch dataset library to efficiently access datasets stored in S3 buckets. It provides streaming data access to datasets of any size and thus eliminates the need to provision local storage capacity. The library is designed to leverage the high throughput that S3 offers to access objects with minimal latency.

The users have the flexibility to use either map-style or iterable-style dataset interfaces based on their needs. The library itself is file-format agnostic and presents objects in S3 as a binary buffer(blob). Users are free to apply any additional transformation on the data received from S3.

Installation

You can install this package by following the below instructions.

Prerequisite

  • Python 3.6 (or Python 3.7) is required for this installation.

  • AWS CLI for configuring S3 access.

  • Pytorch >= 1.5 (If not available, S3-plugin installs latest Torch)

  • Note: To run on Mac, AWS_SDK_CPP must be installed.

Installing S3-Plugin via Wheel

<div class="highlight highlight-source-shell position-relative" data-snippet-clipboard-copy-content="# TODO Add final public wheels
aws s3 cp .
pip install
“>

# TODO Add final public wheels
aws s3 cp <S3 URI> .
pip install <whl name awsio-0.0.1-cp...whl>