Delta Sharing

Delta Sharing is an open protocol for secure real-time exchange of large datasets, which enables organizations to share data in real time regardless of which computing platforms they use. It is a simple REST protocol that securely shares access to part of a cloud dataset and leverages modern cloud storage systems, such as S3, ADLS, or GCS, to reliably transfer data.

With Delta Sharing, a user accessing shared data can directly connect to it through pandas, Tableau, Apache Spark, Rust, or other systems that support the open protocol, without having to deploy a specific compute platform first. Data providers can share a dataset once to reach a broad range of consumers, while consumers can begin using the data in minutes.

System Requirements

Python 3.6+

Installation

pip install delta-sharing

If you are using Databricks Runtime, you can follow Databricks Libraries doc to install the library on your clusters.

Accessing Shared Data

The connector accesses shared tables based on profile files, which are JSON files containing a user's credentials to access a Delta Sharing Server. We have several ways to get started:

Download the profile file to access an open, example Delta Sharing Server that we're hosting here. You can try the connectors with this sample data.
Start your own Delta Sharing Server and create your own profile file following profile file format to connect to this server.
Download a profile file from your data provider.

Quick Start

After you save the profile file, you can use it in the connector to access shared tables.

import delta_sharing

# Point to the profile file. It can be a file on the local file system or a file on a remote storage.
profile_file = "<profile-file-path>"

# Create a SharingClient.
client = delta_sharing.SharingClient(profile_file)

# List all shared tables.
client.list_all_tables()

# Create a url to access a shared table.
# A table path is the profile file path following with `#` and the fully qualified name of a table (`<share-name>.<schema-name>.<table-name>`).
table_url = profile_file + "#<share-name>.<schema-name>.<table-name>"

# Load a table as a Pandas DataFrame. This can be used to process tables that can fit in the memory.
delta_sharing.load_as_pandas(table_url)

# If the code is running with PySpark, you can use `load_as_spark` to load the table as a Spark DataFrame.
delta_sharing.load_as_spark(table_url)

You can try this by running our examples with the open, example Delta Sharing Server.

Details on Profile Paths

The profile file path for SharingClient and load_as_pandas can be any URL supported by FSSPEC (such as s3a://my_bucket/my/profile/file). If you are using Databricks File System, you can also preface the path with /dbfs/ to access the profile file as if it were a local file.
The profile file path for load_as_spark can be any URL supported by Hadoop FileSystem (such as s3a://my_bucket/my/profile/file).
A table path is the profile file path following with # and the fully qualified name of a table (<share-name>.<schema-name>.<table-name>).

Apache Spark Connector

The Apache Spark Connector implements the Delta Sharing Protocol to read shared tables from a Delta Sharing Server. It can be used in SQL, Python, Java, Scala and R.

System Requirements

Java 8+
Scala 2.12.x
Apache Spark 3+ or Databricks Runtime 7+

Accessing Shared Data

The connector loads user credentials from profile files. Please see Download the share profile file to download a profile file for our example server or for your own data sharing server.

Configuring Apache Spark

You can set up Apache Spark to load the Delta Sharing connector in the following two ways:

Run interactively: Start the Spark shell (Scala or Python) with the Delta Sharing connector and run the code snippets interactively in the shell.
Run as a project: Set up a Maven or SBT project (Scala or Java) with the Delta Sharing connector, copy the code snippets into a source file, and run the project.

If you are using Databricks Runtime, you can skip this section and follow Databricks Libraries doc to install the connector on your clusters.

Set up an interactive shell

To use Delta Sharing connector interactively within the Spark’s Scala/Python shell, you can launch the shells as follows.

PySpark shell

pyspark --packages io.delta:delta-sharing-spark_2.12:0.1.0

Scala Shell

bin/spark-shell --packages io.delta:delta-sharing-spark_2.12:0.1.0

Set up a standalone project

If you want to build a Java/Scala project using Delta Sharing connector from Maven Central Repository, you can use the following Maven coordinates.

Maven

You include Delta Sharing connector in your Maven project by adding it as a dependency in your POM file. Delta Sharing connector is compiled with Scala 2.12.

<dependency>
  <groupId>io.delta</groupId>
  <artifactId>delta-sharing-spark_2.12</artifactId>
  <version>0.1.0</version>
</dependency>

SBT

You include Delta Sharing connector in your SBT project by adding the following line to your build.sbt file:

libraryDependencies += "io.delta" %% "delta-sharing-spark" % "0.1.0"

Quick Start

After you save the profile file and launch Spark with the connector library, you can access shared tables using any language.

SQL

-- A table path is the profile file path following with `#` and the fully qualified name of a table (`<share-name>.<schema-name>.<table-name>`).
CREATE TABLE mytable USING deltaSharing LOCATION '<profile-file-path>#<share-name>.<schema-name>.<table-name>';
SELECT * FROM mytable;

Python

# A table path is the profile file path following with `#` and the fully qualified name of a table (`<share-name>.<schema-name>.<table-name>`).
table_path = "<profile-file-path>#<share-name>.<schema-name>.<table-name>"
df = spark.read.format("deltaSharing").load(table_path)

Scala

// A table path is the profile file path following with `#` and the fully qualified name of a table (`<share-name>.<schema-name>.<table-name>`).
val tablePath = "<profile-file-path>#<share-name>.<schema-name>.<table-name>"
val df = spark.read.format("deltaSharing").load(tablePath)

Java

// A table path is the profile file path following with `#` and the fully qualified name of a table (`<share-name>.<schema-name>.<table-name>`).
String tablePath = "<profile-file-path>#<share-name>.<schema-name>.<table-name>";
Dataset<Row> df = spark.read.format("deltaSharing").load(tablePath);

R

# A table path is the profile file path following with `#` and the fully qualified name of a table (`<share-name>.<schema-name>.<table-name>`).
table_path <- "<profile-file-path>#<share-name>.<schema-name>.<table-name>"
df <- read.df(table_path, "deltaSharing")

You can try this by running our examples with the open, example Delta Sharing Server.

Table paths

A profile file path can be any URL supported by Hadoop FileSystem (such as s3a://my_bucket/my/profile/file).
A table path is the profile file path following with # and the fully qualified name of a table (<share-name>.<schema-name>.<table-name>).

Delta Sharing Reference Server

The Delta Sharing Reference Server is a reference implementation server for the Delta Sharing Protocol. This can be used to set up a small service to test your own connector that implements the Delta Sharing Protocol. Please note that this is not a completed implementation of secure web server. We highly recommend you to put this behind a secure proxy if you would like to expose it to public.

Here are the steps to setup a server to share your own data.

Get the pre-built package

Download the pre-built package delta-sharing-server-x.y.z.zip from GitHub Releases.

Config the shared tables for the server

Unpack the pre-built package and copy the server config template file conf/delta-sharing-server.yaml.template to create your own server yaml file, such as conf/delta-sharing-server.yaml.
Make changes to your yaml file, such as add the Delta Lake tables you would like to share in this server. You may also need to update some server configs for special requirements.

Config the server to access tables on S3

We only support sharing Delta Lake tables on S3 right now. There are multiple ways to config the server to access S3.

EC2 IAM Metadata Authentication (Recommended)

Applications running in EC2 may associate an IAM role with the VM and query the EC2 Instance Metadata Service for credentials to access S3.

Authenticating via the AWS Environment Variables

We support configuration via the standard AWS environment variables. The core environment variables are for the access key and associated secret:

export AWS_ACCESS_KEY_ID=my.aws.key
export AWS_SECRET_ACCESS_KEY=my.secret.key

Other S3 authentication methods

The server is using hadooop-aws to read S3. You can find other approaches in hadoop-aws doc.

More cloud storage supports will be added in the future.

Authorization

The server supports a basic authorization with pre-configed bearer token. You can add the following config to your server yaml file:

authorization:
  bearerToken: <token>

Then any request should send with the above token, otherwise, the server will refuse the request.

If you don't config the bearer token in the server yaml file, all requests will be accepted without authorization.

To be more secure, you recommend you to put the server behind a secure proxy such as NGINX to set up JWT Authentication.

Start the server

Run the following shell command:

bin/delta-sharing-server -- --config <the-server-config-yaml-file>

<the-server-config-yaml-file> should be the path of the yaml file you created in the previous step. You can find options to config JVM in sbt-native-packager.

API Compatibility

The REST APIs provided by Delta Sharing Server are stable public APIs. They are defined by Delta Sharing Protocol and we will follow the entire protocol strictly.

The interfaces inside Delta Sharing Server are not public APIs. They are considered internal, and they are subject to change across minor/patch releases.

Delta Sharing Protocol

The Delta Sharing Protocol specification details the protocol.

Building this Project

Python Connector

To execute tests, run

python/dev/pytest

To install in develop mode, run

cd python/
pip install -e .

To install locally, run

cd python/
pip install .

To generate a wheel file, run

cd python/
python setup.py sdist bdist_wheel

It will generate python/dist/delta_sharing-x.y.z-py3-none-any.whl.