terrareg

Open source Terraform Registry.

Provides features to aid usage and discovery of modules:

  • Provide analytics about usage of modules;
  • Interactive ‘Usage builder’, helping users build terraform to use the terraform modules;
  • Hooks for git SCM applications to automatically index modules;

Getting started

Install depdencies:

pip install -r requirements.txt

Initialise database and start server:

alembic upgrade head
python ./terrareg.py

Upload a terraform module:

terrareg_root=$PWD

cd source/of/my/module

# OPTIONAL: Create an terrareg meta-data file
echo '{ "description": "My first module", "owner": "ME!", "source": "https://github.com/me/my-tf-module" }' > ./terrareg.json

# Zip up module
zip -r * ../my-tf-module.zip

# Upload to terrareg
bash $terrareg_root/scripts/upload_module.sh http://localhost:5000 me my-tf-module aws 1.0.0 source/of/my/my-tf-module.zip

Navigate to http://localhost:5000 to get started, or http://localhost/modules/me/my-tf-module to see the uploaded example!

Running with docker

docker build . -t terrareg:latest
docker run -p 5000:5000 -e MIGRATE_DATABASE=True terrareg:latest

Docker environment variables

The following environment variables are available to configure the docker container

MIGRATE_DATABASE

Whether to perform a database migration on container startup.

Set to True to enable database migration

Default: False

MIGRATE_DATABASE_ONLY

Whether to perform database migration and exit immediately.

This is useful for scheduling migrations by starting a ‘migration’ instance of the application.

Set to True to exit after migration.

The MIGRATE_DATABASE environment variable must also be set to True to perform the migration, otherwise nothing will be performed and the container will exit.

Default: False

SSH_PRIVATE_KEY

Provide the contents of the SSH key to perform git clones.

This is an alternative to mounting the ‘.ssh’ directory of the root user.

Default: ”

Application environment variables

The following environment variables are available to configure the application

ADMIN_AUTHENTICATION_TOKEN

Token to use for authorisation to be able to modify modules in the user interface.

Default: “

ADMIN_SESSION_EXPIRY_MINS

Session timeout for admin cookie sessions

Default: 5

ALLOWED_PROVIDERS

Comma-seperated list of allowed providers.

Leave empty to disable allow-list and allow all providers.

Default: “

ALLOW_CUSTOM_GIT_URL_MODULE_PROVIDER

Whether module providers can specify their own git repository source.

Default: True

ALLOW_CUSTOM_GIT_URL_MODULE_VERSION

Whether module versions can specify git repository in terrareg config.

Default: True

ALLOW_MODULE_HOSTING

Whether uploaded modules can be downloaded directly. If disabled, all modules must be configured with a git URL.

Default: True

ALLOW_UNIDENTIFIED_DOWNLOADS

Whether modules can be downloaded with terraform without specifying an identification string in the namespace

Default: False

ANALYTICS_AUTH_KEYS

List of comma-separated values for terraform auth tokens for deployment environments.

E.g. xxxxxx.deploy1.xxxxxxxxxxxxx:dev,zzzzzz.deploy1.zzzzzzzzzzzzz:prod In this example, in the ‘dev’ environment, the auth token for terraform would be: xxxxxx.deploy1.xxxxxxxxxxxxx and the auth token for terraform for prod would be: zzzzzz.deploy1.zzzzzzzzzzzzz.

To disable auth tokens and to report all downloads, leave empty.

To only record downloads in a single environment, specify a single auth token. E.g. zzzzzz.deploy1.zzzzzzzzzzzzz

For information on using these API keys, please see Terraform: https://docs.w3cub.com/terraform/commands/cli-config.html

Default: “

ANALYTICS_TOKEN_DESCRIPTION

Describe to be provided to user about analytics token (e.g. The name of your application)

Default: “

ANALYTICS_TOKEN_PHRASE

Name of analytics token to provide in responses (e.g. application name, team name etc.)

Default: analytics token

APPLICATION_NAME

Name of application to be displayed in web interface.

Default: Terrareg

AUTO_CREATE_MODULE_PROVIDER

Whether to automatically create module providers when uploading module versions, either from create endpoint or hooks.

If disabled, modules must be created using the module provider create endpoint (or via the web interface).

Default: True

AUTO_PUBLISH_MODULE_VERSIONS

Whether new module versions (either via upload, import or hook) are automatically published and available.

If this is disabled, the publish endpoint must be called before the module version is displayed in the list of module versions.

NOTE: Even whilst in an unpublished state, the module version can still be accessed directly, but not used within terraform.

Default: True

CONTRIBUTED_NAMESPACE_LABEL

Custom name for ‘contributed namespace’ in UI.

Default: Contributed

DATABASE_URL

URL for database. Defaults to local sqlite database.

To setup SQLite datbase, use sqlite:///<path to sqlite DB>

To setup MySQL, use mysql+mysqlconnector://<user>:<password>@<host>[:<port>]/<database>

Default: sqlite:///modules.db

DATA_DIRECTORY

Default: ./data

DEBUG

Whether flask and sqlalchemy is setup in debug mode.

Default: False

DELETE_EXTERNALLY_HOSTED_ARTIFACTS

Whether uploaded modules, that provide an external URL for the artifact, should be removed after analysis. If enabled, module versions with externally hosted artifacts cannot be re-analysed after upload.

Default: False

EXAMPLES_DIRECTORY

Directory with a module’s source that contains examples.

Examples are expected to be within sub-directories of the examples directory.

E.g. If EXAMPLES_DIRECTORY is set to examples, with the root module, the following would be expected for an example: examples/myexample/main.tf.

Default: examples

EXAMPLE_ANALYTICS_TOKEN

Example analytics token to provide in responses (e.g. my-tf-application, my-slack-channel etc.).

Note that, if this token is used in a module call, it will be ignored and treated as if an analytics token has not been provided. If analaytics tokens are required, this stops users from accidently using the example placeholder in terraform projects.

Default: my-tf-application

GIT_PROVIDER_CONFIG

Git provider config. JSON list of known git providers. Each item in the list should contain the following attributes:

An example for public repositories might be:

[{"name": "Github", "base_url": "https://github.com/{namespace}/{module}", "clone_url": "ssh://[email protected]:{namespace}/{module}.git", "browse_url": "https://github.com/{namespace}/{module}/tree/{tag}/{path}"},
{"name": "Bitbucket", "base_url": "https://bitbucket.org/{namespace}/{module}", "clone_url": "ssh://[email protected]:{namespace}/{module}-{provider}.git", "browse_url": "https://bitbucket.org/{namespace}/{module}-{provider}/src/{tag}/{path}"},
{"name": "Gitlab", "base_url": "https://gitlab.com/{namespace}/{module}", "clone_url": "ssh://[email protected]:{namespace}/{module}-{provider}.git", "browse_url": "https://gitlab.com/{namespace}/{module}-{provider}/-/tree/{tag}/{path}"}]

Default: []

LISTEN_PORT

Port for server to listen on.

Default: 5000

LOGO_URL

URL of logo to be used in web interface.

Default: /static/images/logo.png

MODULES_DIRECTORY

Directory with a module’s source that contains sub-modules.

submodules are expected to be within sub-directories of the submodule directory.

E.g. If MODULES_DIRECTORY is set to modules, with the root module, the following would be expected for a submodule: modules/submodulename/main.tf.

This can be set to an empty string, to expected submodules to be in the root directory of the parent module.

Default: modules

PUBLISH_API_KEYS

List of comma-separated list of API keys to publish module versions.

To disable authentication for publish endpoint, leave empty.

Default: “

REQUIRED_MODULE_METADATA_ATTRIBUTES

Comma-seperated list of metadata attributes that each uploaded module must contain, otherwise the upload is aborted.

Default: “

SECRET_KEY

Flask secret key used for encrypting sessions.

Can be generated using: python -c 'import secrets; print(secrets.token_hex())'

Default: “

SSL_CERT_PRIVATE_KEY

Path to SSL private certificate key.

If running in a container, the key must be mounted inside the container. This value must be set to the path of the key within the container.

This must be set in accordance with SSL_CERT_PUBLIC_KEY – both must either be set or left empty.

Default: “

SSL_CERT_PUBLIC_KEY

Path to SSL public key.

If running in a container, the key must be mounted inside the container. This value must be set to the path of the key within the container.

This must be set in accordance with SSL_CERT_PRIVATE_KEY – both must either be set or left empty.

Default: “

TERRAFORM_EXAMPLE_VERSION_TEMPLATE

Template of version number string to be used in terraform examples in the UI. This is used by the snippet example of a terraform module and the ‘resource builder’ example.

The template can contain the following placeholders:

  • {major}, {minor}, {patch}
  • {major_minus_one}, {minor_minus_one}, {patch_minus_one}
  • {major_plus_one}, {minor_plus_one}, {patch_plus_one}

Some examples:

  • >= {major}.{minor}.{patch}, < {major_plus_one}.0.0
  • ~> {major}.{minor}.{patch}

For more information, see terraform documentation: https://www.terraform.io/language/expressions/version-constraints

Default: {major}.{minor}.{patch}

TRUSTED_NAMESPACES

Comma-separated list of trusted namespaces.

Default: “

TRUSTED_NAMESPACE_LABEL

Custom name for ‘trusted namespace’ in UI.

Default: Trusted

UPLOAD_API_KEYS

List of comma-separated list of API keys to upload/import new module versions.

For bitbucket hooks, one of these keys must be provided as the ‘secret’ to the webhook.

To disable authentication for upload endpoint, leave empty.

Default: “

VERIFIED_MODULE_LABEL

Custom name for ‘verified module’ in UI.

Default: Verified

VERIFIED_MODULE_NAMESPACES

List of namespaces, who’s modules will be automatically set to verified.

Default: “

Terrareg module metadata

A metadata file can be provided each an uploaded module’s archive to provide additional metadata to terrareg.

For an example, please see: docs/example-terrareg-module-metadata.json

The following attributes are available at the root of the JSON object:

Key Description
owner Name of the owner of the module
description Description of the module
variable_template Structure holding required input variables for module, used for ‘Usage Builder’. See table below
repo_clone_url Url to clone the repository. Optional placeholders {namespace}, {module} and {provider} can be used. E.g. ssh://gitlab.corporate.com/scm/{namespace}/{module}.git
repo_base_url Formatted base URL for project’s repo. E.g. https://gitlab.corporate.com/{namespace}/{module}
repo_browse_url Formatted URL for user-viewable source code. Must contain {tag} and {path} placeholders. E.g. https://github.com/{namespace}/{module}-{provider}/blob/{tag}/{path}

Usage builder configuration

The usage builder requires an array of objects, which define the name, type and description of the variable.

In the following the ‘config input’ refers to the HTML inputs that provide the user with the ability to select/enter values. The ‘terraform input’ refers to the value used for the variable in the outputted terraform example.

There are common attributes that can be added to each of variable objects, which include:

Attribute Description Default
name The name of the ‘config input’. This is also used as the module variable in the ‘terraform input’. Required
type The type of the input variable, see table below. Required
quote_value Boolean flag to determine whether the value generated is quoted for the ‘terraform input’. false
additional_help A description that is provided, along with the terraform variable description in the usage builder Empty
Variable type Description Type specific attributes
text A plain input text box for users to provide a value that it directly used as the ‘terraform input’
boolean Provides a checkbox that results in a true/false value as the ‘terraform input’
static This does not appear in the ‘Usage Builder’ ‘config input’ table, but provides a static value in the ‘terraform input’
select Provides a dropdown for the user to select from a list of choices “choices” must be added to the object, which may either be a list of strings, or a list of objects. If using a list of objects, a “name” and “value” must be provided. Optionally an “additional_content” attribute can be added to the choice, which provides additional terraform to be added to the top of the terraform example. The main variable object may also contain a “allow_custom” boolean attribute, which allows the user to enter a custom text input.

Changelog

See CHANGELOG.md

Local development

Since terraform requires HTTPS with a valid SSL cert, this must be provided in local development

On linux, by default, non-privileged users cannot listen on privileged ports, so the following can be used to route requests locally to port 5000:

sudo iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 443 -j REDIRECT --to-ports 5000

Example to run in local development environment:

virtualenv -ppython3.8 venv
. venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt

# Without SSL cert
ALLOW_CUSTOM_GIT_URL_MODULE_PROVIDER=False ALLOW_CUSTOM_GIT_URL_MODULE_VERSION=False GIT_PROVIDER_CONFIG='[{"name": "Github", "base_url": "https://github.com/{namespace}/{module}", "clone_url": "ssh://[email protected]:{namespace}/{module}.git", "browse_url": "https://github.com/{namespace}/{module}/tree/{tag}/{path}"}, {"name": "Bitbucket", "base_url": "https://bitbucket.org/{namespace}/{module}", "clone_url": "ssh://[email protected]:{namespace}/{module}-{provider}.git", "browse_url": "https://bitbucket.org/{namespace}/{module}-{provider}/src/{tag}/{path}"}, {"name": "Gitlab", "base_url": "https://gitlab.com/{namespace}/{module}", "clone_url": "ssh://[email protected]:{namespace}/{module}-{provider}.git", "browse_url": "https://gitlab.com/{namespace}/{module}-{provider}/-/tree/{tag}/{path}"}]' SECRET_KEY=ec9b8cc5ed0404acb3983b7836844d828728c22c28ecbed9095edef9b7489e85 ADMIN_AUTHENTICATION_TOKEN=password ANALYTICS_AUTH_KEYS=xxxxxx.atlasv1.zzzzzzzzzzzzz:dev,xxxxxx.atlasv1.xxxxxxxxxx:prod VERIFIED_MODULE_NAMESPACES=hashicorp TRUSTED_NAMESPACES=test DEBUG=True AUTO_PUBLISH_MODULE_VERSIONS=False LISTEN_PORT=5001 python ./terrareg.py

# With SSL Cert
# Add the following argument
#  --ssl-cert-private-key ./example/ssl-certs/private.pem --ssl-cert-public-key ./example/ssl-certs/public.pem

Generating DB changes

alembic revision --autogenerate

Applying DB changes

alembic upgrade head

GitHub

View Github