Installation

Quilt has two user-facing components:

• A Python client

• A web catalog

Python client

Python 3.7 or higher is required.

$ pip install 'quilt3[pyarrow]'

If you do not need to serialize and deserialize dataframes with Quilt, you can obtain a smaller install, useful in disk-constrained environments like AWS Lambda, with pip install quilt3.

If you plan to use Quilt Catalog Local Development Mode, add catalog extra while installing quilt3, e.g.:

$ pip install 'quilt3[catalog,pyarrow]'

If you wish to use AWS resources, such as S3 buckets, you will need valid AWS credentials. If this is your first time using the AWS CLI, run the following:

$ pip install awscli
$ aws configure

If you are already using the AWS CLI, you may use your existing profile, or create a new profile.

Developer

Install the current Quilt client from master:

$ pip install git+https://github.com/quiltdata/quilt.git#subdirectory=api/python

Enterprise Installation (on AWS)

Quilt is a data mesh that verifies the integrity of your data so that teams can find, understand, and file discoveries based on data of any size or in any format.

A Quilt instance is a private portal that runs in your virtual private cloud (VPC).

Quilt supports multiple deployment methods including CloudFormation, AWS Marketplace, and Terraform.

Help and Advice

We encourage users to contact us before deploying Quilt. We will make sure that you have the latest version of Quilt, and walk you through the CloudFormation deployment.

We recommend that all users do one or more of the following:

• Schedule a Quilt engineer to guide you through the installation

• Join Quilt on Slack to ask questions and connect with other users

• Email Quilt

Requirements and Prerequisites

Knowledge Requirements

Running Quilt requires working knowledge of AWS CloudFormation, AWS S3 and Elasticsearch Service.

Before you install Quilt

You will need the following:

1. An AWS account.

   1. The service-linked role for Elasticsearch

   This role is not created automatically when you use Cloudformation or other APIs.

   You can create the role as follows: aws iam create-service-linked-role --aws-service-name es.amazonaws.com

2. IAM Permissions to create the CloudFormation stack (or Add products in Service Catalog).

   1. You may choose to use a CloudFormation service role for stack creation and updates.

   2. Refer to this example service role and modify as needed to fit your use case.

   Ensure that your service role is up-to-date with the example before every stack update so as to prevent installation failures.

3. The ability to create DNS entries, such as CNAME records, for your company's domain.

4. An SSL certificate in the same region as your Quilt instance to secure the domain where your users will access your Quilt instance.

   1. For example, to make your Quilt catalog available at https://quilt.mycompany.com, you require a certificate for either *.mycompany.com or for the following 3 domains: quilt.mycompany.com, quilt-registry.mycompany.com and quilt-s3-proxy.mycompany.com in the AWS Certificate Manager.

   2. You may either create a new certificate, or import an existing certificate.

   3. The ARN for this certificate or set of certificates is required for use as the CertificateArnELB CloudFormation parameter.

5. For maximum security, Quilt requires a region that supports AWS Fargate. As of this writing, all U.S. regions support Fargate.

6. An S3 Bucket for your team data. This may be a new or existing bucket. The bucket should not have any notifications attached to it (S3 Console > Bucket > Properties > Events). Quilt will need to install its own notifications. Installing Quilt will modify the following Bucket characteristics:

   1. Permissions > CORS configuration (will be modified for secure web access).

   2. Properties > Object-level logging (will be enabled).

   3. Properties > Events (will add one notification).

   Buckets in Quilt may choose to enable versioning or disable versioning. It is strongly recommended that you keep versioning either on or off during the entire lifetime of the bucket. Toggling versioning on and off incurs edge cases that may cause bugs with any state that Quilt stores in ElasticSearch due to inconsistent semantics of ObjectRemoved:DeleteMarkerCreated.

7. Available CloudTrail Trails in the region where you wish to host your stack (learn more).

8. A license key or an active subscription to Quilt Business on AWS Marketplace.

   1. Click Continue to Subscribe on the Quilt Business Listing to subscribe then return to this page for installation instructions.

   2. The CloudFormation template and instructions on AWS Marketplace are infrequently updated and may be missing critical bugfixes.

Installation Methods

AWS Marketplace

You can install Quilt via AWS Marketplace. As indicated above, we recommend that you contact us first.

AWS Service Catalog

1. Email [email protected] with your AWS account ID to request access to Quilt through the AWS Service Catalog and to obtain a license key.

2. Click the service catalog link that you received from Quilt. Arrive at the Service Catalog. Click IMPORT, lower right.

   
3. Navigate to Admin > Portfolios list > Imported Portfolios. Click Quilt Enterprise.

   
4. On the Portfolio details page, click ADD USER, GROUP OR ROLE. Add any users, including yourself, whom you would like to be able to install Quilt.

   
5. Click Products list, upper left. Click the menu to the left of Quilt CloudFormation Template. Click Launch product. (In the future, use the same menu to upgrade Quilt when a new version is released.)

   
6. Continue to the CloudFormation section. Note: the following screenshots may differ slightly from what you see in Service Catalog.

CloudFormation

You can perform stack update and creation with the AWS Console, AWS CLI, Terraform, or other means.

In all cases it is highly recommended that you set the --on-failure policy to ROLLBACK so as to avoid incomplete rollback and problematic stack states. In the AWS Console this option appears under the phrase "Stack failure options."

1. Specify stack details in the form of a stack name and CloudFormation parameters. Refer to the descriptions displayed above each text box for further details. Service Catalog users require a license key. See Before you install Quilt for how to obtain a license key.

   
2. If you wish to use a service role, specify it as follows:

   
3. Service Catalog users, skip this step. Under Stack creation options, enable termination protection. This protects the stack from accidental deletion. Click Next.

   
4. Service Catalog users, skip this step. Check the box asking you to acknowledge that CloudFormation may create IAM roles, then click Create.

   
5. CloudFormation may take between 30 and 90 minutes to create your stack. You can monitor progress under Events. On completion you will see CREATE_COMPLETE.

   
6. To finish the installation, you will want to view the stack Outputs.

   

Terraform

You can also install Quilt using Terraform, which enables more granular infrastructure-as-code control.

Terraform users must request a compatible CloudFormation template from Quilt:

Contact your account manager to obtain a template that works with Terraform and includes necessary variables.

1. Set up your project directory as follows:

   Copy

   quilt_stack/
   ├── main.tf
   └── my-company.yml

   Use examples/main.tf as a template.

2. Define your AWS profile:

   Copy

   export AWS_PROFILE=your-profile-name

3. Initialize Terraform:

   Copy

   terraform init

4. Plan and apply:

   Copy

   terraform plan -out=tfplan
   terraform apply tfplan

5. Use terraform output to obtain values such as the admin password or endpoint URLs.

Note: We recommend using remote state and not storing passwords in version control.

For detailed configuration options, including search sizing and common pitfalls, see the Terraform README.

CNAMEs

In order for your users to reach the Quilt catalog you must set three CNAMEs that point to the LoadBalancerDNSName as shown below and in the Outputs of your stack.

Quilt is now up and running. You can click on the QuiltWebHost value in Outputs and log in with your administrator password to invite users.

Routine Maintenance and Upgrades

Releases are sent to customers over email. We recommend that you apply new releases as soon as possible to benefit from the latest security updates and features.

CloudFormation updates

To update your Quilt stack, apply the latest CloudFormation template in the CloudFormation console as follows.

By default, previous parameter values carry over.

1. Navigate to AWS Console > CloudFormation > Stacks

2. Select your Quilt stack

3. Click Update (upper right)

4. Choose Replace current template

5. Enter the Amazon S3 URL for your template

6. Click Next (several times) and proceed to apply the update

Terraform updates

See above.

Upgrading from network 1.0 to network 2.0

Upgrading to the Quilt 2.0 network configuration provides improved security by means of isolated subnets and a preference for private routing.

An upgrade to the 2.0 network, unlike routine Quilt upgrades, requires you to create a new stack with a new load balancer. You must therefore also update your CNAMEs to point to the new load balancer.

Create a new stack with an existing configuration

Terraform users can create a new Quilt stack with the same configuration as an existing stack. This is typically useful when upgrading to the 2.0 network.

Configuration refers to the Quilt stack buckets, roles, policies, and other administrative settings, all of which are stored in RDS.

Perform the following steps:

1. Contact your Quilt account manager for a template that supports Terraform.

2. Take a manual snapshot of the current Quilt database instance. For an existing Quilt stack this resource has the logical ID "DB". Note the snapshot identifier ("Snapshot name" in the AWS Console, DBSnapshotIdentifier in the following AWS CLI command):

   Copy

   aws rds describe-db-snapshots

   Be sure to take a manual snapshot. Do not rely on automatic snapshots, which are deleted when the parent stack is deleted.

3. Apply the quilt Terraform module to your new template and provide the snapshot identifier to the db_snapshot_identifier= argument.

   You must use a Quilt CloudFormation template that supports an existing database, existing search domain, and existing vpc in order for the terraform modules to function properly.

4. You now have a new Quilt stack with a configuration equivalent to your prior stack. Verify that the new stack is working as desired. Delete the old stack.