# About Quilt

Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.

***

## How to Get Started

Quilt consists of three main elements:

* [Quilt Platform](#quilt-platform-overview) which is a cloud platform for interacting with, visualizing, searching and querying Quilt Packages, which is hosted in an organization's AWS Account.
* [Quilt Python SDK](#quilt-python-sdk) which provides the ability to create, push, install and delete Quilt Packages.
* [Quilt Ecosystem](#quilt-ecosystem-and-integrations) which provide extension of the core Quilt Capabilities to enable typical elements of life sciences workflows, such as incorporating orchestration data, and connecting packages to Electronic Lab Notebooks.

To dive deeper into the capabilities of Quilt, start with our [Quick Start Guide](/quilt-python-sdk/quickstart) or explore the [Installation Instructions](/quilt-python-sdk/installation) for setting up your environment.

If you have any questions or need help, join our [Slack community](https://slack.quilt.bio) or submit a support request to <support@quilt.bio>.

***

## Navigating the Documentation

The Quilt documentation is structured to guide users through different layers of the platform, from basic concepts to advanced integrations. Whether you're a business user, developer, or platform administrator, the docs will help you quickly find the information you need.

### Quilt Platform Overview

The **Quilt Platform** powers the core features of the Quilt data catalog, providing tools for browsing, searching, and visualizing data stored in AWS S3. The platform is ideal for teams needing to collaborate on data, with capabilities like document previews and metadata collection.

**Core Sections:**

* [Architecture](/architecture) - Learn how Quilt is architected.
* [Mental Model](/mentalmodel) - Understand the guiding principles behind Quilt.
* [Metadata Management](/metadata) - Manage metadata at scale.

For users of the Quilt Platform (often referred to as the Catalog):

* [Bucket Browsing](/quilt-platform-catalog-user/filebrowser) - Navigate through S3 buckets.
* [Document Previews](/quilt-platform-catalog-user/preview) - Visualize documents and datasets directly in the web interface.
* [Search](/quilt-platform-catalog-user/search) - Leverage Quilt’s powerful search capabilities.
* [Visualization & Dashboards](/quilt-platform-catalog-user/visualizationdashboards) - Create visual dashboards for data insights.

For administrators managing Quilt deployments:

* [Admin Settings UI](/quilt-platform-administrator/admin) - Control platform settings and user access.
* [Catalog Configuration](/quilt-platform-administrator/preferences) - Set platform preferences.
* [Cross-Account Access](/quilt-platform-administrator/crossaccount) - Manage multi-account access to S3 data.

### Quilt Python SDK

The **Quilt Python SDK** allows users to programmatically manage data packages, version datasets, and automate data workflows. Whether you're uploading a package, fetching data, or scripting custom workflows, the SDK provides the flexibility needed for deeper integrations.

* [Installation](/quilt-python-sdk/installation) - Get started with the Quilt SDK.
* [Quick Start](/quilt-python-sdk/quickstart) - Follow a step-by-step guide to building and managing data packages.
* [Editing and Uploading Packages](/quilt-python-sdk/editing-a-package) - Learn how to version, edit, and share data.
* [API Reference](/quilt-python-sdk/api-reference/api) - Detailed API documentation for developers.

### Quilt Ecosystem and Integrations

The **Quilt Ecosystem** extends the platform with integrations and plugins to fit your workflow. Whether you're managing scientific data or automating packaging tasks, Quilt can be tailored to your needs with these tools:

* [Benchling Packager](/quilt-ecosystem-integrations/benchling) - Package electronic lab notebooks from Benchling.
* [Nextflow Plugin](/quilt-ecosystem-integrations/nextflow) - Integrate with Nextflow pipelines for bioinformatics.

***

## Who Should Use Quilt?

Quilt is for teams across industries like machine learning, biotech, and analytics who need to manage large datasets, collaborate seamlessly, and track the lifecycle of their data. Whether you're a data scientist, engineer, or administrator, Quilt helps streamline your data management workflows.

## What Can You Do with Quilt?

* **Share**: Easily share versioned data using simple URLs and email invites.
* **Understand**: Enrich data with inline documentation and visualizations for better insights.
* **Discover**: Use metadata and search tools to explore data relationships across projects.
* **Model**: Version and manage large data sets that don't fit traditional git repositories.
* **Decide**: Empower your team with auditable data for better decision-making.

***


# Architecture

## Overview

Quilt packages unify data and metadata. Quilt data packages are reproducible, discoverable, and trusted data sets that live in blob storage.

A package manifest includes both *metadata* and physical keys to primary data.

All package metadata and data are stored in your S3 buckets. A slice of the package-level metadata, as well as S3 object contents, are sent to an ElasticSearch cluster managed by Quilt. All Quilt package manifests are accessible via SQL using AWS Athena.

![](/files/-MXXF1KPojFE7Uju4zo4)

## Enterprise Architecture

Each Quilt instance consists of a CloudFormation stack that is privately hosted in your AWS account. The stack includes backend services for the web catalog, single sign-on, user identification and access, an ElasticSearch cluster, and more.

Quilt uses subnets and security groups to isolate network services and runs key services within the VPC.

A private stack with an inward load balancer is shown below.

![Architecture (private ELBv2)](/files/KdvyaMUf3pY8ZUWs1WEK)

For an internet-facing load balancer the data plane remains the same, as shown below.

![Architecture (public ELBv2)](/files/E2bQzlDzrhbqOpEENODP)

### Network Architecture

You may provide your own VPC and subnets to a Quilt stack or have the Quilt stack create its own network.

> If you provide the subnets you may choose to reuse subnets across parameters. For example you can use the same subnets for the Private and User subnet parameters.

You may optionally provide your own VPC CIDR block if the default block of 10.0.0.0/16 conflicts with shared or peered VPC services. We recommend a CIDR block no smaller than /24 (256 addresses) for production, multi-AZ deployments. Larger CIDR blocks are easier to upgrade to new Quilt versions with expanded services.

> For cost-sensitive deployments, Quilt ECS services can be configured to use a single AZ.

> You may use a combination of interface endpoints and gateway endpoints to restrict the data plane traffic shown above to your VPC. See [Private endpoint access](/quilt-platform-administrator/advanced/private-endpoint-access) for more.

#### Production, multi-AZ subnet division for private ELBv2 (you provide the network)

| Type    | AZ | Description                    | Services                                | IPs needed† |
| ------- | -- | ------------------------------ | --------------------------------------- | ----------- |
| Private | a  | Routes to Internet             | ECS, Lambda                             | 32          |
| Private | b  | "                              | "                                       | 32          |
| Intra   | a  | Does not route to Internet     | RDS, OpenSearch\*                       | 32          |
| Intra   | b  | "                              | "                                       | 32          |
| User    | a  | Reachable by GUI catalog users | App load balancer, API Gateway Endpoint | 16          |
| User    | b  | "                              | "                                       | 16          |

> \* One IP per master node, one IP per data node

> † Includes 5 IPs for AWS (network, routing, DNS, reserved, broadcast) plus room for new services in future updates.

Below are the subnet configurations and sizes for Quilt version 2.0 networks, new as of June 2023. The configuration is similar to the [AWS Quick Start VPC](https://aws-quickstart.github.io/quickstart-aws-vpc/).

#### Subnet division when Quilt creates the VPC

* 2 public subnets (1/4 the VPC CIDR), needed for:
  * an internet-facing application load balancer
  * NAT gateways, which give private-subnet services outbound internet access (for example, reaching Quilt's validation service — without it, password-reset emails can't be sent)
* 2 private subnets for Quilt services in ECS or Lambda, and an inward facing application load balancer (1/2 of the VPC CIDR)
* 2 private subnets for intra-VPC traffic to and from the Quilt RDS database and OpenSearch domain (1/8 of the VPC CIDR)
* Unused (1/8 of the VPC CIDR)

### Sizing

The Quilt CloudFormation template will automatically configure appropriate instance sizes for RDS, ECS (Fargate), Lambda and Elasticsearch Service. Some users may choose to adjust the size and configuration of their Elasticsearch cluster. All other services should use the default settings.

### Elasticsearch Service Configuration

By default, Quilt configures an Elasticsearch cluster with 3 master nodes and 2 data nodes. Please contact the Quilt support team before adjusting the size and configuration of your cluster to avoid disruption.

### Cost

The infrastructure costs of running a Quilt stack vary with usage. Baseline infrastructure costs start at $620 and go up from there. See below for a breakdown of baseline costs for `us-east-1` at 744 hours per month.

| Service               | Cost                         |
| --------------------- | ---------------------------- |
| Elasticsearch Service | $258.42                      |
| RDS                   | $75.56                       |
| ECS (Fargate)         | $26.64                       |
| Lambda                | Variable                     |
| CloudTrail            | Variable                     |
| Athena                | Variable                     |
| **Total**             | **$619.03 + Variable Costs** |

## Security Architecture

All customer data and metadata in Quilt is stored in S3. It may also be cached in Elasticsearch Service (shown in red in the diagram below). No other services in the Quilt stack store customer data.

![](/files/-Mf0M7JB6dyaNZkrYz9-)

We recommend using [S3 encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html) and [Elasticsearch Service encryption at rest](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/encryption-at-rest.html) to provide maximum protection.

User email addresses are stored by the Identity Service in RDS Postgres (part of the Quilt stack). User email addresses are also sent through an encrypted channel to the customer support messaging system ([Intercom](https://www.intercom.com/)).


# Mental Model

This guide explains the fundamental concepts behind Quilt's data management system. Think of it as your roadmap to understanding how Quilt organizes, versions, and manages data.

## 🎯 The Big Picture

Quilt treats data like code - with versioning, immutability, and collaboration built-in. Instead of managing individual files scattered across storage systems, you work with **packages** that bundle related data together with metadata and provenance.

```
Traditional Approach          →    Quilt Approach
├── file1.csv                      📦 myteam/customer-data
├── file2.json                     ├── 📄 customers.csv
├── file3.parquet                  ├── 📄 transactions.json  
└── README.txt                     ├── 📄 analytics.parquet
                                   ├── 📄 README.md
                                   └── 🏷️  metadata + version hash
```

## 📦 Core Concept: Packages

### What is a Package?

A **package** is Quilt's fundamental unit of data organization. Think of it as a versioned, immutable collection of related files with a clear identity and history.

**Key Properties:**

* **Immutable**: Once created, package contents never change
* **Versioned**: Each change creates a new version with a unique hash
* **Named**: Human-readable names like `myteam/customer-analytics`
* **Tracked**: Complete history and provenance of all changes

### Package Anatomy

Every package consists of:

```
📦 Package: myteam/customer-data
├── 🏷️  Name: "myteam/customer-data"
├── 🔐 Hash: "a1b2c3d4..." (unique version identifier)
├── 📋 Manifest: (maps logical → physical locations)
├── 📁 Files:
│   ├── customers.csv
│   ├── transactions.json
│   └── README.md
└── 📊 Metadata: {"description": "Q3 customer analysis", "version": "2.1"}
```

### Real-World Example

```python
import quilt3

# Load a package (using public example)
pkg = quilt3.Package.browse("examples/hurdat", "s3://quilt-example")

# Package info
print(f"Package hash: {pkg.top_hash}")     # Unique version identifier
print(f"Files: {len(pkg)}")                # Number of files in package

# List available files
for key in pkg:
    print(f"File: {key}")
```

## 🗂️ The Manifest System

### Understanding Manifests

The **manifest** is Quilt's "table of contents" - it maps user-friendly names to actual file locations and includes integrity information.

**Manifest Entry Structure:**

```
(LOGICAL_KEY, PHYSICAL_KEYS, HASH, METADATA)
```

### Logical vs Physical Keys

| Aspect        | Logical Key            | Physical Key                                       |
| ------------- | ---------------------- | -------------------------------------------------- |
| **Purpose**   | User-friendly name     | Actual storage location                            |
| **Example**   | `"data/customers.csv"` | `"s3://bucket/a1b2c3/customers.csv?versionId=xyz"` |
| **Stability** | Stable across versions | Changes with storage                               |
| **Usage**     | Code references        | Internal system use                                |

### Example Manifest Entry

```json
{
    "logical_key": "data/customers.csv",
    "physical_keys": [
        "s3://company-data/datasets/customers_v2.csv?versionId=abc123"
    ],
    "size": 1048576,
    "hash": {
        "type": "SHA256",
        "value": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
    },
    "meta": {
        "schema_version": "2.1",
        "last_updated": "2024-08-26",
        "data_quality": "validated"
    }
}
```

**Why This Matters:**

* ✅ **Portability**: Move data between storage systems without breaking code
* ✅ **Integrity**: Cryptographic hashes ensure data hasn't been corrupted
* ✅ **Metadata**: Rich context about each file's purpose and properties
* ✅ **Versioning**: Track exactly what changed between package versions

## 🏢 Registries: Where Packages Live

### Registry Concept

A **registry** is where Quilt stores package manifests and optionally the data itself. Think of it as a "database" of packages.

**Supported Registry Types:**

* 🌐 **S3 Buckets**: Cloud-native, scalable, with built-in versioning
* 💻 **Local Disk**: For development and testing
* 🔮 **Future**: GCP, Azure, NAS (on roadmap)

### Registry Examples

```python
import quilt3

# Different registry types
local_packages = quilt3.list_packages()                    # Local registry
cloud_packages = quilt3.list_packages("s3://my-bucket")   # S3 registry
public_data = quilt3.list_packages("s3://quilt-example")  # Public registry
```

## 🌊 Buckets as Branches

### The Git Analogy

In Quilt, **S3 buckets function like Git branches** - each represents a different stage or environment in your data lifecycle.

```
Git Workflow              →    Quilt Workflow
├── feature-branch             ├── s3://dev-bucket
├── develop                    ├── s3://staging-bucket  
├── staging                    ├── s3://prod-bucket
└── main                       └── s3://archive-bucket
```

### Recommended Bucket Strategy

```mermaid
graph LR
    A[Raw Data] --> B[s3://company-raw]
    B --> C[s3://company-staging] 
    C --> D[s3://company-prod]
    D --> E[s3://company-archive]
    
    B -.-> F[Data Validation]
    C -.-> G[Quality Assurance]
    D -.-> H[Production Use]
```

**Three-Bucket Minimum:**

1. **🔴 Raw Bucket** (`s3://company-raw`)
   * Ingested data, minimal processing
   * Experimental datasets
   * Temporary analysis results
2. **🟡 Staging Bucket** (`s3://company-staging`)
   * Validated and cleaned data
   * Ready for testing and QA
   * Pre-production datasets
3. **🟢 Production Bucket** (`s3://company-prod`)
   * Fully validated, production-ready data
   * Used by live applications and dashboards
   * Strict access controls and governance

### Package Promotion Workflow

```python
# Promote a package through environments
import quilt3

# 1. Start in raw environment
raw_pkg = quilt3.Package()
raw_pkg.set("data.csv", "raw_data.csv")
raw_pkg.push("myteam/dataset", registry="s3://company-raw")

# 2. Validate and promote to staging
staging_pkg = quilt3.Package.browse("myteam/dataset", registry="s3://company-raw")
# ... perform validation ...
staging_pkg.push("myteam/dataset", registry="s3://company-staging")

# 3. Final promotion to production
prod_pkg = quilt3.Package.browse("myteam/dataset", registry="s3://company-staging")
# ... final checks ...
prod_pkg.push("myteam/dataset", registry="s3://company-prod")
```

## 🔄 Immutability and Versioning

### Why Immutability Matters

**Immutable packages** mean that once created, a package version never changes. This provides:

* ✅ **Reproducibility**: Analyses can be exactly repeated
* ✅ **Audit Trail**: Complete history of all changes
* ✅ **Rollback Safety**: Easy to revert to previous versions
* ✅ **Parallel Work**: Teams can work simultaneously without conflicts

### Version Management

```python
# Working with package versions
import quilt3

# Get latest version (using public example)
latest = quilt3.Package.browse("examples/hurdat", "s3://quilt-example")
print(f"Latest hash: {latest.top_hash}")

# Get specific version
specific = quilt3.Package.browse("examples/hurdat", "s3://quilt-example", top_hash=latest.top_hash)
print(f"Specific version")

# Compare versions
if latest.top_hash == specific.top_hash:
    print("Same version")
```

## 🎯 Practical Mental Model

### Think of Quilt Like...

| If you're familiar with... | Think of Quilt as...                                                         |
| -------------------------- | ---------------------------------------------------------------------------- |
| **Git**                    | Git for data - versioning, branching (buckets), immutable commits (packages) |
| **Docker**                 | Container images for data - immutable, portable, with manifests              |
| **Package Managers**       | npm/pip for datasets - named packages, versions, dependencies                |
| **Databases**              | Schema-aware data warehouse with built-in versioning and lineage             |

### Key Principles to Remember

1. **📦 Package-Centric**: Always think in terms of related collections, not individual files
2. **🔒 Immutable**: Versions never change - create new versions instead of modifying
3. **🏷️ Named & Hashed**: Every package has a human name and cryptographic identity
4. **🌊 Bucket Workflows**: Use different buckets for different data lifecycle stages
5. **📋 Manifest-Driven**: Logical names abstract away physical storage details

## 🚀 Next Steps

Now that you understand Quilt's mental model:

1. **Try It**: Follow the [Quick Start](/quilt-python-sdk/quickstart) to create your first package
2. **Learn Workflows**: Explore [package workflows](/quilt-python-sdk/uploading-a-package)
3. **Set Up Team Access**: Configure [team access and roles](/quilt-platform-administrator/admin)
4. **Advanced Topics**: Learn about [schemas and validation](/workflows)

***

**Remember**: Quilt transforms chaotic data management into organized, versioned, collaborative workflows. The mental model is simple - treat your data like code, and Quilt handles the complexity!


# Metadata Management

### Why metadata matters

Data without labels and documentation quickly become meaningless. In Quilt, metadata are represented as dictionaries that can refer to specific objects or entire packages.

Metadata solves the following problems:

* *Collaboration* — Metadata are the clues that enable developers, non-developers, and code to create a shared understanding and vocabulary
* *Discoverability* — Quilt package-level metadata are searchable via ElasticSearch; package-level and object-level metadata are queryable via AWS Athena
* *Trust* — Quilt metadata are screened against JSON schemas that you define to ensure that annotations are complete and type-safe. (See [Workflows](/workflows))
* *Understandability* - Metadata are a love letter to the future; with metadata in hand, users can better understand what data mean, where they came from, and how they might be used in the future
* *Longitudinal analysis* — Need to cut across packages and isolate data sets based on varying dimensions? Metadata makes this possible.

#### Metadata in Quilt

Quilt packages contain one of two types of metadata:

* *Object-level metadata* for each object or entry in the package
* *Package-level metadata* for each revision of the package

#### The Goldilocks problem

If you require your users to input too much metadata, they'll avoid your system. Too little metadata and it's hard to understand or trust your collection. Our rule of thumbs are:

> Minimize human-entered metadata to less than a dozen fields

> Maximize machine-entered data to capture any facts or dimensions that might be useful in the future: date, author, etc. (by default, Quilt automatically captures metadata like file size and SHA-256 hash)

### Entering metadata

When you create or revise a package in Quilt, you can edit the package-level metadata.

![](/files/-MWdjTFFrn9MbTKWXiGs)

Metadata can be entered by hand, or you can drag and drop one of the following file types on the Metadata section:

* CSV
* XLS
* XLSX
* XLSM
* ODS
* FODS

The keys of your dataset may be represented as either column headers, or as the values of a single column. Quilt picks the orientation that best overlaps with the workflow schema, be that orientation row-major or column-major.

#### Spreadsheet example

Suppose your spreadsheet looks like this:

| Metal   | Color  | Price |
| ------- | ------ | ----- |
| gold    | yellow | 1700  |
| silver  | gray   | 25    |
| rhodium | gray   | 25000 |

When you drag and drop the spreadsheet, Quilt converts it:

```json
{
  "Metal": ["gold", "silver", "rhodium"],
  "Color": ["yellow", "gray", "gray"],
  "Price": [1700, 25, 25000]
}
```

> Note: empty cells have a value of `null`

#### Supported types

Metadata may contain strings, numbers, objects, booleans, and dates. Dates are converted to the YYYY-MM-DD format.

If your workflow Schema has `{ type: "array" }` for a cell, Quilt converts this string to an array by splitting the cell on comma. If your input file contains JSON fragments, such as `"{"a": 1, "b", 2}"`, Quilt will convert those strings to objects.

### Limitations

Quilt recommends, and the APIs will soon enforce, that users limit each instance of package-level and object-level metadata to 1MB or less so that your package works well with S3 Select (1MiB row limit) and AWS Athena (32MB row limit), both of which are used by the Quilt backend.


# Metadata Workflows

> This feature requires the `quilt3` API version 3.3 or higher.

### Overview

A Quilt *workflow* is a quality gate that you set on a bucket to ensure the quality of your data and metadata *before* it becomes a Quilt package. You can create as many workflows as you like to accommodate all of your data creation patterns.

### Why metadata quality matters

Under the hood, Quilt workflows use [JSON Schema](https://json-schema.org) to check that package metadata have the right *shape*. Metadata shape determines which keys are defined, their values, and the types of the values.

Ensuring the quality of your data has long-lasting implications:

1. *Consistency* — if labels and other metadata don't use a consistent, controlled vocabulary, reuse becomes difficult and trust in data declines
2. *Completeness* — if your workflows do not require users to include files, documentation, labels, etc. then your data is on its way towards becoming mystery data and ultimately junk data that no one can use
3. *Context* — data can only be reused if users know where it came from, what it means, who touched it, and what the related datasets are

From the standpoint of querying engines like Amazon Athena, metadata that lacks consistency and completeness is extremely difficult to query longitudinally and depreciates over time (as team members change, platforms change, and tribal knowledge is lost).

### Use cases

* Ensure that labels are correct and drawn from a controlled vocabulary (e.g. ensure that the only labels in a package of images are either "bird" or "not bird"; avoid data entry errors like "birb")
* Ensure that users provide a `README.md` for every new package
* Ensure that included files are non-empty
* Ensure that every new package (or dataset) has enough labels so that it can be reused (e.g. Date, Creator, Type, etc.)

### Get started

To get started, create a configuration file in your Quilt S3 bucket at `s3://BUCKET-NAME/.quilt/workflows/config.yml`.

Here's a complex example:

```yaml
version:
  base: "1"
  catalog: "1"
workflows:
  alpha:
    name: Search for aliens
    is_message_required: true
  beta:
    name: Studying superpowers
    metadata_schema: superheroes
  gamma:
    name: Nothing special
    description: TOP SECRET
    is_message_required: true
    metadata_schema: top-secret
    handle_pattern: ^(employee1|employee2)/(staging|production)$
    entries_schema: must-have-foo-bar-meta
    catalog:
      package_handle:
        files: <%= username %>/<%= directory %>
        packages: <%= username %>/production
schemas:
  superheroes:
    url: s3://quilt-dev-metadata/schemas/superheroes.schema.json
  top-secret:
    url: s3://quilt-dev-metadata/schemas/top-secret.schema.json
  must-have-foo-bar-meta:
    url: s3://quilt-dev-metadata/schemas/must-have-foo-bar-meta.schema.json
```

#### Setting a default workflow

Once a configuration file is present, you usually **must** specify a workflow when pushing a package from `quilt3` to that bucket. You can however specify a `default_workflow` value at the top-level in your `config.yml` file:

```yaml
version:
  base: "1"
  catalog: "1"
default_workflow: "experiment"
is_workflow_required: False
workflows:
  experiment:
    name: Experiment
    metadata_schema: experiment-universal
schemas:
  experiment-universal:
    url: s3://quilt-dev-metadata/.quilt/workflows/schemas/experiment-universal.json
```

This specifies which workflow will be used (`experiment`) if a `workflow` parameter in the `Package.push()` API call or CLI is not provided. Otherwise, all `push` calls must specify a workflow or the push will fail.

#### Bypassing workflow validation

If you wish for your users to be able to skip workflow validation altogether, you can make workflow validation optional with `is_workflow_required: False` at the top-level in your `config.yml` file:

```yaml
version:
  base: "1"
  catalog: "1"
is_workflow_required: False
```

Now your users can specify `workflow=None` in the Python API (or `--workflow ''` in the CLI) when they push packages (or drop the `workflow` parameter altogether).

```python
quilt3.Package().push(
        'test/package',
        registry='s3://quilt-dev-metadata',
        workflow=None)

# Package test/package@06b2815 pushed to s3://quilt-dev-metadata
```

#### Specifying a workflow

Otherwise, once you have a configuration you must specify a workflow when pushing a package:

```python
import quilt3
quilt3.Package().push('test/package', registry='s3://quilt-dev-metadata')

# QuiltException: Workflow required, but none specified.
```

Let's retry, adding the `workflow=` parameter:

```python
quilt3.Package().push('test/package', registry='s3://quilt-dev-metadata', workflow='alpha')

# QuiltException: Commit message is required by workflow, but none was provided.
```

The above `QuiltException` is caused by `is_message_required: true`. Here's how we can pass the workflow:

```python
quilt3.Package().push(
        'test/package',
        registry='s3://quilt-dev-metadata',
        message='added info about UFO',
        workflow='alpha')

# Package test/package@bc9a838 pushed to s3://quilt-dev-metadata
```

Now let's push with `workflow='beta'`:

```python
quilt3.Package().push(
        'test/package',
        registry='s3://quilt-dev-metadata',
        workflow='beta')

# QuiltException: Metadata failed validation: 'superhero' is a required property.
```

We encountered another exception because the `beta` workflow specifies `metadata_schema: superheroes`. Therefore, the `test/package` metadata must validate against the [JSON Schema](https://json-schema.org/) located at `s3://quilt-dev-metadata/schemas/superheroes.schema.json`:

```json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "http://example.com/superheroes.schema.json",
  "properties": {
    "superhero": {
      "enum": [
        "Spider-Man",
        "Superman",
        "Batman"
      ]
    }
  },
  "required": [
    "superhero"
  ]
}
```

Note that `superhero` is a required property of the package metadata, which is specified using `set_meta`.

```python
quilt3.Package().set_meta({'superhero': 'Batman'}).push(
        'test/package',
        registry='s3://quilt-dev-metadata',
        workflow='beta')

# Package test/package@c4691d8 pushed to s3://quilt-dev-metadata
```

For the `gamma` workflow, both `is_message_required: true` and `metadata_schema` are set, so both `message` and package metadata are validated:

```python
quilt3.Package().push(
        'test/package',
        registry='s3://quilt-dev-metadata',
        workflow='gamma')

# QuiltException: Metadata failed validation: 'answer' is a required property.

quilt3.Package().set_meta({'answer': 42}).push(
        'test/package',
        registry='s3://quilt-dev-metadata',
        workflow='gamma')

# QuiltException: Commit message is required by workflow, but none was provided.

quilt3.Package().set_meta({'answer': 42}).push(
        'test/package',
        registry='s3://quilt-dev-metadata',
        message='at last all is set up',
        workflow='gamma')

# Package test/package@6331508 pushed to s3://quilt-dev-metadata
```

### Data quality controls

In addition to package-level metadata. Quilt workflows enable you to validate package names, and basic file metadata.

> You must include the following schema version at the root of your config.yml in order for any catalog-specific features to function:

```yaml
version:
  base: "1"
  catalog: "1"
```

#### Package name defaults (Quilt catalog)

By default the Quilt catalog auto-fills the package handle **prefix** when creating or revising a package according to the following logic:

* Packages tab: username (everything before the @ in your sign-in email). Equivalent to

```yaml
catalog:
  package_handle:
    packages: <%= username %>
```

* Files tab: parent directory name. Equivalent to

```yaml
catalog:
  package_handle:
    files: <%= directory %>
```

You can customize the default prefix with `package_handle` key in one or both of the following places:

* Set `catalog.package_handle.(files|packages)` at the root of config.yml to affect all workflows
* Set `workflows.WORKFLOW.catalog.package_handle.(files|packages)` to affect the tabs and workflow in question

**Example**

```yaml
catalog:
  # default for all workflows for Packages tab
  package_handle:
    packages: analysis/
workflows:
  my-workflow:
    name: My workflow
    catalog:
      # defaults for my-workflow, different for each tab
      package_handle:
        files: <%= username %>/<%= directory %>
        packages: <%= username %>/production
```

#### Package name validation

You can validate package names with `WORKFLOW.handle_pattern`, which accepts [JavaScript regular expression](https://datatracker.ietf.org/doc/html/draft-handrews-json-schema-validation-01#section-6.3.3).

> By default, patterns are not anchored. You can explicitly add start (`^`) and end (`$`) markers as needed.

**Example**

```yaml
workflows:
  name: My workflow
  my-workflow:
    handle_pattern: ^(employee1|employee2)/(production|staging)$
```

#### Package file validation

You can validate the names, sizes and metadata of files in the package with `WORKFLOW.entries_schema`. The provided schema runs against an array of objects known as *package entries*. Each package entry defines a logical key (its relative path and name in the parent package), size (in bytes) and metadata.

**Example**

```yaml
workflows:
  myworkflow-1:
    name: 'My workflow #1'
    entries_schema: must-contain-readme
  myworkflow-2:
    name: 'My workflow #2'
    entries_schema: must-contain-readme-summarize-at-least-1byte
    description: Must contain non-empty README.md and quilt_summarize.json 
        at package root; no more than 4 files
  myworkflow-3:
    name: 'My workflow #3'
    entries_schema: must-have-foo-bar-meta
    description: Must contain at least one file with { foo: bar } metadata  object
schemas:
  must-contain-readme:
    url: s3://bucket/must-contain-readme.json
  must-contain-readme-summarize-at-least-1byte:
    url: s3://bucket/must-contain-readme-summarize-at-least-1byte.json
  must-have-foo-bar-meta:
    url: s3://bucket/must-have-foo-bar-meta.json
```

**`s3://bucket/must-contain-readme.json`**

Requires a README

```json
{
  "type": "array",
  "contains": {
    "type": "object",
    "properties": {
      "logical_key": {
        "type": "string",
        "pattern": "^README\\.md$"
      }
    }
  }
}
```

**`s3://bucket/must-contain-readme-summarize-at-least-1byte.json`**

```json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "allOf": [
    {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "size": {
            "type": "number",
            "minimum": 1,
            "maximum": 100000
          }
        }
      },
      "minItems": 2,
      "maxItems": 4
    },
    {
      "type": "array",
      "contains": {
        "type": "object",
        "properties": {
          "logical_key": {
            "type": "string",
            "pattern": "^README\\.md$"
          }
        }
      }
    },
    {
      "type": "array",
      "contains": {
        "type": "object",
        "properties": {
          "logical_key": {
            "type": "string",
            "pattern": "^quilt_summarize\\.json$"
          }
        }
      }
    }
  ]
}
```

**`s3://bucket/must-have-foo-bar-meta.json`**

Requires `{ "foo": "bar" }` object as user specified metadata in README.md

```json
{
  "type": "array",
  "contains": {
    "type": "object",
    "properties": {
      "logical_key": {
        "type": "string",
        "pattern": "^README\\.md$"
      },
      "meta": {
        "type": "object",
        "properties": {
          "foo": {
            "type": "string",
            "pattern": "^bar$"
          }
        },
        "required": ["foo"]
      }
    }
  }
}
```

#### Cross-bucket package push (Quilt catalog)

In Quilt, S3 buckets are like git branches but for data. With `quilt3` you can `browse` any package and then `push` it to any bucket that you choose.

As a rule, cross-bucket pushes or "merges" reflect change in a package's lifecycle. For example, you might push a package from *my-staging-bucket* to *my-production-bucket* as it matures and becomes trusted.

The catalog's [Push to bucket](/quilt-platform-catalog-user/working-with-the-catalog) feature can be enabled by adding a `successors` property to the config. A *successor* is a destination bucket.

```yaml
version:
  base: "1"
workflows:
  dummy:
    name: Dummy
successors:
  s3://bucket1:
    title: Staging
    copy_data: false
  s3://bucket2:
    title: Production
```

If `copy_data` is `true` (the default), all package entries will be copied to the destination bucket. If `copy_data` is `false`, all entries will remain in their current locations.

### JSON Schema

* Quilt workflows support the [Draft 7 JSON Schema](https://json-schema.org/specification-links.html#draft-7).
* JSON schemas can be stored anywhere in any accessible Amazon S3 bucket. Provided the path to the file is accessible in `config.yml`, the schema will successfully validate your package metadata shape.

#### Default values

Quilt supports the [`default` keyword](https://json-schema.org/understanding-json-schema/reference/generic.html?highlight=default).

#### Auto-fill dates

If you wish to pre-populate dates in the Quilt catalog, you can use the custom keyword `dateformat` in your schemas. For example:

```json
{
    "type": "string",
    "format": "date",
    "dateformat": "yyyy-MM-dd"
}
```

The `dateformat` template follows [Unicode Technical Standard #35](https://www.unicode.org/reports/tr35/tr35-dates.html#Date_Field_Symbol_Table).

#### Arrays, tuples and enums

Quilt supports the [`array` data type](https://json-schema.org/understanding-json-schema/reference/array.html). You can use `array` if you need to define a list of metadata values for a metadata key. These elements can be of any type.

If the order in the list is not significant, use "arrays" (using `"items"` and `"anyOf"`):

```json
{
    "type": "array",
    "items": {
        "anyOf": [
            {
                "type": "string"
            },
            {
                "type": "number"
            }
        ]
    }
}
```

With this Schema you can create a list of metadata values such as: `["Any string A", 123, "Any string B"]` or `[123, "Any string", 456]`

If the order in the list is important and the list is fixed in length, then use "tuples" (using `"items"`, `"minItems"`, and `"maxItems"`):

```json
{
    "type": "array",
    "items": [
        {
            "type": "string"
        },
        {
            "type": "number"
        }
    ],
    "minItems": 2,
    "maxItems": 2
}
```

With this Schema you can create strictly ordered lists, such as `["Any string", 123]`.

An incorrect order will return an error `[123, "Any string"] // invalid`.

> Remember that you should define `"minItems"` and `"maxItems"` or `"minItems"` and `"additionalItems": false`, because "tuples" must have a fixed size.

Instead of letting users set any metadata value, you can define list of available options with `enum`:

```json
{
    "type": "array",
    "items": {
        "type": "string",
        "enum": ["Fixed 1", "Fixed 2"]
    }
}
```

With this Schema you can create a list of any length with predefined values, such as `["Fixed 1", "Fixed 2", "Fixed 1"]`.

```json
{
    "type": "array",
    "items": [
        "type": "string",
        "enum": ["Fixed 1", "Fixed 2"]
    ],
    "minItems": 1,
    "additionalItems": false,
}
```

With this Schema users are allowed to create tuples like `["Fixed 1"]` or `["Fixed 2"]`.

If you want to provide users with a list of predefined metadata values but additionally let them add any values outside of this list, you can use the `anyOf` keyword:

```json
{
    "type": "array",
    "items": {
        "anyOf": [
            {
                "type": "string"
                "enum": ["Fixed 1", "Fixed 2"]
            },
            {
                "type": "string"
            }
        ]
    }
}
```

Metadata lists such as `["Fixed 1", "Fixed 2"]`, `["Fixed 1", "Any string"]` or `["Any string 1", "Any string 2"]` are all valid.

In certain use cases you may want to define metadata lists that have first-ordered items of predefined values, and the rest are any other outside of the predefined values. Then you create tuples with `"additionalItems": true`:

```json
{
    "type": "array",
    "items": [
        "type": "string",
        "enum": ["Fixed 1", "Fixed 2"]
    ],
    "minItems": 1,
    "additionalItems": true,
}
```

With this Schema lists such as `["Fixed 1", "Any string", 123]` are valid but `["Any string", 123]` are invalid.

#### Example properties

The following examples show how you can specify complex `properties` such as `object`, `array`, and compound `enum` types.

**Objects**

```json
{
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "id": {
              "default": 123,
              "type": "number"
            },
            "name": {
              "default": "Optional default value",
              "type": "string"
            }
        }
    }
}
```

**Compound enums: arrays**

```json
{
    "type": "array",
    "enum": [
        [1, 2, 3],
        [3, 4, 5],
        [6, 7, 8]
    ]
}
```

**Compound enums: objects**

```json
{
    "type": "object",
    "enum": [
        {"id": 1},
        {"id": 2},
        {"id": 3}
    ]
}
```

**Compound enums: arrays and objects**

```json
{
    "type": "array",
    "enum": [
        ["miles", {
            "format": "12h"
        }],
        ["kilometers", {
            "format": "24h"
        }],
        {
            "name": "unspecified"
        }
    ]
}
```

This allows for flexible and extensible schema definition, and hence validation, of complex metadata schemas to any depth.

> Quilt currently uses the Draft 7 Json Schema where tuples are validated with `items`, and not `prefixItems`. The `prefixItems` keyword was added in Draft 2020-12, and is not currently supported.

### `config.yml` JSON Schema

See [workflows-config\_catalog-1.0.0.json](https://github.com/quiltdata/quilt/blob/master/shared/schemas/workflows-config_catalog-1.0.0.json) and [workflows-config-1.1.0.json](https://github.com/quiltdata/quilt/blob/master/shared/schemas/workflows-config-1.1.0.json).

### Known limitations

* Only [Draft 7 Json Schemas](https://json-schema.org/specification-links.html#draft-7) are supported
  * If a workflow schema includes a non-supported keyword, the user interface displays an `unknown keyword: <non-supported keyword>` error
* Schemas with [`$ref`](https://json-schema.org/draft-07/json-schema-core.html#rfc.section.8.3) are not supported
* Schemas must be in an S3 bucket for which the Quilt user has read permissions


# About the Catalog

The Quilt Catalog is the second half of Quilt. It provides an interface on top of your S3 bucket that brings Quilt features like data packages and search to the web.

[**For a hands-on demo, check out the public demo catalog**](https://open.quiltdata.com/b/quilt-example)**.**

Note that you can use the Quilt Python API without using the Quilt Catalog, but they are designed to work together.

### Brief tour

The Quilt Catalog provides a homepage for your data catalog, based on a `README.md` file that you can optionally create at the top of your bucket.

#### Browse

![Homepage](/files/-LfaqE2HhrTPKY0UMSEE)

The Catalog lets you navigate packages in the registry using the "Packages" tab.

![Packages tab](/files/-MMTjQkqCc1cM_T2t_QO)

You can also browse the underlying S3 objects using the "Bucket" tab.

![Files tab](/files/k9XCSKXi7b4ENdI9TUcN)

#### Search

Catalogs also enable you to search the contents of your bucket. We support both unstructured (e.g. "`San Francisco`") and structured with [Query String Queries](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl-query-string-query.html#query-string-syntax) (e.g. "`metadata_key: metadata_value`") search. Hits are previewed right in the search results.

![Search](/files/-MK0AFKkPUx6nbCc_OEX)

#### Upload package

You can upload a new package providing the name of the package, commit message, files, metadata, and [workflow](/workflows).

The name should have the format `namespace/package-name`.

The message needs to add notes on a new revision for this package.

Files are the content of your package.

The associated workflow contains the rules for validating your package.

The metadata can be added with the JSON editor for both packages and individual file entries within a package, represented as a key/value table with infinite nesting. If your workflow contains a JSON schema, you will have predefined key/value pairs based on the schema.

**JSON editor**

To add a new key/value field double click on an empty cell and type the key name, then press "Enter" or "Tab", or click outside of the cell. To change the value double click on that value.

Values can be strings, numbers, arrays, or objects. Every value that you type will be parsed as JSON.

> Limitations
>
> * References and compound types are not currently supported.

#### Push to bucket

You can push an existing data package from one S3 bucket to another. To use this feature consult the [Workflows](/workflows) page.

#### Compare revisions

To see what changed between two revisions of a package, click the double-arrow diff icon on the package revisions list. The diff view highlights added, removed, and modified entries between the selected revisions.

#### Summarize

Adding a `quilt_summarize.json` file to a data package (or S3 directory path) will enable content preview right on the landing page.

See [Visualization & dashboards documentation](https://docs.quilt.bio/quilt-platform-catalog-user/pages/TI1BMRExTH2qFUEjKhOJ#quilt_summarize.json) for details.

![Package landing page](/files/-M7Q92LUYckJ1gIvXqHB)

### Admin UI

The Quilt Catalog includes an Admin panel where you can manage users and buckets in your stack and customize the display of your Quilt Catalog. See [Admin UI docs](/quilt-platform-administrator/admin) for details.

[**To learn more, check out the public demo catalog**](https://open.quiltdata.com/b/quilt-example).


# Bucket Browsing

Every S3 bucket attached to Quilt has a "Bucket" tab in the Catalog that displays all files in the bucket.

![Files browser tab](/files/NTwcH295qtcgkP0RPOZh)

> If desired, [this tab can be hidden](/quilt-platform-administrator/preferences).

### Uploading and deleting files

You can upload files directly to a bucket by dragging them onto the file listing or using the **Add Files** button (including into a new subfolder). One or more files can also be deleted from the listing or via the **Organize** menu without leaving the Catalog. On versioning-enabled buckets, deletion adds a delete marker so prior versions remain available from packages; on unversioned buckets the object is permanently removed.

File and directory delete buttons are hidden by default. Administrators can enable them by setting `ui.actions.deleteObject` in the [Catalog configuration](/quilt-platform-administrator/preferences).

### Creating packages from bucket files

You can create packages directly from files already stored in S3 buckets without downloading and re-uploading them. Even without a workflow configuration file, users can create packages from files in the current bucket, either by selecting them directly or by using **Add Files from Bucket** when revising a package. To disable adding files from any bucket (including the current one), set `ui.sourceBuckets` to an empty dictionary `{}` in the [Catalog configuration](/quilt-platform-administrator/preferences).

Package creation uses the current bucket by default. With workflow configuration, only explicit `successors` are available as destinations. See [workflow configuration](/workflows#cross-bucket-package-push-quilt-catalog) for details.

![Create package](/files/3o1njEDOLACMwoDNVZ60)

### Bookmarks

To create a package that includes multiple files from different directories in a single S3 bucket, or even across different S3 buckets attached to Quilt, you can browse and create a "bookmark" of chosen files. Select files by checking the box and clicking "Add to bookmarks". You can also navigate to a specific file and bookmark an individual file by clicking "Add to bookmarks".

![Select files](/files/D8inFwJDKTZn7BU9SBPT)

![Add selected files to bookmarks](/files/4BShH0qRvvQnaSz89bPY)

Open the Bookmarks pane (listed in the User account menu) and optionally create a new package from the bookmarked files.

![Open bookmarks](/files/2PXqDzIStFB5YWHqISZG)

![Browse bookmarks](/files/HZnTVSoO79lfRr8Wpw2w)

### Text editor

Inline editing of plain text, Markdown, JSON and YAML file formats is supported.

![Edit button](/files/EapXhUX0NVMeboju8l9K)

New text files can be created individually in editable file formats. To create one, click the «kebab» menu (three vertical dots) located in the far-right, above the file browser. Choose a file name and format (the default is README.md), enter your content, and click save.

![Open menu](/files/ImnUfHb2Xayx87nm9TRx)

![Choose name](/files/6EPGhPAsLuTW8r02KkFF)

![Edit file](/files/Anmzkh0dMQn26toWCKpn)

### Copy URI button

Files and directories include a copy-URI action on the download button, making it easy to copy an `s3://` URI for use in scripts, notebooks, and CLI workflows.

### Working with Amazon S3 Glacier storage classes

Glacier storage classes are built for data archiving. Quilt is compatible with S3 bucket lifecycle rules that transition S3 objects to Glacier storage classes.

There are currently three types of S3 object archive storage class that work differently with the Quilt Catalog, `quilt3` CLI and Python API.

1. **S3 Glacier Instant Retrieval:** Objects in this storage class are available as normal in the Bucket and Packages tabs in the Quilt Catalog.
2. **S3 Glacier Flexible Retrieval (formerly S3 Glacier):** Objects are not immediately available and appear "grayed out" in the Catalog.
3. **S3 Glacier Deep Archive:** Objects are not immediately available and appear "grayed out" in the Catalog.

![Glacier S3 objects list
view](/files/wmzKHSCIcTp1PTffOApY)

Previewing a specific "glacierized" S3 object shows an "Object Archived" message explaining that the preview is unavailable until the object is rehydrated, along with a **Rehydrate** button.

![Glacier S3 object preview with Rehydrate
button](/files/3hw08SZ5K2JuDuzBSWZH)

#### Restoring archived objects from the Catalog

> New in Quilt Platform version 1.70

You can restore (rehydrate) archived Glacier and Deep Archive objects directly from the file preview in the Quilt Catalog — no AWS Console or CLI required. Click **Rehydrate** on the archived object's page, then choose a retrieval **tier** (Standard, Bulk, or Expedited) and a **duration** (1–90 days) for how long the restored copy stays downloadable:

![Rehydrate from Glacier dialog with retrieval tier and
duration](/files/81hI1Fz3RuACvGmTZJ2o)

Quilt issues the `s3:RestoreObject` request on your behalf and reads restore state back from the object's S3 metadata, so the page reflects whether a restore is in progress, complete, or expired. Rehydration is temporary — the object returns to archived storage after the chosen duration, and the archived object itself is never lost.

Managed read/write roles automatically include `s3:RestoreObject`. Admins can hide the **Rehydrate** control via the `ui.actions.restore` preference (see [Preferences](/quilt-platform-administrator/preferences)).

> The AWS Glacier service is rapidly evolving and may impact the functionality of the Quilt Catalog and/or API.


# Document Previews

The Quilt catalog renders previews of the following file types. Whenever possible, Quilt streams the smallest possible subset of the data needed to generate the preview.

Previews are supported for uncompressed files as well as for gzip archives (.gz).

### Plain text previews

Quilt can display any plaintext file format, including the following.

* Most programming languages, with syntax highlighting (.cpp, .json, .js, .py, .sh, .sql, etc.)
* Biological file formats (.bed, .cef, .gff, .fasta, .fastq, .sam, .pdbqt, .vcf, etc.)
* Text files (.csv, .md, .readme, .tsv, .txt, etc.)

#### Markdown rendering

> Changed in Quilt Platform version 1.70

Markdown file previews (.md) render using standard [GitHub-Flavored Markdown](https://github.github.com/gfm/) (GFM), a superset of [CommonMark](https://commonmark.org/). Idiosyncratic Pandoc/PHP-Markdown-Extra shortcuts are no longer supported; author Markdown against the CommonMark + GFM spec for consistent rendering.

### Chemical structures

The Quilt catalog uses the [NGL Viewer library](https://github.com/nglviewer/ngl) to render structures. By default, v3000 Molfiles are converted to v2000 by the JavaScript client for rendering.

The following file formats are supported:

* Mol files (.mol, .mol2, .sdf)
* .cif
* .ent
* .pdb

### Image previews

The Quilt Catalog uses a [Lambda function](https://github.com/quiltdata/quilt/tree/master/lambdas/thumbnail) to automatically generate thumbnail previews of common image formats and select microscopy image formats such as .bmp, .gif, .jpg, .jpeg, .png, .webp, .tif, .tiff (including `OME-TIFF`), and .czi.

8-bit, 16-bit, 32/64-bit integer (signed or unsigned), and floating-point pixel data are all supported. Quilt reduces multi-dimensional microscopy images to a single 2D preview: channels are laid out as a grid of per-channel previews, Z-stacks are flattened with a maximum-intensity projection, and time-series show their middle timepoint.

#### Display normalization

> Thumbnails are a visual aid, not a faithful reproduction of the original pixel values.

Previews of high-bit-depth and microscopy images are **contrast-stretched per image** to keep faint, low-contrast detail visible: each grayscale image — and each channel of a multi-channel image — is rescaled from its own range of pixel values, after clipping the most extreme values so a few hot or dead pixels don't flatten the rest. Color previews are stretched the same way, jointly across their channels so the colors aren't skewed.

As a result, displayed brightness does **not** represent absolute pixel intensity: it is not comparable between thumbnails, nor between the channels of a multi-channel montage. Ordinary 8-bit images (such as .jpg files) are already display-ready and are shown unchanged.

#### Limitations

* Very large images may fail to preview if they exceed memory or pixel limits.

### Binary and special file format previews

* AnnData (.h5ad) — annotated matrix metadata, with QC metrics for small files
* FCS Flow Cytometry files (.fcs)
* Media (.mp4, .webm, .flac, .m2t, .mp3, .mp4, .ogg, .ts, .tsa, .tsv, .wav)
* Jupyter notebooks (.ipynb)
* .parquet
* PDF (.pdf)
* PowerPoint (.pptx)
* Excel (.xls, .xlsx)

### Advanced: HTML rendering and Quilt Package File Server

The Quilt Catalog supports HTML and JavaScript in preview via iframes. By default, preview iframes do not have IAM permissions and are therefore unable to access private files in S3.

If you wish for your HTML to access data within the enclosing package or bucket (at the viewer's level of permissions) and/or use origin-aware Web APIs such as data storage/cookies, you must opt in to `Enable permissive HTML rendering` in [Bucket settings](/quilt-platform-administrator/admin#buckets). This explicitly allows cross-origin resource sharing (CORS).

> You should *only enable this feature for buckets where you implicitly trust* the contents of the HTML files.

Depending on the context where the HTML file is rendered (package vs bucket view), the iframe gets the following origin:

* Inside a package view with permissive rendering **enabled**: the origin is the **Quilt Package File Server**.
* Inside a bucket view with permissive rendering **enabled**: the origin is the AWS S3 bucket endpoint.
* With permissive rendering **disabled** (irrespective of package or bucket view): the resource is treated as being from a special origin that always fails the same-origin policy ([`allow-same-origin` iframe sandbox token](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe#sandbox) is not set).

> An important implication of same-origin policy is that the scripts executed under the same origin share LocalStorage data and cookies.

#### Allowing Forms and Popups

> New in Quilt Platform version 1.59.0 or higher

Enabling Permissive HTML now allows forms and popups to work from iframes.

#### Package view example with permissive rendering enabled

1. `report.html` is a file in a package that includes a publicly available JS library and a custom embedded script.
2. Opening `report.html` in a package view generates a new session `temporary-session-id`.
3. The file is served by the **Quilt Package File Server** under the `/temporary-session-id/report.html` path.
4. All relative media and scripts are rendered in the same iframe relative-path format:
   * `./img.jpg` is resolved to `/temporary-session-id/img.jpg`
   * `script.js` is resolved to `/temporary-session-id/script.js`
5. The `allow-same-origin` iframe sandbox token is enabled, the origin is the **Quilt Package File Server**, the LocalStorage API is **available**.

#### Bucket view example with permissive rendering enabled

1. `report.html` is a file in a bucket `example-bucket` that includes a publicly available JS library and custom embedded script.
2. When opening `report.html` in bucket view, it is served directly by S3 via a signed HTTPS URL, e.g. `https://example-bucket.s3.region.amazonaws.com/report.html?versionId=...&X-Amz-...`.
3. All relative media and scripts are rendered in the same iframe relative-path format:
   * `./img.jpg` is resolved to `/img.jpg`
   * `script.js` is resolved to `/script.js`
4. The `allow-same-origin` iframe sandbox token is **enabled**, the origin is the **S3 bucket endpoint** (e.g. `https://example-bucket.s3.region.amazonaws.com`), the LocalStorage API is **available**.

#### Example with permissive rendering disabled

1. `report.html` is a file in a bucket `example-bucket` that includes a publicly available JS library and custom embedded script.
2. When opening `report.html` in any view it is served directly by S3 via a signed HTTPS URL, e.g. `https://example-bucket.s3.region.amazonaws.com/report.html?versionId=...&X-Amz-...`.
3. All relative media and scripts are rendered in the same iframe relative-path format:
   * `./img.jpg` is resolved to `/img.jpg`
   * `script.js` is resolved to `/script.js`
4. The `allow-same-origin` iframe sandbox token is **disabled**, a virtual unique origin is used (always failing the same-origin policy), the LocalStorage API is **unavailable**.

#### Live packages

* [Dynamic visualizations; interactive IGV dashboard; Perspective datagrids with images](https://open.quiltdata.com/b/quilt-example/packages/examples/package-file-server)


# Packaging Engine

> This feature requires Quilt Platform version 1.58.0 or higher

## Overview

The Quilt Packaging Engine in the Quilt Platform allows administrators and developers to automate the process of creating Quilt packages from data stored in Amazon S3. It serves as a key component of Quilt's functionality as a Scientific Data Management System, enabling automated data ingestion and standardization. It currently consists of:

1. Admin Settings GUI to enable package creation based on notifications from:
   1. AWS Health Omics
   2. Nextflow workflows using the WRROC ([Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/)) format from [nf-prov](https://github.com/nextflow-io/nf-prov).
2. SQS queue that will process package descriptions
3. Documentation for creating custom EventBridge rules to invoke that queue

## Admin Settings

The simplest way to enable package creation is through the Admin Settings GUI, which supports the following built-in event sources:

![Admin Settings](/files/HQlYqDKpfTji74XjSD84)

### AWS Health Omics

When enabled, this will create a package from the `runOutputUri` provided in a `aws.omics` completion event. For example, if the `runOutputUri` is `s3://quilt-example/omics-quilt/3395667`, the package will be created in that same bucket with the name `omics-quilt/3395667`.

### Workflow Run RO-Crate

When enabled, this will create a package from the enclosing folder when an `ro-crate-metadata.json` file is written to a bucket that is already part of the stack.

[RO-Crate](https://www.researchobject.org/ro-crate/) is a metadata standard for describing research data. The Workflow Run working group adds three additional profiles, which are supported in the latest versions of [nf-prov](https://github.com/nextflow-io/nf-prov). You will need to explicitly configure `nf-prov` to use `wrroc`, by using a `nextflow.config` file [like this](https://github.com/famosab/wrrocmetatest):

```groovy
plugins {
 id 'nf-prov@1.4.0'
}

prov {
 enabled = true
 formats {
   wrroc {
     file = "${params.outdir}/ro-crate-metadata.json"
     overwrite = true
     agent {
       name = "John Doe"
       orcid = "https://orcid.org/0000-0000-0000-0000"
     }
      license = "https://spdx.org/licenses/MIT"
      profile = "provenance_run_crate"
   }
 }
}
```

Note that Research Objects identify people using an ORCID iD, which anyone can get for free at [the ORCID website](https://orcid.org/).

The package will be created in the same bucket as the `outdir`, with the package name inferred from the S3 key. For example, if the key is `my/s3/folder/ro-crate-metadata.json`, the package name will be `my_s3/folder`.

## Architecture

The Quilt Packaging Engine is built on top of the existing packaging lambdas used by the Quilt Platform, including the ability to parallelize creation of S3 Checksums for existing objects. We have exposed this functionality to customers via an SQS queue, which is invoked by the EventBridge rules created by the Admin Settings GUI.

### SQS Parameters

You can also send messages directly to the SQS queue, which is part of the Quilt stack. The queue information will be listed as `PackagerQueueArn` and `PackagerQueueUrl` under the `Outputs` tab in CloudFormation section of the AWS Console. The URL will be something like:

```
https://sqs.REGION.amazonaws.com/ACCOUNT_ID/stack-name-PackagerQueue-XXX
```

Where REGION and ACCOUNT\_ID will be the same as for the Quilt stack.

The body of the message is the stringified JSON of a package description. There is only one required parameter:

```json
{
  "source_prefix": "s3://data_bucket/source/folder/metadata.json"
}
```

This is assumed to be a folder if it ends in a `/`; otherwise, we will remove the last component of the path to get the folder. The contents of the folder will be used to create a package in the same bucket as the source folder, with the package name being inferred from the source URI.

Optionally, you can control the package name, metadata, and other settings by explicitly specifying any of the following fields:

```jsonc
{
  "source_prefix": "s3://data_bucket/source/folder/", // trailing '/' for folder
  "registry": "package_bucket", // may be the same as `data_bucket`
  "package_name": "prefix/suffix",
  "metadata": { "key": "value" }, // object (or metadata URI, but not both)
  "metadata_uri": "metadata.json", // S3 URI to read, relative or absolute
  "commit_message": "Commit message for the package revision", // string
  "workflow": "alpha", // name of a valid metadata workflow
}
```

The job will fail if you try to specify both `metadata` and `metadata_uri`.

### SendMessage API

If you have appropriate IAM permissions, and the SQS URL, you can send a message to the queue using the AWS SDK or the AWS CLI. Here is an example using the AWS CLI:

```bash
export QUEUE_URL=https://sqs.us-east-1.amazonaws.com/XXX/PackagerQueue-XXX
aws sqs send-message --queue-url $QUEUE_URL \
--message-body '{"source_prefix":"s3://data_bucket/source/folder/"}'
```

### Custom EventBridge Rules

EventBridge rules can be used to transform EventBridge events from any bus in your account into a conforming SQS message.

#### Example: Event-Driven Packaging (EDP)

[Event-Driven Packaging](/quilt-ecosystem-integrations/event-driven-packaging), currently in private preview, coalesces multiple S3 uploads into a single `package-objects-ready` event, which infers the appropriate top-level folder. When ready, it creates an event like this on its own EventBridge bus:

```json
{
  "version":"0",
  "id":"XXXXXXXXXXXXXX",
  "detail-type":"package-objects-ready",
  "source":"com.quiltdata.edp",
  "account":"XXX",
  "time":"2022-12-08T20:01:34Z",
  "region":"us-east-1",
  "resources":[
    "arn:aws:s3:::bucket-name"
  ],
  "detail":{
    "version":"0.1",
    "bucket":"bucket-name",
    "prefix":"prefix-path-1/prefix-path-2/"
  }
}
```

The following Python code creates an EventBridge rule that targets the packager queue when matching that event:

```python
import boto3
import json

# AWS Region and Account Details
region = "us-east-1"
account_id = "XXXXXXXXXXXX"
sqs_queue_name = "PackagerQueue-XXXXXXXXXXXX"
sqs_arn = f"arn:aws:sqs:{region}:{account_id}:{sqs_queue_name}"

# Initialize AWS Clients
eventbridge = boto3.client("events", region_name=region)

# Step 1: Create EventBridge Rule

event_pattern = {
    "source": ["com.quiltdata.edp"],
    "detail-type": ["package-objects-ready"]
}

response = eventbridge.put_rule(
    Name="RouteEDPEventsToSQS",
    EventPattern=json.dumps(event_pattern),
    State="ENABLED",
    Description="Routes package-objects-ready events to SQS",
)

print(f"EventBridge Rule Created: {response['RuleArn']}")

# Step 2: Attach the SQS Target with Input Transformation

targets = [
    {
        "Id": "SQS_PackagerQueue",
        "Arn": sqs_arn,
        "InputTransformer": {
            "InputPathsMap": {
                "bucket": "$.detail.bucket",
                "prefix": "$.detail.prefix"
            },
            "InputTemplate": """{
                \"source_prefix\": \"s3://<bucket>/<prefix>metadata.json\"
            }"""
        }
    }
]

response = eventbridge.put_targets(
    Rule="RouteEDPEventsToSQS",
    Targets=targets
)

print("SQS Target Attached to EventBridge Rule:", response)
```

## Caveats

1. The package creation process is asynchronous, so you may need to wait a few minutes before the package is available (longer if the source data is large).
2. If you send the same message multiple times before the folder is updated, it will not actually create a new revision, since the content hash will be the same. However, that would still waste computational cycles, so you should avoid doing so.


# Query

[Amazon Athena](https://aws.amazon.com/athena/) is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

The Catalog's Queries tab allows you to run Athena queries against your S3 buckets, and any other data sources your users have access to. There are prebuilt tables for packages and objects, and you can create your own tables and views. See, for example, [Tabulator](/quilt-platform-administrator/advanced/tabulator).

NOTE: This page describes how to use Athena for precise querying of specific tables and fields. For full-text searching using Elasticsearch, see the [Search](/quilt-platform-catalog-user/search) page.

### Basics

"Run query" executes the selected query and waits for the result.

![ui](/files/7IfNBidZeS4remCQ1oNK)

Individual users will also see their past queries, and easily re-run them.

### Example: query package-level metadata

Suppose we wish to find all packages produced by algorithm version 1.3 with a cell index of 5. As of Quilt Platform version 1.70, package-level metadata lives in the per-bucket [Iceberg `package_manifest` table](/quilt-platform-catalog-user/advanced/iceberg-tables), which replaced the old `*_packages-view` Athena view:

```sql
SELECT * FROM "YOUR-BUCKET_package_manifest"
-- extract and query package-level metadata
WHERE json_extract_scalar(metadata,
  '$.user_meta.nucmembsegmentationalgorithmversion') LIKE '1.3%'
AND json_array_contains(json_extract(metadata, '$.user_meta.cellindex'), '5');
```

### Example: query object-level metadata

Suppose we wish to find all .tiff files produced by algorithm version 1.3 with a cell index of 5. Object-level (file entry) metadata is now in the per-bucket [Iceberg `package_entry` table](/quilt-platform-catalog-user/advanced/iceberg-tables), which replaced the old `*_objects-view` Athena view:

```sql
SELECT * FROM "YOUR-BUCKET_package_entry"
WHERE substr(logical_key, -5) = '.tiff'
-- extract and query object-level metadata
AND json_extract_scalar(metadata,
  '$.user_meta.nucmembsegmentationalgorithmversion') LIKE '1.3%'
AND json_array_contains(json_extract(metadata, '$.user_meta.cellindex'), '5');
```

### Configuration

Athena queries saved from the AWS Console for a given workgroup will be available in the Quilt Catalog for all users to run.

Administrators can hide the "Queries" tab by setting `ui > nav > queries: false` ([learn more](/quilt-platform-administrator/preferences)).


# Quilt+ URIs

Every package and object in the Bucket and Packages views has a "CODE" pane, which contains code snippets that can be used to download and upload a package or object via either:

* Python (API)

  ![Python](/files/kDBzIOKMMjdjehjggV2X)
* CLI (shell commands)

  ![CLI](/files/e4vUWFc4gudB5Yy0MrDI)

In addition, Packages have a third tab that returns a Quilt+ URI:

* URI (identifier)

  ![URI](/files/zPfZK8VOmELbV3iPkbMb)

They all have a `copy` button that copies the code to the clipboard.

### Quilt+ URIs

Quilt+ URIs are a way to uniquely identify a package or sub-package (e.g., folder or entry) in the Quilt catalog, relative to an S3 bucket. For example:

`quilt+s3://quilt-example#package=akarve/cord19@e21682f00929661879633a5128aaa27cc7bc1e2973d49d4c868a90f9fad9f34b&path=CORD19.ipynb`

The URI above references a specific version of the `CORD19.ipynb` notebook in the `akarve/cord19` package of the `quilt-example` bucket.

#### Catalog Usage

URIs can be used to quickly navigate to a specific package or object from the Catalog. If your window is wide enough, there will be a "URI" button to the right of the search bar. Clicking this button will display a dialog where you can paste a URI and "Resolve" it to navigate to the package or object it references.

![Resolving URIs](/files/HPADpYPSulgcjakGDVZA)

**Redirects**

If you want to invoke a Quilt+ URI via a standard HTTP URL, you can use a redirect page from the Quilt catalog. For example:

`https://your-catalog-host/redir/quilt%2Bs3%3A%2F%2Fquilt-example%23package%3Dakarve%2Fcord19%40e21682f00929661879633a5128aaa27cc7bc1e2973d49d4c868a90f9fad9f34b%26path%3DCORD19.ipynb`.

#### Syntax

A Quilt+ URI contains the following components:

* `quilt+<protocol>`: The scheme of the URI. This always begins with `quilt+`. Currently the only supported protocol is `s3`.
* `<bucket>`: The name of the bucket containing the package, e.g. `quilt-example`.
* `#package=<package_name[specifier]>`: A fragment for the name of the package, e.g. `akarve/cord19`, plus an optional specifier. The specifier identifies a particular revision, using either `@<top_hash>` (e.g. `@e21682f00929661879633a5128aaa27cc7bc1e2973d49d4c868a90f9fad9f34b`) or `:<tag>` (defaults to `:latest` when omitted). You may not specify both a top\_hash and a tag.
* `&path=<path>`: An optional fragment after the package, specifying the path to a particular subpackage (i.e., folder or entry) within the package. This is always a relative path, e.g. `CORD19.ipynb` in the example.
* `&catalog=<catalog>`: An optional fragment specifying the DNS name of the catalog that generated the URI. This is used to help clients generate the human-readable URL for that package.


# Platform MCP Server

The **Quilt Platform MCP Server** lets AI assistants interact with your organization's data through natural language. Built on the open [Model Context Protocol](https://modelcontextprotocol.io/), it connects Claude, Cursor, and other MCP-compatible clients directly to your Quilt environment — so users can search, browse, read, create, and query data without leaving their AI workflow.

All actions respect your existing Quilt roles and permissions. Data never leaves your AWS environment.

## Capabilities

### Search

Ask your AI assistant to find packages or S3 objects by name, content, metadata, or any combination. Searches use Elasticsearch query syntax under the hood, so you can be as broad ("RNA-seq experiments") or specific (`ext:.parquet AND key:results/*`) as you like. Results are paginated automatically — just ask for more if the first page isn't enough.

### Packages

Your AI assistant can list packages in a bucket, browse a package's file tree, inspect revision metadata and history, create new packages from S3 objects or inline content, and patch existing packages by adding, updating, or removing entries — all without leaving the conversation. Workflows and user metadata are supported on create and update.

When you try to create a package that already exists, the assistant will not silently replace it: `package_create` refuses and points you to `package_patch` for incremental changes, and requires `overwrite=true` to replace the package outright. An overwrite reports an added / removed / kept entry diff against the previous revision.

### S3 Objects

List, read, inspect, download, and upload S3 objects. The assistant can read text, images, and binary files directly from `s3://` or `quilt+s3://` URIs, retrieve object metadata (size, content type, last modified), generate presigned download URLs you can open in a browser, and upload new content to S3.

### Athena

Run SQL queries against your data lake through Amazon Athena and get results back in the conversation. This works with both standard Athena tables and Quilt Tabulator tables — the assistant automatically has access to available databases and catalogs.

### Tabulator

List, create, rename, and manage Tabulator table definitions that control how package data is projected into queryable Athena tables.

### Utilities

The assistant can list your accessible buckets, generate shareable Quilt catalog links for any resource, and read platform configuration such as search syntax help and Athena setup details.

***

## Resources

The Platform MCP Server also exposes read-only resources that give AI assistants additional context about your environment:

| Resource          | Description                                             |
| ----------------- | ------------------------------------------------------- |
| **Search Syntax** | Elasticsearch query string syntax reference for search  |
| **Athena**        | Available databases, catalogs, and query configuration  |
| **Buckets**       | Accessible buckets with names, titles, and descriptions |
| **Current User**  | Identity and role of the authenticated user             |

***

## Getting Started

### Supported Clients

The Platform MCP Server works with any MCP-compatible AI client, including:

* **Claude.ai** (web)
* **Claude Code** (CLI)
* **Cursor** (desktop)
* **ChatGPT** (web)
* **Databricks** (web)
* **Benchling AI** (web)
* **OpenAI Codex** (desktop/IDE)
* **Any client** supporting the [Model Context Protocol](https://modelcontextprotocol.io/)

### Connecting Claude.ai

An Organization administrator adds Quilt as a connector:

1. Go to [Organization Settings -> Connectors](https://claude.ai/admin-settings/connectors)
2. Click **Add Custom Connector**
3. Enter your Connect Server URL: `https://<connect-host>/mcp/platform/mcp`

### Connecting Cursor and other desktop clients

Add the following to your MCP client configuration (in Cursor: **Settings -> MCP -> Add new global MCP server**):

```json
{
  "mcpServers": {
    "quilt": {
      "url": "https://<connect-host>/mcp/platform/mcp"
    }
  }
}
```

> Your administrator must include the client's custom scheme (e.g. `cursor://`) in `ConnectAllowedHosts` for the OAuth flow to complete.

### Connecting ChatGPT

> Requires Quilt **1.70 or later**.

In ChatGPT, go to **Settings -> Apps -> Create app** (Developer mode required). Set:

* **MCP Server URL:** `https://<connect-host>/mcp/platform/mcp`
* **Authentication:** `OAuth`
* **OIDC enabled:** on, with **OIDC scopes supported:** `platform`

Leave the OAuth endpoint fields on their auto-discovered values.

`chatgpt.com` must be in `ConnectAllowedHosts` (see [Connect.md](/quilt-platform-administrator/connect#connectallowedhosts-entry-formats)).

### Connecting Databricks

> Requires Quilt **1.70 or later**.

In the Databricks Catalog **HTTP connection** UI, fill in:

| Field             | Value                    |
| ----------------- | ------------------------ |
| Connection type   | `HTTP`                   |
| Is MCP connection | `true`                   |
| Host              | `https://<connect-host>` |
| Base path         | `/mcp/platform/mcp`      |

Databricks discovers the OAuth endpoints from `/.well-known/oauth-authorization-server` and uses `https://<region>.cloud.databricks.com/api/2.0/http/oauth/redirect` as its redirect URI (the workspace region determines the exact host).

`.cloud.databricks.com` must be in `ConnectAllowedHosts` so DCR accepts that redirect URI (see [Connect.md](/quilt-platform-administrator/connect#connectallowedhosts-entry-formats)). Quilt Connect already emits the `:443`-explicit metadata Databricks requires — see [Connect.md OAuth Metadata](/quilt-platform-administrator/connect#oauth-metadata) for why.

> **Serverless egress caveat.** Databricks Apps and serving endpoints run on a serverless network plane that blocks outbound traffic by default. Two distinct outbound legs need egress, and a Databricks **account admin** must allow both in the serverless network policy attached to the workspace:
>
> * `<connect-host>` — the **tool-calling** leg. Without it, tool listing fails with `Access to <connect-host> is denied because of serverless network policy` (the host named in this error).
> * `<catalog-host>` — the **OAuth authorize-redirect** leg. The authorize endpoint is cross-served: `<connect-host>/connect/authorize` returns a 302 to the catalog UI on `<catalog-host>`, which the client follows during sign-in. This leg does not surface the error above, so allowing only `<connect-host>` is not enough.
>
> This is not a per-connection or per-app setting; the connection creator cannot fix it.
>
> Confirm the block from a Databricks SQL warehouse:
>
> ```sql
> SELECT * FROM system.access.outbound_network
> WHERE event_time >= CURRENT_TIMESTAMP() - INTERVAL 2 HOUR
> ORDER BY event_time DESC;
> ```
>
> See the Databricks docs: [serverless network policies overview](https://docs.databricks.com/aws/en/security/network/serverless-network-security/network-policies) and [managing serverless network policies](https://docs.databricks.com/aws/en/security/network/serverless-network-security/manage-network-policies).

### Connecting Benchling AI

Benchling AI's [AI Connectors](https://help.benchling.com/hc/en-us/articles/42715696739341-Configure-AI-Connectors-for-Benchling-AI) let Chat and Deep Research query external MCP servers — including the Quilt Platform MCP Server — so scientists can reach Quilt data without leaving Benchling. Chat or Deep Research must be enabled on your tenant.

A Benchling **tenant admin** adds Quilt as a Custom AI Connector:

1. Go to **Tenant admin console -> Settings -> AI Connectors**
2. Click **Add AI Connector**
3. Complete the configuration:
   * **Name:** `Quilt` (this is what users see)
   * **Server:** `https://<connect-host>/mcp/platform/mcp`
   * **Type:** `HTTP`
4. Review the tools exposed by the server and select which ones users may access (at least one must be enabled)
5. Click **Save**

Each Benchling user then enables the connector once:

1. In the navigation bar, click **AI**, then the **Settings** icon
2. Open the **AI Connectors** tab and click **Connect** next to Quilt
3. Complete the Quilt OAuth flow in the window that opens (see [User Authorization](#user-authorization) below)
4. Return to Benchling to finalize the connector

> Benchling completes its OAuth handshake from `https://<tenant>.benchling.com/...`, so `.benchling.com` must be in `ConnectAllowedHosts` (see [Connect.md](/quilt-platform-administrator/connect#connectallowedhosts-entry-formats)).

### Connecting OpenAI Codex

> Requires Quilt **1.70 or later**.

In the [Codex](https://developers.openai.com/codex/) desktop app or IDE extension, open **Settings -> MCP servers -> + Add server**, choose **Streamable HTTP**, and enter the URL:

```
https://<connect-host>/mcp/platform/mcp
```

Leave **Bearer token env var**, **Headers**, and **Headers from environment variables** blank to use OAuth, then **Save** and **Authenticate**.

You can also configure it via the `codex mcp add` CLI or by editing `~/.codex/config.toml` directly; see [Codex MCP configuration](https://developers.openai.com/codex/mcp).

Codex starts the OAuth flow on first connect and opens a browser to the Quilt authorization page.

### User Authorization

Each user must authorize their MCP connection once:

**Web clients (e.g. Claude.ai):**

1. Log in to your Quilt stack as usual (e.g. via Okta SSO)
2. Go to [Customize -> Connectors](https://claude.ai/customize/connectors)
3. Click **Connect**

**Desktop clients (e.g. Cursor):** the OAuth flow starts automatically the first time the client connects to the MCP server.

In both cases, you will see the Quilt authorization page at `/connect/authorize`, showing the name of the AI client and what it is requesting access to. Click **Continue** to grant access or **Cancel** to deny it.

After authorizing, the AI assistant receives a session token scoped to your Quilt user — it cannot access data beyond what your assigned Quilt role permits. You do not need to re-authorize the same client unless your session expires or the Quilt stack is redeployed.

Once authenticated, you may also need to authorize individual tools when used. You can pre-authorize them by clicking **Configure** on the connector page.

![Quilt Connect Server](/files/bNNl0VlGcZfIWMHRIrv7)

![Quilt MCP Configuration](/files/SULvgKPsH6Y0dxppDjWg)

### Headless Access with API Keys

For automation, AWS-side services, and other non-interactive clients that cannot complete an OAuth flow, the MCP server also accepts a [Quilt API key](/quilt-python-sdk/api-reference/authentication#api-keys) as a bearer token:

```http
POST https://<connect-host>/mcp/platform/mcp
Authorization: Bearer qk_...
```

For stdio-mode MCP clients, set the key in the environment instead:

```bash
export QUILT_API_KEY=qk_...
```

The MCP request executes under the API key owner's role and bucket permissions, exactly as an OAuth-issued session would. Generate, list, and revoke keys via `quilt3.api_keys` (see the [Authentication guide](/quilt-python-sdk/api-reference/authentication)).

***

## Administrator Reference

The Platform MCP Server runs behind [Quilt Connect Server](/quilt-platform-administrator/connect), which handles OAuth authentication, session tokens, and request routing within your AWS environment. See the [Quilt Connect](/quilt-platform-administrator/connect) page for CloudFormation parameters, DNS configuration, and IP allowlisting.


# Qurator Omni

`Qurator Omni` is an always-available AI assistant embedded directly into the Quilt web catalog. It allows users to interact with S3 buckets and search functionality through natural language. Qurator Omni leverages advanced models like Claude, integrated via Amazon Bedrock, enabling users to query, retrieve, and summarize data instead of having to click through the GUI.

Qurator Omni is designed to streamline interaction with Quilt data by offering a conversational interface. Instead of navigating through various tabs and menus, or learning complicated search syntax, users can opt into the Qurator feature to ask complex questions in plain language and receive structured, actionable responses.

For example, users can ask for summaries of research on topics like "melanoma" or request key insights from a specific dataset.

#### Key Features

* **Natural Language Queries**: Ask complex questions like “What are the latest asthma treatments?” or “Summarize research on BRCA1 mutations.”
* **Instant Summaries**: Quickly digest scientific papers, datasets, or reports without reading everything.
* **Platform Tools via MCP**: Search packages and S3 objects, browse and create packages, read objects, run Athena SQL, and manage Tabulator tables — all through the same [Quilt Platform MCP Server](/quilt-platform-catalog-user/mcp-server) used by external MCP clients. Available in-catalog without enabling Quilt Connect; tools execute under the user's catalog session and respect existing role and bucket permissions.
* **Fine-Grained Permissions with RAG**: Ensure Retrieval-Augmented Generation only queries the data you're authorized to access, ensuring compliance with strict organizational policies
* **Secure Cloud Environment**: Work within your private AWS cloud, ensuring data stays secure while using state-of-the-art AI models.

#### Developer Tools

The Developer Tools menu (upper right of the Qurator chat window) provides:

* **Swappable Models**: Override the default Bedrock model for the current session by pasting a Bedrock Model ID or Inference Profile ID. The model must be enabled in the same region as your Quilt stack and support text, document, and image inputs.
* **Session Recordings**: Record a portion of a Qurator session and download (or clear) the resulting JSON log. Useful for tuning or debugging prompts and capturing structured results.

#### Connector Status

Qurator's chat input shows the live connection status of each tool backend (e.g. the Platform MCP Server). When a backend is unhealthy the input is gated and inline actions appear in the helper-text region:

* `connecting…` / `reconnecting…` — auto-progressing, no action required.
* `couldn't connect` — click **reconnect** to retry, or **continue without** to proceed with reduced tool access for the rest of the conversation. The latter dismisses the error and moves the connector to `unavailable`.
* `unavailable` — sticky; click **reconnect** to try again at any time.

![Qurator platform tools connector ready](/files/3iqBEw5qKuzHybSKFRH4)

### Getting Started

To enable Qurator Omni:

1. **Opt-In to Qurator**:
   * Install Release 1.55 or later of the Quilt Platform CloudFormation template.
   * Set the `Qurator` parameter to `Enabled` in the CloudFormation template to enable the Qurator chatbot.
2. **Configure Claude Model**:
   * Log in to the Amazon Bedrock console.
   * Ensure that the Claude Sonnet 4.5 (`us.anthropic.claude-sonnet-4-5-20250929-v1:0`) inference profile is available in the same region as your Quilt deployment. Check [Model support by AWS Region](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html) for details.
   * Enable the model by configuring it within your Bedrock environment.
   * Optionally, set the `QuratorDefaultModel` stack parameter to a different Bedrock model ID to override the built-in default.
   * Carefully monitor the model's cost implications. The Claude model is charged based on usage, so ensure that you have the necessary budget allocated. Initial estimates are roughly a penny per page for complex documents.
3. **Start Using Qurator**:
   * Once activated, the Qurator chatbot will appear in the Quilt web catalog interface.
   * Click the Qurator icon on the bottom right of the screen to open the chat interface. ![qurator icon](/files/ohAu9yFY8nfCiVRDgJj9)
   * You can begin by typing questions into the chat interface. For example, queries like *“What are the key findings on small molecule delivery?”* will prompt Qurator to search for relevant data and present a summarized overview.

#### Example Use Cases

* **Search**: *“What are the latest papers on melanoma?”*\
  Qurator will search through the Quilt catalog using Elastic Search and retrieve the most relevant data.
* **Summarize**: *“Summarize the key points of this BRCA1 research.”*\
  After selecting a specific document, Qurator will generate a clear, useful summary of the paper.
* **Quick Scan**: *“List some of the authors doing breast cancer research?”*\
  Qurator Omni will list authors and their contributions based on Quilt’s indexed datasets.

#### Key Benefits

* **Enhanced Productivity**: Eliminate the need for manual search navigation, enabling faster access to critical information.
* **Improved Insights**: Gain deeper insights from large datasets with automatic summaries.
* **Streamlined Collaboration**: Leveraging AI chat to provide background and context when working across disciplines.


# Search

Each Quilt stack includes an Elasticsearch cluster that indexes objects and packages as documents. The objects in Amazon S3 buckets connected to Quilt are synchronized to an Elasticsearch cluster, which provides Quilt's search and package listing features.

NOTE: This page is about full-text searching using Elasticsearch. For precise querying of specific fields, see the [Queries](/quilt-platform-catalog-user/query) page.

### Indexing

Quilt maintains a near-realtime index of the objects in your S3 bucket in Elasticsearch. Each bucket corresponds to one or more Elasticsearch indexes. As objects are mutated in S3, Quilt uses an event-driven system (via SNS and SQS) to update Elasticsearch.

There are two types of indexing in Quilt:

* *shallow* indexing includes object metadata (such as the file name and size)
* *deep* indexing includes object contents. Quilt supports deep indexing for the following file extensions:
  * .csv, .html, .json, .md, .rmd, .rst, .tab, .txt, .tsv (plain-text formats)
  * .fcs (FlowJo)
  * .ipynb (Jupyter notebooks)
  * .parquet
  * .pdf
  * .pptx
  * .xls, .xlsx

#### Search page

The search page in the catalog, accessible from the search button in the top menu bar, provides a convenient way for searching objects and packages in an Amazon S3 bucket.

NOTE: Quilt uses Elasticsearch 7.10 [query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl-query-string-query.html#query-string-syntax).

The following are all valid search parameters:

**Object-specific Fields**

| Name            | Type      | Description          | Example                                    |
| --------------- | --------- | -------------------- | ------------------------------------------ |
| `content`       | `text`    | Object content       | `content:Hello`                            |
| `ext`           | `keyword` | Object extension     | `ext:*.fastq.gz`                           |
| `key`           | `keyword` | Object key           | `key:phase*`                               |
| `key_text`      | `text`    | Analyzed object key  | `key:"phase"`                              |
| `last_modified` | `date`    | Last modified date   | `last_modified:[2022-02-04 TO 2022-02-20]` |
| `size`          | `long`    | Object size in bytes | `size:>=4096`                              |
| `version_id`    | `keyword` | Object version id    | `version_id:t.LVVCx*`                      |

**Package-specific Fields**

All the package metadata is indexed in ES as three different types of documents:

* Manifests (`mnfst`) contain package metadata, including the hash, comment, and workflow information.
* Pointers (`ptr`) represent named package revisions, and associate Manifests with names and tags.
* Entries (`entry`) represent individual objects in the package with their metadata.

Top-level hits displayed by the search page are always Pointers, merged with the corresponding Manifest and matching Entries. A Pointer is considered a hit if it matches the search query, or if any of its associated documents, i.e. Manifest or Entries (either metadata or contents), match the search query.

| Name                            | Type      | Description                                                                    | Example                                                         |
| ------------------------------- | --------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------- |
| `ptr_name`                      | `keyword` | Package name                                                                   | `ptr_name:examples\/metadata`                                   |
| `ptr_name.text`                 | `text`    | Analyzed version of the above                                                  | `ptr_name.text:examples`                                        |
| `ptr_tag`                       | `keyword` | Package revision tag in S3; either "latest" or a timestamp (e.g. "1741661321") | `ptr_tag:latest`                                                |
| `ptr_last_modified`             | `date`    | Package revision last modified date                                            | `ptr_last_modified:[2022-02-04 TO 2022-02-20]`                  |
| `mnfst_hash`                    | `keyword` | Package manifest hash                                                          | `mnfst_hash:3192ac1*`                                           |
| `mnfst_stats.total_bytes`       | `long`    | Package total bytes                                                            | `mnfst_stats.total_bytes:>1000000`                              |
| `mnfst_stats.total_files`       | `long`    | Package total files                                                            | `mnfst_stats.total_files:<100`                                  |
| `mnfst_metadata`                | `text`    | Package metadata                                                               | `mnfst_metadata:dapi`                                           |
| `mnfst_message`                 | `text`    | Commit message / comment                                                       | `mnfst_message:TODO`                                            |
| `mnfst_workflow.id`             | `keyword` | Package workflow ID                                                            | `mnfst_workflow.id:verify-metadata`                             |
| `entry_lk`                      | `keyword` | Entry logical key                                                              | `entry_lk:examples\/metadata\/example.csv`                      |
| `entry_lk.text`                 | `text`    | Analyzed version of the above                                                  | `entry_lk.text:example`                                         |
| `entry_pk`                      | `keyword` | Entry physical key                                                             | `entry_pk:*example.csv`                                         |
| `entry_pk.text`                 | `text`    | Analyzed version of the above                                                  | `entry_pk.text:example`                                         |
| `entry_pk_parsed.s3.bucket`     | `keyword` | S3 bucket of the entry                                                         | `entry_pk_parsed.s3.bucket:my-bucket`                           |
| `entry_pk_parsed.s3.key`        | `keyword` | S3 key of the entry                                                            | `entry_pk_parsed.s3.key:example.csv`                            |
| `entry_pk_parsed.s3.key.text`   | `text`    | Analyzed version of the above                                                  | `entry_pk_parsed.s3.key.text:example`                           |
| `entry_pk_parsed.s3.version_id` | `keyword` | S3 version ID of the entry                                                     | `entry_pk_parsed.s3.version_id:abc123*`                         |
| `entry_size`                    | `long`    | Entry size in bytes                                                            | `entry_size:>1000000`                                           |
| `entry_hash.type`               | `keyword` | Entry hash type                                                                | `entry_hash.type:sha2-256-chunked`                              |
| `entry_hash.value`              | `keyword` | Entry hash value                                                               | `entry_hash.value:T12mNNNsyfzayKFQQEXI6Ichf8AtuMbhw5c0oPg7fTo=` |
| `entry_metadata`                | `text`    | Entry metadata                                                                 | `entry_metadata:example`                                        |

**Logical operators and grouping**

| Syntax     | Description                                    | Example             |
| ---------- | ---------------------------------------------- | ------------------- |
| `AND`      | Conjunction                                    | `a AND b`           |
| `OR`       | Disjunction                                    | `a OR b`            |
| `NOT`      | Negation                                       | `NOT a`             |
| `_exists_` | Matches any non-null value for the given field | `_exists_: content` |
| `()`       | Group terms                                    | `(a AND b) NOT c`   |

**Wildcard and regular expressions**

| Syntax | Description                                                    | Example              |
| ------ | -------------------------------------------------------------- | -------------------- |
| `*`    | Zero or more characters, avoid leading `*` (slows performance) | `ext:config.y*ml`    |
| `?`    | Exactly one character                                          | `ext:React.?sx`      |
| `//`   | Regular expression (slows performance)                         | `content:/lmnb[12]/` |

#### ELASTICSEARCH tab

When you click into a specific bucket, you can access the Elasticsearch tab to run more complex queries. The Elasticsearch tab provides a more powerful search interface than the search bar, allowing you to specify the Elasticsearch index and query parameters.

![catalog-es-queries-default](/files/-MbJKX7ViWHofX6daNwC)

Quilt Elasticsearch queries support the following keys:

* `index` — comma-separated list of indexes to search ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/multi-index.html))
* `filter_path` — to reducing response nesting, ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/common-options.html#common-options-response-filtering))
* `_source` — boolean that adds or removes the `_source` field, or a list of fields to return ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/search-request-source-filtering.html))
* `size` — limits the number of hits ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/search-uri-request.html))
* `from` — starting offset for pagination ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/search-uri-request.html))
* `body` — the search query body as a JSON dictionary ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/search-request-body.html))

#### Secure Search

Secure Search adds object-level permission enforcement to search results. When enabled, it filters Elasticsearch hits by verifying the user's actual S3 permissions using HEAD requests. This ensures that users only see results they are authorized to access, providing stronger restrictions beyond just bucket-level indices.


# Visualization & Dashboards

Quilt packages are not only reproducible units of data and metadata, but units of *reporting*. You can use the following features to include interactive visualizations and light applications inside of packages.

Importantly, relative references to data are resolved relative to the parent package. This means that all of your reports are backed by immutable, versioned data, providing a common frame of reference that is lacking in BI applications that read from fast-moving databases and file systems.

In addition to rendering a wide variety of images, binary files, and text files, the Quilt catalog supports the following libraries for visualization and exploration:

* [Vega](#vega-and-vega-lite)
* [Vega-lite](#vega-and-vega-lite)
* [ECharts](#echarts)
* [Voila](#voila) (Developer preview)
* [Perspective](#perspective)
* [IGV](#integrative-genomics-viewer-igv)

The above systems provide you with hundreds of charts out of the box.

## `quilt_summarize.json`

`quilt_summarize.json` is a configuration file that renders one or more dashboard elements in both Bucket view and Packages view. The contents of `quilt_summarize.json` are a JSON array of files that you wish to preview in the catalog. Each file may be represented as a string or, if you wish to provide more configuration, as an object.

The simplest summary is a list of relative paths to files that you wish to preview:

```json
// quilt_summarize.json
[
  "file1.json",
  "file2.csv",
  "file3.ipynb"
]
```

By default, each list element renders in its own row.

![](/files/1tS2guTI0I8CgDryVRzx)

For multi-column layouts, you can provide an array instead of a string for a given row:

```json
// quilt_summarize.json
[
  "file1.json",
  [{
    "path": "file2.csv",
    "width": "200px"
  }, {
    "path": "file3.ipynb",
    "title": "Scientific notebook",
    "description": "[See docs](https://docs.com)"
  }]
]
```

![](/files/7P3Jweml8ZtBFriYXzTb)

Each element of an array in `quilt_summarize.json` can either be a path string or an object with one or more of the following properties:

* `path` - file path relative to `quilt_summarize.json`
* `title` - title rendered instead of file path
* `description` - description in markdown format
* `expand` - Display the file (`true`) or display a preview in an expandable box (`false`, default)
* `width` - column width either in pixels or ratio (default is ratio `1`)
* `types` - a list of render types (at present only singleton lists are supported):
  * `["echarts"]` to render JSON as an EChart
  * `["perspective"]` to render tabular data (csv, xlsx etc.) with Perspective
  * `["igv"]` to render JSON with Integrative Genomics Viewer
  * `["voila"]` to render a Jupyter notebook as an interactive Voila dashboard
  * `["html"]` to render HTML in iframes. See also [Advanced HTML rendering](/quilt-platform-catalog-user/preview#advanced-html-rendering-and-quilt-package-file-server)
  * `["text"]` to render anything as text with syntax highlighting

If you need to control the height of an element (useful for Voila dashboards), use the following extended syntax:

```json
// quilt_summarize.json
[
  {
    "path": "file1.json",
    "types": [
      {
        "name": "echarts",
        "style": { "height": "1000px" }
      }
    ]
  }
]
```

At present `height` is the only supported `style` element.

> Limitations:
>
> * Objects linked via `quilt_summarize.json` are always the **latest** version, even when browsing an older package version.
> * Object titles and image thumbnails link to the file view, even in the package view.

## Images

If your Amazon S3 bucket contains images, by default the Quilt Catalog displays a preview of those images *before* any `quilt_summarize.json`-referenced files.

In the **Overview** tab, the Catalog parses the entire Amazon S3 bucket contents and displays thumbnail image previews in a paginated grid (25 per page by default) of all [supported image types](/quilt-platform-catalog-user/preview#binary-and-special-file-format-previews).

> To hide this block, use the `gallery` field in your [bucket preferences](/quilt-platform-administrator/preferences) file.

In the **Bucket** tab, the Catalog displays thumbnail image previews in a similarly paginated grid but *only from the current directory viewed*.

In the **Packages** tab, when a specific package has been opened the Catalog displays thumbnail image previews in a similarly paginated grid but *only those image files in the selected package*.

## Vega and Vega-lite

The Quilt catalog uses [vega-embed](https://github.com/vega/vega-embed) to render [vega](https://vega.github.io/vega/) and [vega-lite](https://vega.github.io/vega-lite/) visualizations. See [package.json](https://github.com/quiltdata/quilt/blob/master/catalog/package.json) for specific library versions and compatibility.

In order to display a Vega or Vega-Lite visualization, simply reference a JSON file with a library-compatible schema in a JSON file as follows in your `quilt_summarize.json` file:

```json
[
    "relative/path/to/my/vega.json",
    "optionally/some/other/file.csv"
]
```

For both Vega and Vega Lite you may specify relative paths to package files as data sources and the Quilt catalog correctly resolves them. Vega treats any data source as JSON by default. If you wish to use a different format than JSON, please [specify the file type](https://vega.github.io/vega-lite/docs/data.html#format). For example:

```json
{
  "data": {
    "url": "./datasource.csv",
    "format": {
      "type": "csv"
    }
}
```

### Altair

The easiest way to create Vega-lite visualizations for Quilt packages is with [Altair](https://altair-viz.github.io/).

Here's a simple example:

```python
import pandas as pd
from numpy import random
import altair as alt

# Create Dataframe with two columns of random values
scores = random.randint(60, 100, size=5)
densities = random.random_sample(5)
df = pd.DataFrame({'score': scores, 'density': densities})

# Create Chart with two Quantitative axes

alt.Chart(df).mark_area(
    color="gray",
    opacity=.2
).encode(
    x="score:Q",
    y='density:Q',
    tooltip=['count(score):Q']
).save("vega.json")
```

> To create plots that directly embed a dataset with more than 5000 rows (a large dataset), you will encounter a `MaxRowsError`. You can get around this error in [several different ways](https://altair-viz.github.io/user_guide/faq.html#maxrowserror-how-can-i-plot-large-datasets)

### Live packages

* [Interactive map of California with slider scale](https://open.quiltdata.com/b/quilt-example/packages/akarve/reef-check)
* [Interactive map of 2015 United States by-county smoking & poverty data](https://open.quiltdata.com/b/quilt-example/packages/robnewman/us_county_smoking_vs_poverty)

## ECharts

To render an [EChart](https://echarts.apache.org/), you provide a JSON file (a dictionary that specifies the ECharts [option parameter](https://echarts.apache.org/en/option.html)) and you set the `"types"` property to `[ "echarts" ]`.

```json
// quilt_summarize.json
[
  {
    "path": "echarts-option-file.json",
    "title": "Awesome line chart",
    "types": ["echarts"]
  }
]
```

### Example

The following example is a [simple line chart from the ECharts documentation](https://echarts.apache.org/examples/en/editor.html?c=line-simple).

```json
// echarts.json
{
  "dataset": {
    "source": [
      ["Mon", 150],
      ["Tue", 230],
      ["Wed", 224],
      ["Thu", 218],
      ["Fri", 135],
      ["Sat", 147],
      ["Sun", 250]
    ]
  },
  "xAxis": {
    "type": "category"
  },
  "yAxis": {
    "type": "value"
  },
  "series": [
    {
      "type": "line"
    }
  ]
}
```

![](/files/hCmawFaZo5TIFgv9dGaU)

### External files

As with Vega, you can provide either a relative path or URL to the dataset file.

```json
// echarts.json
{
  "dataset": {
    "source": "./dataset.csv"
  },
  "xAxis": {
    "type": "category"
  },
  "yAxis": {
    "type": "value"
  },
  "series": [
    {
      "type": "line"
    }
  ]
}
```

Relative paths are resolved relative to your echarts.json file and relative to the parent package.

### Limitations

At present, ECharts in Quilt does not support custom JavaScript. You are therefore limited to JSON types (numbers, strings, objects, arrays, etc.). Functions like [`symbolSize`](https://echarts.apache.org/en/option.html#series-scatter.symbolSize) are not available.

### Live packages

* [Various EChart types](https://open.quiltdata.com/b/quilt-example/packages/examples/echarts)

## Voila

*This feature is a Developer preview, details are subject to change.*

Enterprise deployments of Quilt support interactive Jupyter notebooks with [Voilà](https://github.com/voila-dashboards/voila).

In brief, a Voila dashboard version of your notebook will display all of the output cells and none of the input cells from the underlying notebook. This enables you to create interactive, Jupyter-driven apps for your Quilt catalog users.

The Voila libraries execute a remote Jupyter Kernel and stream the results to the browser with tornado. Jupyter kernels run on a single EC2 instance (`t3.small` by default) in Linux containers that have network access but do not have access to persistent storage. The catalog users's AWS credentials are passed to Jupyter kernel as [environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list).

### Environment variables

When you have a Voila dashboard inside of a Quilt package, you may wish to reference files *in the current package revision*. The Quilt catalog sets the following environment variables and passes them to the Voila kernel:

* `QUILT_PKG_BUCKET`
* `QUILT_PKG_NAME`
* `QUILT_PKG_TOP_HASH`

You can access these variables in Python and browse the package:

```python
import io
import os

import pandas as pd
import quilt3 as q3
# https://open.quiltdata.com/b/allencell/packages/aics/data_handoff_4dn/tree/260c3a46581a324e3a495570886e07b62cb4ff54f20b334c5d73a5a370e678c1/

bucket = os.environ.get("QUILT_PKG_BUCKET") or "allencell"
handle = os.environ.get("QUILT_PKG_NAME") or "aics/data_handoff_4dn"
top_hash = os.environ.get("QUILT_PKG_TOP_HASH") or "260c3a46581a324e3a495570886e07b62cb4ff54f20b334c5d73a5a370e678c1"

pkg = q3.Package.browse(handle, registry=f"s3://{bucket}", top_hash=top_hash)
# Read data.csv from the current package from Voila
df = pkg["metadata.csv"].deserialize()
```

### Included Python packages

By default, Quilt Voila containers provide the following modules:

```
altair
bqplot
ipykernel
ipyvolume
ipywidgets
pandas
perspective-python
PyYAML
quilt3
scipy
```

## Perspective

Quilt renders tabular data formats into a [Perspective](https://perspective.finos.org/) Datagrid, including the following file extensions: .csv, .xls, .xlsx, .jsonl, .parquet, and .tsv.

For speed, Quilt loads the first few rows stored in S3. Click Load More to fetch up to about 6MB of zipped data. To see the entire file contents for large files, download the file (lower left).

### Filter, plot, pivot

Click Filter and Plot to open the side drawer. Drag and drop columns from the sidebar to Group By, Split By, Order By, and Where to pivot, filter, and more.

![](/files/6CFGvPU3cPQ6pFEf6mOP)

Select from a variety for visualizations by clicking the upper left menu that initially displays "Datgrid".

![](/files/8EsLS7ZGDQVuwv9AQSCw)

Click Toggle Theme to use a fixed-width font (useful for comparing strings).

Use the controls along the bottom to reset, download, copy, resize the grid, and more.

### Saving and configuring the datagrid

To open the drawer by default, set the `config.settings` property in `quilt_summarize.json` as follows:

```json
// quilt_summarize.json
[
  {
    "path": "file1.csv",
    "types": [
      {
        "name": "perspective",
        "config": {
          "settings": true
        }
      }
    ]
  }
]
```

You can save the state of the datagrid, as shown below. To restore a saved datagrid use the `config` property of `quilt_summarize.json`:

![](/files/p0Vt2fQy91oQPfsauc0y)

All filters and columns will be restored:

```json
// quilt_summarize.json
[
  {
    "path": "file1.csv",
    "types": [
      {
        "name": "perspective",
        "config": {
          "columns": ["YOUR_COLUMN_0", "YOUR_COLUMN_1"],
          "group_by": ["YOUR_COLUMN_1"],
          "settings": true,
          "theme": "Material Light Mono"
        }
      }
    ]
  }
]
```

### Known issues

Several customers have reported that Perspective Datagrids fail to automatically render in the Quilt web catalog. We have isolated this problem to clashes with third party browser extensions in both Mozilla Firefox and Google Chrome. At least one extension, [Zotero Connector](https://chrome.google.com/webstore/detail/zotero-connector/ekhagklcjbdpajgpjgmbionohlpdbjgc?hl=en), has been reported and the error reproduced.

If you encounter a rendering error, please first try a different browser (Firefox, Safari, Edge) on the same machine. If the error persists, next disable all third-party extensions, turning each one back on, one-by-one, until the problem extension is identified. Please then notify <support@quilt.bio> with the extension name and version.

### Live packages

* [CSV file automatically rendered in Perspective Datagrid](https://open.quiltdata.com/b/quilt-example/packages/examples/smart-report)

## Integrative Genomics Viewer (IGV)

To render genome tracks, you can select "View as IGV" in the catalog, or you can invoke [igv.js](https://igv.org/) in quilt\_summarize, as shown below:

```json
// quilt_summarize.json
[
  {
    "path": "igv-options-file.json",
    "title": "Awesome genome",
    "types": ["igv"]
  }
]
```

In the above example, `igv-options-file.json` is an [IGV browser configuration](https://github.com/igvteam/igv.js/wiki/Browser-Creation#browser-configuration-options).

You may specify relative paths to package files or absolute S3 URLs as data sources, and the Quilt catalog will resolve them. HTTP URLs will remain unchanged.

**Note: Please be mindful of rendering large sequences** You can limit the downloaded file size of the sequence by using the [`visibilityWindow` parameter](https://github.com/igvteam/igv.js/wiki/Tracks-2.0#options-for-all-track-types) (`-1` is for downloading the whole file, which could potentially be several gigabytes in size - this may impact rendering speed and interactive performance).

Note that tracks are [grouped by type and file format](https://github.com/igvteam/igv.js/wiki/Tracks-2.0).

```json
// igv-options-file.json
{
  "tracks": [{
    "name": "Absolute URL track",
    "url": "s3://bucket/file" // will be resolved
  }, {
    "name": "Relative path track",
    "url": "./file" // will be resolved
  }, {
    "name": "HTTP URL track",
    "url": "https://some-url-even-url-to-s3-file" // will stay intact
  }]
}
```

### Live packages

* [Single and multiple track genome sequences](https://open.quiltdata.com/b/quilt-example/tree/examples/simple-igv/)


# Advanced


# Athena

Quilt stores package data and metadata in S3. Metadata lives in a per-package manifest file in a each bucket's `.quilt/` directory.

You can therefore query package metadata wth SQL engines like AWS Athena. Users can write SQL queries to select packages (or files from within packages) using predicates based on package or object-level metadata.

Packages can be created from the resulting tabular data. To be able to create a package, the table must contain the columns `logical_key`, `physical_keys` (or `physical_key`) and `size`. (See also [Mental Model](https://docs.quilt.bio/mentalmodel))

![Athena page with results ready to be packaged](/files/F8UZlxBHnTeSfVzZXtZ1)

## Defining package tables and views in Athena

> This step is not required for users of Quilt enterprise, since tables and views are managed by Quilt. Check the value of `UserAthenaDatabaseName` output in your CloudFormation stack to know the name of the Athena database it created.

The first step in configuring Athena to query the package contents and metadata is to define a set of tables and views that represent the metadata fields as columns. The easiest way to do this is using the pre-built CloudFormation templates available in the [examples repository](https://github.com/quiltdata/examples/tree/master/athena_cfn/).

## Example: query object-level metadata

Suppose we wish to find all .tiff files produced by algorithm version 1.3 with a cell index of 5.

```sql
SELECT * FROM "YOUR-BUCKET_objects-view"
WHERE substr(logical_key, -5) = '.tiff'
-- extract and query object-level metadata
AND json_extract_scalar(meta, '$.user_meta.nucmembsegmentationalgorithmversion') LIKE '1.3%'
AND json_array_contains(json_extract(meta, '$.user_meta.cellindex'), '5');
```


# Iceberg Tables

> **NOTE:** This feature requires Quilt Platform version 1.70.0 or higher.

Quilt automatically maintains Apache Iceberg tables that provide high-efficiency, externally queryable access to package information. This is particularly useful for buckets that contain thousands of packages or for queries that span multiple buckets.

You can query package revisions, tags, file entries, and metadata using Amazon Athena or external data warehouses that support Iceberg (e.g., Databricks, Snowflake).

## Tables

For each bucket registered with Quilt, four per-bucket tables are maintained in the Iceberg Glue database, whose name is exposed as the `IcebergDatabaseName` output in your CloudFormation stack:

* `{bucket}_package_revision` — package revisions with timestamps
* `{bucket}_package_tag` — named package tags (currently only `latest`)
* `{bucket}_package_manifest` — package-level metadata and commit messages
* `{bucket}_package_entry` — individual file entries within packages

The bucket is encoded in the table name, so the tables do not carry a `bucket` column. Because S3 bucket names can contain hyphens, the resulting Glue table names (e.g. `my-bucket_package_tag`) must be double-quoted in Athena SQL, as shown in the examples below.

Every Quilt role automatically receives Athena read access to the per-bucket tables for the buckets it can read — managed users are scoped to their readable buckets via the registry-applied session policy; non-managed roles have stack-wide access by design.

> **Admin note — Lake Formation.** If AWS Lake Formation enforcement is enabled on your **account** (data lake), you must set the `EnableLakeFormationGrants` CloudFormation parameter to `true` so the stack emits the `PrincipalPermissions` (Lake Formation) grants its service roles need to reach the data lake. **This is opt-in and off by default:** if your account enforces Lake Formation but the stack has *not* enabled these grants, Lake Formation denies the stack and per-bucket Iceberg access (among other things) breaks. Leave the parameter off only on accounts that do not enforce Lake Formation.

## Finding the tables in the Queries tab

As of Quilt Platform 1.71.0, managed users can select the Iceberg package-index database directly from the **Database** dropdown in the catalog's **Queries** tab, instead of typing its fully-qualified name.

## Example: Get entries and metadata for the latest version of a package

```sql
SELECT
  e.logical_key,
  e.physical_key,
  e.size,
  e.metadata
FROM "my-bucket_package_tag" t
JOIN "my-bucket_package_entry" e
  ON t.top_hash = e.top_hash
WHERE t.pkg_name = 'analytics/results'
  AND t.tag_name = 'latest'
```

## Example: Find latest packages matching specific metadata

```sql
SELECT
  t.pkg_name,
  t.tag_name,
  m.metadata
FROM "my-bucket_package_tag" t
JOIN "my-bucket_package_manifest" m
  ON t.top_hash = m.top_hash
WHERE t.tag_name = 'latest'
  AND json_extract_scalar(m.metadata, '$.experiment_id') = 'EXP-123'
  AND json_extract_scalar(m.metadata, '$.status') = 'complete'
```

## Cross-bucket queries

To search across multiple buckets, `UNION ALL` the per-bucket tables explicitly:

```sql
SELECT 'bucket-a' AS bucket, pkg_name, tag_name, top_hash
FROM "bucket-a_package_tag"
WHERE tag_name = 'latest'
UNION ALL
SELECT 'bucket-b' AS bucket, pkg_name, tag_name, top_hash
FROM "bucket-b_package_tag"
WHERE tag_name = 'latest'
```

## See also

* [Query](/quilt-platform-catalog-user/query): Use the Catalog's Queries tab
* [Athena](/quilt-platform-catalog-user/advanced/athena): Query package manifests using AWS Athena
* [Tabulator](/quilt-platform-administrator/advanced/tabulator): Query tabular data within packages


# Elasticsearch

Each Quilt stack includes an Elasticsearch cluster that indexes objects and packages as documents. The cluster is deployed in the AWS OpenSearch service. You can connect to your Elasticsearch domain to query documents.

Each Amazon S3 bucket connected to Quilt implies two Elasticsearch index aliases:

1. `YOUR_BUCKET_NAME`: Contains one document per object in the bucket.
2. `YOUR_BUCKET_NAME_packages`: Contains one document per package revision in the bucket.

### Configuring Saved Queries

You can provide pre-canned Elasticsearch queries for your users by providing a configuration file at `s3://YOUR_BUCKET/.quilt/queries/config.yaml`:

```yaml
version: "1"
queries:
  query-1:
    name: My first query
    description: Optional description
    url: s3://BUCKET/.quilt/queries/query-1.json
  query-2:
    name: Second query
    url: s3://BUCKET/.quilt/queries/query-2.json
```

The Quilt catalog displays your saved queries in a drop-down for your users to select, edit, and execute.

### Managing Elasticsearch

Quilt uses Amazon Elasticsearch 7.10 ([docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/index.html)), which also supports cost-effective Graviton (ARM) clusters. New CloudFormation and Terraform deployments default to Graviton2 (`m6g.xlarge` / `m6g.large`) instances. Customers with Reserved Instances should contact Quilt support before switching instance types to avoid double-paying.

1. If your Quilt stack uses private endpoints for Elasticsearch you will need to connect to the cluster from a machine in the same VPC as the cluster.
2. By default, Quilt indexes a limited number of bytes per document for specified file formats (100KB). Both the max number of bytes per document and which file formats to deep index can be customized per Bucket in the Catalog Admin settings.

![Example of Admin Bucket indexing options](/files/S0JLN0tz5vax8e1cP3DD)

### Querying Elasticsearch with Python

You can use [`elasticsearch`](https://elasticsearch-py.readthedocs.io/en/) as follows:

```python
from aws_requests_auth.boto_utils import BotoAWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection

es_host = "check.aws.console.for.your.host.us-east-1.es.amazonaws.com"

auth = BotoAWSRequestsAuth(
    aws_host=es_host,
    aws_region='us-east-1',
    aws_service='es'
)

elastic = Elasticsearch(
    hosts=[
        {"host": f"{es_host}", "port": 443}
    ],
    http_auth=auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    timeout=27
)

query = rbody = {
    "query": {
        # query body here
    }
}

elastic.search(
    index="*_packages", # search all package indexes in this stack
    body=rbody,
    _source=['*'], # return all document fields
    size=1000,
)
```

### Monitoring your Elasticsearch cluster

It's helpful to periodically review the Quilt Elasticsearch cluster health to ensure performance is optimal and cost-effective.

To review your Elasticsearch cluster:

1. In CloudFormation, select your Quilt stack.
2. In the "Resources" tab, type `AWS::Elasticsearch::Domain`. There will be one result (`Logical ID` value of `Search`).
3. Click the link in the `Physical ID` column to access the Quilt Elasticsearch cluster in the AWS OpenSearch service.
4. Select the "Cluster health" tab.
5. Review the "Summary" section (look for **Green** Status):
   * If your cluster Status is **Red** or **Yellow**, notify your Quilt account manager.
6. In the "Overall health" section, update the "Time range" to `2w` and review all graphs, paying particular attention to:
   * Total free storage space: if one or more nodes in your cluster has storage space less than the minimum value of either 20% of available storage space or 20GB of storage space, basic operations (adding documents; creating indexes) can start to fail. Notify your Quilt account manager.
   * Searchable documents: if the total number of searchable documents remains static or reduces but you are adding documents to Quilt, notify your Quilt account manager.
7. In the "Key performance indicators" section, update the "Time range" to `2w` and review all graphs, paying particular attention to:
   * HTTP requests by response code: Excessive 4xx or 5xx HTTP response codes can indicate problems with your cluster configuration.

> Data from the Quilt Elasticsearch cluster is published to Amazon Cloudwatch and retained for two weeks before being discarded. The charts listed above are created from raw data in Cloudwatch. Depending on your needs, you may wish to view cluster data in CloudWatch directly.

#### References

* [Sizing Amazon OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html)

### Auto-Tune and CloudFormation

Auto-Tune is a feature of the Amazon OpenSearch service that analyzes cluster performance over time and suggests optimizations based on workload. There is known bug in CloudFormation when deploying and/or upgrading Quilt stacks using t2 or t3 instance types. During stack deployments the following error may be encountered:

```log
Autotune is not supported in t2/t3 instance types.
Disable autotune or change your instance type.
(Service: AWSElasticsearch; Status Code: 400; Error Code: ValidationException;
Request ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX; Proxy: null)
```

This is due to CloudFormation turning Auto-Tune on by default. There is currently no way to modify Auto-Tune configurations using CloudFormation. Your Quilt administrator will need to take the following actions and re-run the Quilt CloudFormation deployment:

1. Access the Quilt OpenSearch cluster (see steps 1 - 3 above).
2. Select the "Auto-Tune" tab.
3. Review the "Status" value. If the value is **Turned on**, click the "Edit" button.
4. Select the option to "Turn off" Auto-Tune and click the "Save changes" button:

![Auto-Tune configuration](/files/2Ihx9M677zVsJcJxXghE)


# Removing Stacks

If you ever need to delete a Quilt stack from your AWS account, you should know that:

* Your data in S3 buckets remains accessible.
* You can continue using the Quilt SDK for reading and writing packages.
* Stack-managed resources (Athena tables, Postgres database, audit logs) will be affected.
* The deletion behavior varies between CloudFormation and Terraform deployments.

***

### I. Impact Assessment and Preparation

#### What May Be Deleted

* Custom configurations specific to the Quilt stack (e.g., IAM roles, resource mappings, permissions).
* Analytics data in stack-managed buckets.
* Audit logs and Athena querying setups.
* User account and tabulator configurations.

#### Recommended Actions

* Export analytics data you need to keep.
* Save important audit logs.
* Document existing Athena configurations if you'll need to recreate them.
* Export the Postgres database (containing, e.g., user accounts and tabulator configuration) in case you want to reuse it for future stacks.

***

### II. Stack Deletion Process

#### Using CloudFormation

1. **Start the Deletion Process**
   * Navigate to the **CloudFormation** console in your AWS region.
   * Select your stack under **CloudFormation > Stacks**.
   * Click **Delete Stack** to begin the process.
2. **Empty or Ignore Non-Deleted Resources**
   * CloudFormation attempts to delete all stack-managed resources.
   * If a resource cannot be deleted (e.g., non-empty S3 bucket):
     * It will show a **DELETE\_FAILED** status in the **Events** tab.
     * Failed resources remain intact, including non-empty S3 buckets.
     * Other independent resources will still be deleted.
     * The stack enters the **DELETE\_FAILED** state.
   * If you wish to remove those resources:
     * Back up any important files.
     * Remove all files from the bucket via the S3 console or CLI.
     * Retry the deletion process.
   * Alternatively, you can leave them undeleted, though that may require manual cleanup later.

#### Using Terraform

1. **Start the Deletion Process**
   * Open a terminal in your Terraform configuration directory.
   * Run `terraform destroy`.
2. **Empty or Retain Non-Deleted Resources**
   * Terraform stops at the first resource it cannot delete (e.g., non-empty S3 buckets created by CloudFormation).
   * Dependencies of the failed resource are preserved.
   * To resolve, back up important files and clear the bucket contents.
   * Once addressed, resume deletion by re-running `terraform destroy`.

***

### III. Final Steps

* Verify deletion of all non-retained resources.
* Confirm that the stack is no longer listed in CloudFormation or Terraform.
* Check for any orphaned resources or residual costs using the **AWS Cost Explorer**.

***

### IV. After Stack Deletion

* Your S3 bucket data remains intact and accessible.
* The Quilt SDK continues to function for package operations.
* Recreate any necessary Athena/Glue configurations if needed.


# Catalog Installation

Quilt is a data mesh that verifies the integrity of your data so that teams can find, understand, and file discoveries based on data of any size or in any format.

A Quilt *instance* is a private portal that runs in your virtual private cloud (VPC).

Quilt supports multiple deployment methods including CloudFormation, AWS Marketplace, and Terraform.

## Help and Advice

We encourage users to contact us before deploying Quilt. We will make sure that you have the latest version of Quilt, and walk you through the CloudFormation deployment.

We recommend that all users do one or more of the following:

* [Schedule a Quilt engineer](https://calendly.com/d/g6f-vnd-qf3/engineering-team) to guide you through the installation
* [Join Quilt on Slack](https://slack.quilt.bio) to ask questions and connect with other users
* [Email Quilt](mailto:support@quilt.bio)

## Requirements and Prerequisites

### Knowledge Requirements

Running Quilt requires working knowledge of [AWS CloudFormation](https://aws.amazon.com/cloudformation/), [AWS S3](https://aws.amazon.com/s3/) and [Elasticsearch Service](https://aws.amazon.com/elasticsearch-service/).

### Before you install Quilt

You will need the following:

1. **An AWS account**.

   1. **The service-linked role for Elasticsearch**

   > This role is not created automatically when you use Cloudformation or other APIs.

   You can create the role as follows:

   ```bash
   aws iam create-service-linked-role --aws-service-name es.amazonaws.com
   ```
2. **IAM Permissions** to create the CloudFormation stack (or Add products in Service Catalog).

   1. You may choose to use a [CloudFormation service role](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-iam-servicerole.html) for stack creation and updates.

   2. Refer to this [example service role](https://github.com/quiltdata/quilt/blob/master/docs/cfn-service-role.yaml) and modify as needed to fit your use case.

   > Ensure that your service role is up-to-date with the example before every stack update so as to prevent installation failures.
3. The **ability to create DNS entries** (Route 53 alias records or CNAMEs) for your company's domain.
4. **An SSL certificate in the same region as your Quilt instance** to secure the domain where your users will access your Quilt instance.
   1. For example, to make your Quilt catalog available at `https://quilt.mycompany.com`, you require a certificate for either `*.mycompany.com` *or* for the following 3 domains: `quilt.mycompany.com`, `quilt-registry.mycompany.com` and `quilt-s3-proxy.mycompany.com` in the [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/).
   2. You may either [create a new certificate](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html), or [import an existing certificate](https://docs.aws.amazon.com/acm/latest/userguide/import-certificate.html).
   3. The ARN for this certificate or set of certificates is required for use as the `CertificateArnELB` CloudFormation parameter.
5. For maximum security, Quilt requires **a region that supports** [**AWS Fargate**](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate-Regions.html#linux-regions). As of this writing, all U.S. regions support Fargate.
6. **An S3 Bucket** for your team data. This may be a new or existing bucket. The bucket should not have any notifications attached to it (`S3 Console > Bucket > Properties > Events`). Quilt will need to install its own notifications. Installing Quilt will modify the following Bucket characteristics:

   1. Properties > Object-level logging (will be enabled).

   2. Properties > Events (will add one notification).

   > Buckets in Quilt may choose to enable versioning or disable versioning. **It is strongly recommended that you keep versioning either on or off during the entire lifetime of the bucket**. Toggling versioning on and off incurs edge cases that may cause bugs with any state that Quilt stores in ElasticSearch due to inconsistent semantics of `ObjectRemoved:DeleteMarkerCreated`.
7. Available **CloudTrail Trails** in the region where you wish to host your stack ([learn more](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/WhatIsCloudTrail-Limits.html)).
8. A license key or an active subscription to Quilt Business on AWS Marketplace.
   1. Click `Continue to Subscribe` on the [Quilt Business Listing](https://aws.amazon.com/marketplace/pp/B07QF1VXFQ) to subscribe then return to this page for installation instructions.
   2. **The CloudFormation template and instructions on AWS Marketplace are infrequently updated and may be missing critical bugfixes.**

## Installation Methods

### AWS Marketplace

You can install Quilt via AWS Marketplace. As indicated above, we recommend that you [contact us first](#help-and-advice).

### AWS Service Catalog

1. Email <support@quilt.bio> with your AWS account ID to request access to Quilt through the AWS Service Catalog and to obtain a license key.
2. Click the service catalog link that you received from Quilt. Arrive at the Service Catalog. Click IMPORT, lower right.

   ![Import portfolio page](/files/-M7Q91bzb1NUot2F7C1K)
3. Navigate to Admin > Portfolios list > Imported Portfolios. Click Quilt Enterprise.

   ![Portfolio page](/files/-M7Q91c1g3onArG5spHG)
4. On the Portfolio details page, click ADD USER, GROUP OR ROLE. Add any users, **including yourself**, whom you would like to be able to install Quilt.

   ![Portfolio users page](/files/-MK0AF_5DYq7fci3TBrd)
5. Click Products list, upper left. Click the menu to the left of Quilt CloudFormation Template. Click Launch product. (In the future, use the same menu to upgrade Quilt when a new version is released.)

   ![Products list page](/files/-MK0AF_6v_aMw8Nm76tN)
6. Continue to the [CloudFormation](#cloudformation) section. Note: the following screenshots may differ slightly from what you see in Service Catalog.

### CloudFormation

You can perform stack update and creation with the AWS Console, AWS CLI, Terraform, or other means.

> **Important:** Use Quilt-provided CloudFormation templates without modification. Customizing templates may result in deployment issues and can affect your service agreement coverage. If you require specific customizations, please contact your Quilt account manager to discuss supported options.

In all cases it is **highly recommended** that you set the `--on-failure` policy to `ROLLBACK` so as to avoid incomplete rollback and problematic stack states. In the AWS Console this option appears under the phrase "Stack failure options."

1. Specify stack details in the form of a stack *name* and CloudFormation *parameters*. Refer to the descriptions displayed above each text box for further details. Service Catalog users require a license key. See [Before you install Quilt](#before-you-install-quilt) for how to obtain a license key.

   ![Stack details page](/files/-MK0AF_7o5UryOXxMt6K)
2. If you wish to use a service role, specify it as follows:

   ![Specifying stack role](/files/-M8wJbg_6axBPHfB3Jc5)
3. Service Catalog users, skip this step. Under Stack creation options, enable termination protection. This protects the stack from accidental deletion. Click Next.

   ![Enabling stack protection](/files/-M7Q91c5kmIKq0SRDE0f)
4. Service Catalog users, skip this step. Check the box asking you to acknowledge that CloudFormation may create IAM roles, then click Create.

   ![Confirmation page](/files/-MK0AF_Ani71GBM0j4-M)
5. CloudFormation may take between 30 and 90 minutes to create your stack. You can monitor progress under Events. On completion you will see `CREATE_COMPLETE`.

   ![Stack events page](/files/-MMTjQHcyZM_85y2P9VY)
6. To finish the installation, you will want to view the stack Outputs.

   ![Stack outputs page](/files/-MK0AF_Cbs-5Vybcxit6)

### Terraform

You can also install Quilt using [Terraform](https://developer.hashicorp.com/terraform), which enables more granular infrastructure-as-code control.

Terraform users **must** request a compatible CloudFormation template from Quilt:

> Contact your account manager to obtain a template that works with Terraform and includes necessary variables.

> **Important:** Use Quilt-provided Terraform modules and CloudFormation templates without modification. Customizing these resources may result in deployment issues and can affect your service agreement coverage. If you require specific customizations, please contact your Quilt account manager to discuss supported options.

1. Set up your project directory as follows:

   ```bash
   quilt_stack/
   ├── main.tf
   └── my-company.yml
   ```

   Use [examples/main.tf](https://github.com/quiltdata/iac/blob/main/examples/main.tf) as a template.
2. Define your AWS profile:

   ```bash
   export AWS_PROFILE=your-profile-name
   ```
3. Initialize Terraform:

   ```bash
   terraform init
   ```
4. Plan and apply:

   ```bash
   terraform plan -out=tfplan
   terraform apply tfplan
   ```
5. Use `terraform output` to obtain values such as the admin password or endpoint URLs.

**Note:** We recommend using [remote state](https://developer.hashicorp.com/terraform/language/state/remote) and not storing passwords in version control.

> For detailed configuration options, including search sizing and common pitfalls, see the [Terraform README](https://github.com/quiltdata/iac/blob/main/README.md).

### DNS records

In order for your users to reach the Quilt catalog you must create three DNS records (four if Connect is enabled) pointing to the `LoadBalancerDNSName` as shown below and in the Outputs of your stack.

| Hostname                               | Target                         |
| -------------------------------------- | ------------------------------ |
| `<QuiltWebHost>`                       | `<LoadBalancerDNSName>`        |
| `<RegistryHost>`                       | `<LoadBalancerDNSName>`        |
| `<S3ProxyHost>`                        | `<LoadBalancerDNSName>`        |
| `<ConnectHost>` *(if Connect enabled)* | `<ConnectLoadBalancerDNSName>` |

If your hosted zone is in **Route 53**, we recommend Route 53 alias records (record type `A`, alias target = `LoadBalancerDNSName`, hosted zone ID = `LoadBalancerCanonicalHostedZoneID`). Route 53 doesn't charge for alias queries to AWS resources like ALBs, and aliases work at the zone apex (which CNAMEs cannot).

If your DNS is hosted elsewhere, use **CNAME** records pointing to `LoadBalancerDNSName`.

Quilt is now up and running. You can click on the *QuiltWebHost* value in Outputs and log in with your administrator password to invite users.

## Routine Maintenance and Upgrades

Releases are sent to customers over email. We recommend that you apply new releases as soon as possible to benefit from the latest security updates and features.

### CloudFormation updates

To update your Quilt stack, apply the latest CloudFormation template in the CloudFormation console as follows.

> By default, previous parameter values carry over.

1. Navigate to AWS Console > CloudFormation > Stacks
2. Select your Quilt stack
3. Click Update (upper right)
4. Choose Replace current template
5. Enter the Amazon S3 URL for your template
6. Click Next (several times) and proceed to apply the update

### Terraform updates

> See above.

## Upgrading from network 1.0 to network 2.0

Upgrading to the Quilt 2.0 network configuration provides improved security by means of isolated subnets and a preference for private routing.

An upgrade to the 2.0 network, unlike routine Quilt upgrades, requires you to create a new stack with a new load balancer. You must therefore also update your [DNS records](#dns-records) to point to the new load balancer.

## Create a new stack with an existing configuration

Terraform users can create a new Quilt stack with the same configuration as an existing stack. This is typically useful when upgrading to the 2.0 network.

> *Configuration* refers to the Quilt stack buckets, roles, policies, and other administrative settings, all of which are stored in RDS.

Perform the following steps:

1. Contact your Quilt account manager for a template that supports Terraform.
2. Take a manual snapshot of the current Quilt database instance. For an existing Quilt stack this resource has the logical ID "DB". Note the snapshot identifier ("Snapshot name" in the AWS Console, `DBSnapshotIdentifier` in the following AWS CLI command):

   ```sh
   aws rds describe-db-snapshots
   ```

   > Be sure to take a *manual* snapshot. Do not rely on automatic snapshots, which are deleted when the parent stack is deleted.
3. Apply the [quilt Terraform module](https://github.com/quiltdata/iac) to your new template and provide the snapshot identifier to the `db_snapshot_identifier=` argument.

   > You must use a Quilt CloudFormation template that supports an existing database, existing search domain, and existing vpc in order for the terraform modules to function properly.
4. You now have a new Quilt stack with a configuration equivalent to your prior stack. Verify that the new stack is working as desired. Delete the old stack.


# Admin Settings UI

The Quilt catalog includes an admin panel that allows you to manage users and buckets in Quilt, as well as customize the Quilt catalog. You can access the panel via a dropdown menu under username in the navbar.

![](/files/-MgEVYDB-OIUp-dDFhyC)

The admin page is only accessible to designated administrators. The first admin is set during the CloudFormation installation. Subsequent admins may be designated through the panel. Only admins may create other admins.

Quilt requires at least one admin account per stack.

### Users and roles

This section provides comprehensive access control management for the following functions:

* Create/Delete user
* De/activate user
* De/admin user
* Assign roles to users
* Configure access policies

![](/files/-MgEVYDCmHXKpvdPe3Hp)

You may invite new users to your Quilt stack by clicking the + button, upper right. You must assign one or more roles to all new users (default role is pre-selected for you). You can edit existing users' attributes by clicking on underlined cells.

![](/files/FHlcju2RvqABa60mvC3H)

Users can switch between assigned roles via the dropdown menu in the navbar (if assigned more than one).

![](/files/B3wBdovQ6pXuMrLnbwA1)

![](/files/MJXQf11016hFGCKsn7OU)

You must select the default role for all new users, else they will not be able to sign in to the Quilt catalog. The default role is shown in bold.

![](/files/bHdgRSmaXDxg8aV5abS6)

You may create roles for different groups of users by combining up to 5 policies. Users of managed roles — including administrators — only see, list, and search buckets for which their role is explicitly granted read access.

![](/files/zeXotQRAbBFaQgNRt6pD)

![](/files/IyT6LDbdWSt60tI9Jxdk)

Alternatively, you may provide your own IAM roles via ARN:

![](/files/DWLkta1SqDQ5TBcnzpab)

You may create policies providing access to a selected set of buckets:

![](/files/vUwU6ScHSJoAVCvXMKk7)

![](/files/Jik0zXLzlG3YHPVpkp4Q)

![](/files/8m6omhEhtQFBNzIlnyhQ)

![](/files/sr5p5ngqfyLznXW7jLiN)

You may attach policies to managed roles from policy edit and create screens:

![](/files/sTffJUfVHM8xbTzH2Z7I)

You may also provide custom policies via ARN:

![](/files/1N8fLUnUb8EjF7StZfpZ)

The resulting permission set is equivalent to a union of all permissions provided by the policies attached to that role.

> **Note:** a managed role's IAM policy is regenerated when its bucket permissions are saved, so permissions added in a newer Quilt release reach existing managed roles on their next save.

### Buckets

Here you can add or remove buckets from Quilt and configure bucket indexing and display settings.

![](/files/-MgEVYDEwNN3N0xuLZrH)

![](/files/-MgEVYDFzDYCaO9XXJ6F)

#### Reindexing

Stack admins can reindex a bucket via `POST /api/admin/reindex/<bucket>`. The request accepts an optional `prefix` field; when supplied, the existing Elasticsearch indices are left in place and only keys under that prefix are re-walked. This is useful for refreshing a slice of a large bucket without a full reindex.

#### S3 events

By default, when you add a bucket to the Quilt stack one of two things will happen:

1. If there is no existing bucket notification, Quilt will attempt to add a new notification
2. If there is an existing bucket notification, Quilt will use the existing notification if and only if it supports the required events (object creation and deletion)

If either of the above conditions fails, Quilt will not add the bucket in question.

See [S3 Events, EventBridge](/quilt-platform-administrator/advanced/eventbridge) for more.

#### Stack-managed bucket protections

Manual bucket-management operations on stack-managed S3 buckets are denied by bucket policy. All configuration changes to a stack-managed bucket — including notifications, policies, and lifecycle rules — must go through CloudFormation rather than the S3 console, CLI, or API. This keeps each bucket's configuration in sync with the stack template and prevents drift that could break indexing or event delivery.

### Settings

This section allows you to customize your Quilt catalog, including custom links in the navbar, custom logo, and default search mode. The Theme editor accepts a logo as either a direct file upload (PNG, JPEG, WebP, or GIF) or a URL (all of those plus SVG).

![](/files/-MgEVYDGQNdAfa67cnLw)

### Further settings

See [Preferences](/quilt-platform-administrator/preferences) for further control over the catalog user interface.


# Catalog Configuration

## Show and hide features in the Quilt catalog

You can use the configuration file to show or hide certain tabs and buttons in the Quilt catalog. This gives you fine-grained control over how users interact with the Quilt catalog. There is one catalog config file per-bucket. The config file's path is `s3://BUCKET/.quilt/catalog/config.yaml`.

If there is no config.yaml, or your config.yaml file does not override the `ui` key, then the following defaults are used:

```yaml
ui:
  nav:
    files: True
    workflows: True
    packages: True
    queries: True
  actions:
    copyPackage: True
    createPackage: True
    deleteObject: False
    deleteRevision: False
    downloadObject: True
    downloadPackage: True
    restore: True
    revisePackage: True
    writeFile: True
  blocks:
    analytics: True
    browser: True
    code: True
    gallery:
      files: True
      packages: True
      overview: True
      summarize: True
    meta:
      user_meta:
        expanded: False
      workflows:
        expanded: False
  package_description:
    .*:
      message: True
  package_description_multiline: False
```

### Properties

* `ui.nav.files: False` - hide Files tab
* `ui.nav.workflows: False` - hide Workflows tab
* `ui.nav.packages: False` - hide Packages tab
* `ui.nav.queries: False` - hide Queries tab
* `ui.actions: False` - hide all buttons used to create and edit packages and files (make the catalog "read-only")
* `ui.actions.copyPackage: False` - hide buttons to push packages across buckets
* `ui.actions.createPackage: False` - hide buttons to create packages via drag-and-drop or from folders in S3
* `ui.actions.deleteObject: True` - show buttons to delete files and directories (off by default since 1.66 to prevent accidental deletions)
* `ui.actions.deleteRevision: True` - show buttons to delete package revision
* `ui.actions.downloadObject: False` - hide download buttons under "Bucket" tab
* `ui.actions.downloadPackage: False` - hide download buttons under "Packages" tab
* `ui.actions.restore: False` - hide the button to restore (rehydrate) archived S3 Glacier / Deep Archive objects
* `ui.actions.revisePackage: False` - hide the button to revise packages
* `ui.actions.writeFile: False` - hide buttons to create or edit files
* `ui.blocks.analytics: False` - hide Analytics block on file page
* `ui.blocks.browser: False` - hide files browser on both Bucket and Packages tab
* `ui.blocks.code: False` - hide Code block with quilt3 code boilerplate
* `ui.blocks.gallery: False` - hide all galleries (see below for list of galleries)
* `ui.blocks.gallery.files: False` - hide gallery in Bucket tab; this gallery lists all images in the current directory
* `ui.blocks.gallery.packages: False` - hide gallery in Packages tab; this gallery lists all images in the current directory in package
* `ui.blocks.gallery.overview: False` - hide gallery in Overview tab; this gallery lists all images in the current bucket
* `ui.blocks.gallery.summarize: False` - hide gallery when `quilt_summarize.json` is present
* `ui.blocks.meta: False` - hide Metadata block on Package page
* `ui.blocks.meta.user_meta.expanded: True` - expands user\_meta properties
* `ui.blocks.meta.workflows.expanded: 2` - expands workflows two level deep
* `ui.sourceBuckets` - a dictionary of S3 bucket names that map to an empty object reserved for future enhancements; buckets in this dictionary are the ones offered when the user clicks Revise Package > Add files from Bucket; by default, the current bucket is always available; set to an empty dictionary `{}` to disable this feature
* `ui.defaultSourceBucket` - source bucket from `ui.sourceBuckets` that is selected by default; if it doesn't match any bucket then it's ignored
* `ui.package_description` - a dictionary that maps package handle regular expressions to JSONPath expressions of fields to show from package metadata in the package list view.
* `ui.package_description_multiline: True` - expands package metadata's root key/values
* `ui.athena.defaultWorkgroup` - default workgroup to select on the Athena page

#### `ui.sourceBuckets` example

```yaml
ui:
  sourceBuckets:
    s3://bucket-a: {}
    bucket-b: {}  # "s3://" prefix is optional
    s3://bucket-c: {}
  defaultSourceBucket: s3://bucket-b
```

By default, users can add files from the current bucket when creating or revising packages. The current bucket is automatically available in the file picker without requiring configuration. To restrict this functionality and disable adding files from any bucket (including the current one), set `ui.sourceBuckets` to an empty dictionary `{}` in your configuration file.

#### `ui.package_description` example

```yaml
ui:
  package_description:
    # match all packages
    .*:
      # show the message
      message: True
      # show the .labels field
      user_meta:
        - $.labels
    # for any package with a handle prefix of foo
    ^foo/.*:
      # JSONPath expressions to the fields to display
      user_meta:
        - $.key1.key2
        - $.key3
        - $.key4[0]
```

![Example of package\_description use](/files/eUUvulRiK7habI4KrLdq)

#### `ui.athena` example

```yaml
ui:
  athena:
    defaultWorkgroup: primary
```

#### `ui.blocks.meta`

User could set it to boolean or object with additional properties. Object is a truthy value, so any object acts like `True`.

```yaml
# default value, show Metadata block
ui:
  blocks:
    meta: True
```

```yaml
# Hide Metadata block
ui:
  blocks:
    meta: False
```

```yaml
# Show Metadata block and expand `user_meta` object
# consider large `user_meta` objects, they can affect UI responsiveness
ui:
  blocks:
    meta:
      user_meta:
        expanded: True
```


# Quilt Connect Server

> Connect Server requires Quilt Platform version 1.68 or later.

**Quilt Connect Server** is an identity provider and gateway that enables external services to securely interact with your Quilt data and perform actions on behalf of your users. Connect Server:

* Authenticates requests using your organization's identity provider
* Issues session tokens scoped to individual user permissions
* Routes requests to authorized services within your AWS environment

One such service is the [Platform MCP Server](/quilt-platform-catalog-user/mcp-server), which lets AI assistants interact with your Quilt data through natural language.

## Admin Setup

Connect Server is disabled by default. To enable it, set the `ConnectAllowedHosts` CloudFormation parameter to a non-empty value.

### CloudFormation Parameters

| Parameter               | Default   | Description                                                                                                                                                                            |
| ----------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ConnectAllowedHosts`   | *(empty)* | Comma-separated list of allowed OAuth redirect origins. Empty = disabled. See [Entry formats](#connectallowedhosts-entry-formats) below.                                               |
| `ConnectSecurityGroup`  | *(empty)* | Optional EC2 security group ID for Connect ALB IP allowlisting. Empty = allow all.                                                                                                     |
| `CertificateArnConnect` | *(empty)* | Optional ACM certificate ARN for the Connect ALB. Empty = reuses the main stack TLS certificate, which must then cover the Connect host (see [DNS Configuration](#dns-configuration)). |

#### `ConnectAllowedHosts` Entry Formats

Each comma-separated entry can be one of:

| Format                 | Example          | Matches                                                                                                                                                                                                             |
| ---------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Hostname**           | `claude.ai`      | `https://claude.ai/*` (HTTPS only)                                                                                                                                                                                  |
| **Subdomain wildcard** | `.benchling.com` | `https://<any-subdomain>.benchling.com/*` — any subdomain at any depth (e.g. `app.benchling.com`, `app.us.benchling.com`); does **not** match the apex `https://benchling.com/*` (HTTPS only; leading dot required) |
| **Custom scheme**      | `cursor://`      | `cursor://<any-host>/*` (for desktop apps with a custom URI scheme)                                                                                                                                                 |
| **Localhost**          | `localhost`      | `http://localhost:<any-port>/*` and `http://127.0.0.1:<any-port>/*` (HTTP only; either loopback enables both)                                                                                                       |

Canonical example covering the common web, desktop, and local MCP clients (loopback first, then suffix wildcards, then alphabetized hostnames and custom schemes):

```
localhost,.benchling.com,.cloud.databricks.com,chat.openai.com,chatgpt.com,claude.ai,claude.com,cursor://,gemini.google.com,vscode.dev,windsurf://
```

Entries are case-insensitive. Trailing dots on hostnames are ignored. Network schemes (`http://`, `https://`, etc.) are not valid entries and are silently ignored — use a bare hostname (or `.`-prefixed wildcard suffix) for HTTPS clients and a custom scheme (`cursor://`) for desktop clients.

## DNS Configuration

After deploying with Connect enabled, create a DNS record for your Connect subdomain (typically `<stack-name>-connect.<your-domain>`).

If your hosted zone is in **Route 53**, we recommend an alias record:

| Route 53 Field | Value                                              |
| -------------- | -------------------------------------------------- |
| Record type    | `A` (alias)                                        |
| Alias target   | `ConnectLoadBalancerDNSName` CloudFormation output |
| Hosted zone ID | `ConnectLoadBalancerCanonicalHostedZoneID` output  |

If your DNS is hosted elsewhere, create a `CNAME` record pointing to the `ConnectLoadBalancerDNSName` CloudFormation output. See the [Installation CNAMEs section](/quilt-platform-administrator/installation#cnames) for the equivalent catalog DNS records.

> **ALB scheme (internal vs. internet-facing).** As of Quilt 1.71, the Connect ALB follows the same scheme as the main catalog load balancer. On an internet-facing stack the Connect endpoint is reachable from the public internet; on an internal stack it is internal — reachable only from within your VPC or over your VPN, like the catalog itself. On internal stacks, local or desktop MCP clients (Cursor, Claude Code, etc.) must therefore run on a network with access to the Connect host.

> **TLS certificate:** by default (`CertificateArnConnect` empty) the Connect ALB reuses the main stack certificate (`CertificateArnELB`), which must then cover the Connect host (the `ConnectHost` output) — via an explicit Subject Alternative Name or a wildcard at the matching level (`*` matches a single label only). If it does not, set `CertificateArnConnect` to a certificate that does; otherwise DNS resolves but HTTPS fails with a certificate name mismatch.

The final Connect Server hostname is available in the `ConnectHost` CloudFormation output.

## OAuth Metadata

Connect Server publishes OAuth authorization server metadata at `/.well-known/oauth-authorization-server` and OpenID metadata at `/.well-known/openid-configuration`. The `issuer` value is the Connect Server origin with the explicit HTTPS default port (`https://<connect-host>:443`), and all advertised endpoints (`/auth/token`, `/auth/register`, `/auth/revoke`, `/auth/.well-known/jwks.json`, and the cross-served `/connect/authorize`) include the same explicit `:443`.

> **Compatibility note.** Per RFC 3986, `https://host` and `https://host:443` identify the same origin and are equivalent. However, some strict OAuth clients — notably Databricks Apps — perform string-sensitive origin comparisons against the issuer and reject DCR when the issuer omits the default port. Quilt Connect emits `https://<connect-host>:443` to remain compatible with these clients; well-behaved clients that normalize per RFC 3986 are unaffected.

### Protected Resource Identifier

Connect Server registers exactly one OAuth `resource` identifier for the Platform MCP server:

```
https://<connect-host>/mcp/platform
```

Connect normalizes inbound `resource` parameters by stripping a trailing `/mcp` before the membership check, so clients that submit the full MCP transport URL (`https://<connect-host>/mcp/platform/mcp`, as ChatGPT does) are accepted alongside clients that derive the canonical resource from [RFC 9728 Protected Resource Metadata](https://datatracker.ietf.org/doc/html/rfc9728) (as Claude.ai and Cursor do). The token `aud` claim is always the canonical `/mcp/platform` identifier regardless of input form.

## IP Allowlisting (Optional)

To restrict which IP ranges can reach the Connect Server, create an EC2 security group with inbound rules on port 443 for your trusted CIDR ranges, then pass the security group ID as `ConnectSecurityGroup`. If omitted, the Connect ALB accepts traffic from any IP.


# Cross-Account Access

This guide explains how to set up Quilt across multiple AWS accounts, enabling you to separate your control plane (Quilt infrastructure) from your data plane (S3 buckets) for enhanced security, compliance, and organizational structure.

## 🎯 Architecture Overview

### Why Cross-Account Setup?

**Common Use Cases:**

* 🏢 **Organizational Separation**: Different teams/departments own different accounts
* 🔒 **Security Isolation**: Separate sensitive data from application infrastructure
* 📊 **Compliance Requirements**: Regulatory requirements for data segregation
* 💰 **Cost Management**: Separate billing and resource management
* 🛡️ **Blast Radius Reduction**: Limit impact of security incidents

### Account Structure

In this guide, we'll configure two accounts:

```
┌─────────────────────────────────────┐
│           Control Account           │
│  ┌─────────────────────────────────┐│
│  │     Quilt Infrastructure        ││
│  │  • CloudFormation Stack        ││
│  │  • Lambda Functions            ││
│  │  • Elasticsearch/OpenSearch    ││
│  │  • API Gateway                 ││
│  │  • Web Application             ││
│  └─────────────────────────────────┘│
└─────────────────────────────────────┘
                   │
                   │ Cross-Account
                   │ Access
                   ▼
┌─────────────────────────────────────┐
│            Data Account             │
│  ┌─────────────────────────────────┐│
│  │        S3 Buckets              ││
│  │  • Raw Data Bucket            ││
│  │  • Processed Data Bucket      ││
│  │  • Archive Bucket             ││
│  └─────────────────────────────────┘│
└─────────────────────────────────────┘
```

**Account Definitions:**

* **Control Account**: Contains the Quilt CloudFormation stack and infrastructure
* **Data Account**: Contains the S3 buckets with your actual data

## 🚀 Step-by-Step Implementation Guide

### Prerequisites

Before starting, ensure you have:

* ✅ **Administrative access** to both AWS accounts
* ✅ **Quilt already deployed** in the Control Account
* ✅ **S3 buckets created** in the Data Account
* ✅ **AWS CLI configured** with appropriate profiles

### Step 1: Configure S3 Object Ownership

**Why This Matters:** When Quilt (running in Control Account) writes objects to buckets in Data Account, you want the Data Account to own those objects for proper access control.

**Implementation:**

1. **Navigate to S3 Console** in Data Account
2. **Select your bucket** → **Permissions** → **Object Ownership**
3. **Edit Object Ownership** and select **"Bucket owner enforced"**

**Using AWS CLI:**

```bash
# Set object ownership to bucket owner enforced
aws s3api put-bucket-ownership-controls \
    --bucket your-data-bucket \
    --ownership-controls Rules='[{ObjectOwnership=BucketOwnerEnforced}]' \
    --profile data-account
```

**Why "Bucket owner enforced"?**

* ✅ Data Account automatically owns all objects
* ✅ Simplifies access control management
* ✅ Prevents ACL-based access complications
* ✅ Required for cross-account Quilt operations

### Step 2: Create Cross-Account Bucket Policy

**Purpose:** Grant Quilt infrastructure in Control Account the necessary permissions to manage buckets in Data Account.

**Create the Bucket Policy:**

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "QuiltCrossAccountAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::CONTROL-ACCOUNT-ID:root"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAttributes", 
                "s3:GetObjectTagging",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionAttributes",
                "s3:GetObjectVersionTagging",
                "s3:ListBucket",
                "s3:ListBucketVersions",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetBucketNotification",
                "s3:PutBucketNotification"
            ],
            "Resource": [
                "arn:aws:s3:::your-data-bucket",
                "arn:aws:s3:::your-data-bucket/*"
            ]
        }
    ]
}
```

**Apply the Policy:**

**Console Method:**

1. Go to **S3 Console** → **Your Bucket** → **Permissions** → **Bucket Policy**
2. Paste the JSON above (replace `CONTROL-ACCOUNT-ID` and `your-data-bucket`)
3. Click **Save changes**

**CLI Method:**

```bash
# Save policy to file
cat > bucket-policy.json << 'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "QuiltCrossAccountAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:root"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAttributes",
                "s3:GetObjectTagging",
                "s3:GetObjectVersion", 
                "s3:GetObjectVersionAttributes",
                "s3:GetObjectVersionTagging",
                "s3:ListBucket",
                "s3:ListBucketVersions",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetBucketNotification",
                "s3:PutBucketNotification"
            ],
            "Resource": [
                "arn:aws:s3:::your-data-bucket",
                "arn:aws:s3:::your-data-bucket/*"
            ]
        }
    ]
}
EOF

# Apply the policy
aws s3api put-bucket-policy \
    --bucket your-data-bucket \
    --policy file://bucket-policy.json \
    --profile data-account
```

**🔒 Security Note:**

> Quilt admins can still control user access to this bucket through the Quilt Admin Panel's Roles and Policies. The bucket policy only grants access to Quilt infrastructure, not end users.

### Step 3: Configure Cross-Account SNS (Optional)

**When You Need This:** If you're using [EventBridge integration](/quilt-platform-administrator/advanced/eventbridge) or have existing SNS topics in the Data Account that Quilt should use for notifications.

**Create SNS Topic Policy:**

Add this statement to your SNS topic's resource policy in the Data Account:

```json
{
    "Sid": "QuiltCrossAccountSNSAccess",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::CONTROL-ACCOUNT-ID:root"
    },
    "Action": [
        "sns:GetTopicAttributes",
        "sns:Subscribe",
        "sns:Unsubscribe"
    ],
    "Resource": "arn:aws:sns:region:DATA-ACCOUNT-ID:your-topic-name"
}
```

**Apply SNS Policy:**

```bash
# Get current policy
aws sns get-topic-attributes \
    --topic-arn arn:aws:sns:region:DATA-ACCOUNT-ID:your-topic-name \
    --attribute-names Policy \
    --profile data-account

# Update policy (merge with existing statements)
aws sns set-topic-attributes \
    --topic-arn arn:aws:sns:region:DATA-ACCOUNT-ID:your-topic-name \
    --attribute-name Policy \
    --attribute-value file://sns-policy.json \
    --profile data-account
```

**Configure in Quilt:**

1. Open **Quilt Admin Panel** → **Buckets**
2. Add or edit your cross-account bucket
3. Under **"Indexing and notifications"**, set the SNS Topic ARN
4. Save the configuration

### Step 4: Set Up CloudTrail (Required)

**Why CloudTrail is Required:**

* 🔍 **Security & Auditing**: Track all S3 API calls
* 📊 **User Analytics**: Quilt uses CloudTrail data for user-facing analytics
* 🚨 **Compliance**: Many regulatory frameworks require audit trails

**Implementation Options:**

#### Option A: Quilt-Managed CloudTrail (Recommended)

If Quilt manages CloudTrail in the Control Account:

1. **Check CloudFormation Stack** in Control Account
2. Go to **CloudFormation** → **Your Quilt Stack** → **Resources**
3. Look for a **CloudTrail resource** (Quilt will auto-add your buckets)

#### Option B: Existing CloudTrail

If you have existing CloudTrail in either account:

1. **Identify the Trail:**

   ```bash
   # List trails in Data Account
   aws cloudtrail describe-trails --profile data-account

   # List trails in Control Account  
   aws cloudtrail describe-trails --profile control-account
   ```
2. **Add S3 Data Events:**

   ```bash
   # Add data events for your bucket
   aws cloudtrail put-event-selectors \
       --trail-name your-trail-name \
       --event-selectors '[
           {
               "ReadWriteType": "All",
               "IncludeManagementEvents": true,
               "DataResources": [
                   {
                       "Type": "AWS::S3::Object",
                       "Values": ["arn:aws:s3:::your-data-bucket/*"]
                   },
                   {
                       "Type": "AWS::S3::Bucket", 
                       "Values": ["arn:aws:s3:::your-data-bucket"]
                   }
               ]
           }
       ]' \
       --profile data-account
   ```
3. **Update Quilt Configuration:**
   * Go to **CloudFormation** → **Your Quilt Stack** → **Parameters**
   * Update the **CloudTrail bucket parameter** with your existing trail's S3 bucket

#### Option C: Cross-Account CloudTrail Access

If CloudTrail is in Data Account but Quilt needs access:

**CloudTrail Bucket Policy:**

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "QuiltCloudTrailAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::CONTROL-ACCOUNT-ID:root"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-cloudtrail-bucket",
                "arn:aws:s3:::your-cloudtrail-bucket/*"
            ]
        }
    ]
}
```

### Step 5: Add Bucket to Quilt

**Final Configuration:**

1. **Open Quilt Admin Panel** in Control Account
2. Navigate to **Buckets** → **Add Bucket**
3. **Configure the bucket:**
   * **Bucket Name**: `your-data-bucket`
   * **Region**: Same as the bucket
   * **SNS Topic ARN**: (If using cross-account SNS)
   * **Event Notifications**: Leave disabled if using EventBridge
4. **Save and Test:**
   * Click **Save**
   * Upload a test file to verify indexing works
   * Check Quilt catalog for the new file

## 🔧 Testing Your Cross-Account Setup

### Verification Steps

#### 1. Test Basic Access

```bash
# From Control Account, test bucket access
aws s3 ls s3://your-data-bucket --profile control-account

# Upload a test file
echo "Cross-account test" > test.txt
aws s3 cp test.txt s3://your-data-bucket/ --profile control-account
```

#### 2. Verify Quilt Integration

1. **Upload a file** to your cross-account bucket
2. **Wait 2-3 minutes** for processing
3. **Check Quilt catalog** to see if the file appears
4. **Test search functionality** in Quilt

#### 3. Check CloudTrail Logging

```bash
# Verify CloudTrail is capturing events
aws logs filter-log-events \
    --log-group-name CloudTrail/YourLogGroup \
    --filter-pattern "{ $.eventSource = s3.amazonaws.com }" \
    --profile data-account
```

### Troubleshooting Common Issues

#### Issue 1: Access Denied Errors

**Symptoms:**

* Quilt can't access the bucket
* "Access Denied" in Quilt logs

**Solutions:**

1. **Verify bucket policy** is correctly applied
2. **Check object ownership** is set to "Bucket owner enforced"
3. **Confirm account IDs** in policies are correct
4. **Test with AWS CLI** from Control Account

#### Issue 2: Objects Not Appearing in Quilt

**Symptoms:**

* Files upload successfully but don't appear in Quilt catalog

**Solutions:**

1. **Check CloudTrail** is logging S3 data events
2. **Verify SNS configuration** if using custom topics
3. **Review Quilt logs** for processing errors
4. **Manual re-index** the bucket in Quilt Admin Panel

#### Issue 3: Permission Errors in Quilt Admin

**Symptoms:**

* Can't add bucket in Quilt Admin Panel
* IAM permission errors

**Solutions:**

1. **Check Quilt IAM roles** have cross-account assume permissions
2. **Verify bucket policy** allows required actions
3. **Review CloudFormation** stack permissions

## 🔐 Security Best Practices

### Principle of Least Privilege

**Bucket Policy Refinements:** Instead of granting access to the entire Control Account root, consider restricting to specific Quilt roles:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "QuiltSpecificRoleAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::CONTROL-ACCOUNT-ID:role/QuiltLambdaRole",
                    "arn:aws:iam::CONTROL-ACCOUNT-ID:role/QuiltIndexerRole"
                ]
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-data-bucket",
                "arn:aws:s3:::your-data-bucket/*"
            ]
        }
    ]
}
```

### Network Security

**VPC Considerations:**

* ✅ **VPC Endpoints**: Use S3 VPC endpoints to keep traffic within AWS network
* ✅ **Security Groups**: Restrict Lambda function network access
* ✅ **NACLs**: Additional network-level controls if required

**Example VPC Endpoint Policy:**

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-data-bucket",
                "arn:aws:s3:::your-data-bucket/*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalAccount": ["CONTROL-ACCOUNT-ID"]
                }
            }
        }
    ]
}
```

### Monitoring and Auditing

**CloudWatch Alarms:** Set up monitoring for cross-account access:

```bash
# Create alarm for failed S3 access attempts
aws cloudwatch put-metric-alarm \
    --alarm-name "CrossAccountS3AccessFailures" \
    --alarm-description "Monitor failed cross-account S3 access" \
    --metric-name ErrorCount \
    --namespace AWS/S3 \
    --statistic Sum \
    --period 300 \
    --threshold 5 \
    --comparison-operator GreaterThanThreshold \
    --profile control-account
```

**CloudTrail Monitoring:** Monitor specific cross-account activities:

```json
{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AIDACKCEVSQ6C2EXAMPLE",
        "arn": "arn:aws:sts::CONTROL-ACCOUNT-ID:assumed-role/QuiltRole/QuiltLambda",
        "accountId": "CONTROL-ACCOUNT-ID"
    },
    "eventTime": "2024-08-26T10:30:00Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "GetObject",
    "resources": [
        {
            "ARN": "arn:aws:s3:::your-data-bucket/file.csv",
            "accountId": "DATA-ACCOUNT-ID"
        }
    ]
}
```

### Compliance Considerations

**Data Residency:**

* 🌍 **Regional Compliance**: Ensure both accounts are in compliant regions
* 📋 **Data Classification**: Tag buckets with appropriate data classification
* 🔒 **Encryption**: Enable S3 encryption with appropriate KMS keys

**Audit Requirements:**

* 📊 **Access Logging**: Enable S3 access logging for detailed audit trails
* 🔍 **Regular Reviews**: Periodically review cross-account permissions
* 📝 **Documentation**: Maintain documentation of cross-account relationships

## 🚀 Advanced Configurations

### Multi-Region Setup

For multi-region deployments:

```bash
# Replicate bucket policy across regions
for region in us-east-1 us-west-2 eu-west-1; do
    aws s3api put-bucket-policy \
        --bucket "your-data-bucket-${region}" \
        --policy file://bucket-policy.json \
        --region $region \
        --profile data-account
done
```

### Automated Policy Management

**CloudFormation Template for Bucket Policies:**

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Cross-account bucket policies for Quilt'

Parameters:
  ControlAccountId:
    Type: String
    Description: 'Control account ID where Quilt is deployed'
  
  DataBucketName:
    Type: String
    Description: 'Name of the data bucket'

Resources:
  CrossAccountBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref DataBucketName
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: QuiltCrossAccountAccess
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${ControlAccountId}:root'
            Action:
              - 's3:GetObject'
              - 's3:GetObjectAttributes'
              - 's3:ListBucket'
              - 's3:PutObject'
              - 's3:DeleteObject'
            Resource:
              - !Sub 'arn:aws:s3:::${DataBucketName}'
              - !Sub 'arn:aws:s3:::${DataBucketName}/*'
```

## 📚 Additional Resources

### AWS Documentation

* [**Cross-Account Access**](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html) - AWS IAM cross-account access patterns
* [**S3 Bucket Policies**](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html) - Comprehensive S3 policy guide
* [**CloudTrail Cross-Account**](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-sharing-logs.html) - CloudTrail log sharing

### Quilt-Specific Resources

* [**Quilt Admin API**](/quilt-platform-administrator/admin-1) - Programmatic bucket management
* [**EventBridge Integration**](/quilt-platform-administrator/advanced/eventbridge) - Alternative event routing
* [**Security Best Practices**](/quilt-platform-administrator/best-practices/good-practice) - General Quilt security guidance

### Tools and Scripts

* [**AWS CLI Reference**](https://docs.aws.amazon.com/cli/latest/reference/s3api/) - S3 API commands
* [**Policy Generator**](https://awspolicygen.s3.amazonaws.com/policygen.html) - AWS Policy Generator tool
* [**IAM Policy Simulator**](https://policysim.aws.amazon.com/) - Test policies before applying

## 📞 Support

**Need Help with Cross-Account Setup?**

* 📧 **Email**: <support@quilt.bio>
* 💬 **Slack**: [Quilt Community](https://slack.quilt.bio)
* 📖 **Documentation**: [Quilt Docs](https://docs.quilt.bio/)
* 🐛 **Issues**: [GitHub Issues](https://github.com/quiltdata/quilt/issues)

***

**Success!** You now have a secure, compliant cross-account Quilt deployment that separates your control plane from your data plane while maintaining full functionality.


# Enterprise Installs

This document covers advanced configuration and administration for Quilt Enterprise installations.

> For installation instructions, see [Installation](/quilt-python-sdk/installation). For architecture details, see [Architecture](/architecture).

## Health and Monitoring

To check the status of your Quilt stack after bring-up or update, check the stack health in the CloudFormation console.

### Elasticsearch Cluster

If you notice slow or incomplete search results, check the status of the Quilt Elasticsearch cluster. To find the Quilt search cluster from CloudFormation, click on the Quilt stack, then "Resources." Click on the "Search" resource.

If your cluster status is not "Green" (healthy), please contact Quilt support. Causes of unhealthy search clusters include:

* Running out of storage space
* High index rates (e.g., caused by adding or updating very large numbers of files in S3)

### Service Limits

This deployment does not require an increase in limits for your AWS Account.

### External Dependencies

In addition to containers running in Fargate, Quilt includes a set of AWS Lambda functions. These lambda functions are not scanned by AWS Marketplace. The [code for the lambda functions](https://github.com/quiltdata/quilt/tree/master/lambdas) is open-source and has been verified through an independent security audit.

## Advanced configuration

The default Quilt settings are adequate for most use cases. The following section covers advanced customization options.

### Setting the default role

**The Quilt admin must log in and set the default role** in order for new users to be able to sign up.

![](/files/bHdgRSmaXDxg8aV5abS6)

## Single sign-on (SSO)

### Google

You can enable users on your Google domain to sign in to Quilt. Refer to [Google's instructions on OAuth2 user agents](https://developers.google.com/identity/protocols/OAuth2UserAgent) and create authorization credentials to identify your Quilt stack to Google's OAuth 2.0 server.

![](/files/-MK0AF_GTeFmrpZEtl7r)

Copy the `Client ID` and `Client secret` to a safe place. Add `<QuiltWebHost>/oauth-callback` to *authorized redirect URIs*.

### Microsoft Entra ID (Azure Active Directory)

1. Go to [Microsoft Entra admin center](https://entra.microsoft.com) → **Microsoft Entra ID → Applications → App registrations → New registration**.
2. Name the app, select the supported account types, and click **Register**. Note the **Application (client) ID** and **Directory (tenant) ID**.
3. Go to **Authentication → Add a platform → Web**. Add the redirect URI `<QuiltWebHost>/oauth-callback`. Under **Implicit grant and hybrid flows**, enable **ID tokens** (required — login will fail without it). Click **Save**.
4. Go to **Certificates & secrets → New client secret**. Copy the **Value** immediately — it is not shown again. (Do not use the Secret ID.)
5. Go to **API permissions → Add a permission → Microsoft Graph → Delegated**. Add `openid`, `profile`, `email`, `offline_access`, and `User.Read`, then click **Grant admin consent**. Without admin consent, each user is typically prompted to grant these permissions on their first login; granting admin consent approves them tenant-wide (subject to your org's policies) and avoids end-user prompts.
6. Your `AzureBaseUrl` is `https://login.microsoftonline.com/<TENANT_ID>/v2.0`. Reference [Microsoft identity platform and OpenID Connect protocol](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc) and [National clouds](https://learn.microsoft.com/en-us/entra/identity-platform/authentication-national-cloud) for non-standard endpoints.

   > **`AzureBaseUrl` must end in `/v2.0`. Append it if missing.**
7. For SSO Permissions Mapping:

   * Create security groups in Entra and assign users.

   * In the app registration, go to **Token configuration → Add groups claim** (or **Edit** if it already exists) and configure it to emit **Group IDs** in the **ID token**.

   * Create a [configuration file](/quilt-platform-administrator/advanced/sso-permissions) to map Entra Group IDs to Quilt roles.

   > **Guest / multi-tenant users:** the `groups` claim carries only groups from the tenant that issues the token. Users who sign in as **B2B guests** (e.g. employees of one company logging into another company's tenant) will **not** receive their home-tenant security groups, so group-based mappings won't match for them. For these setups, prefer **App roles**: define roles on the app registration (**App roles**), ensure the roles are enabled, and assign users/groups to those roles. Entra includes assigned app roles in the ID token's `roles` claim; this is not configured through **Token configuration**. App roles live on the resource application, so they populate for guests too. Then map on the `roles` claim instead of `groups` (see [SSO permissions mapping](/quilt-platform-administrator/advanced/sso-permissions#which-claims-are-matched)). Note also that Quilt reads only the ID token, so the Entra **group overage** behavior (>\~200 groups) is not resolved — another reason to use app roles at scale.
8. Proceed to [Enabling SSO](#enabling-sso).

### Okta

1. Go to Okta > Admin > Applications > Applications

   ![](/files/eeoPmYqZFu79NzZdCa3Y)
2. Click `Create App Integration`. A new modal window opens.
3. Assign `Sign-in method` radio button to `OIDC - OpenID Connect`.
4. Assign `Application type` radio button to `Web Application`.

   ![](/files/1J0yM0Pr674Co2vq8asI)
5. Click the `Next` button.
6. Rename the default `App integration name` to `Quilt` or something distinctive for your organization to identify it.
7. Add the [Quilt logo](https://user-images.githubusercontent.com/1322715/198700580-da72bd8d-b460-4125-ba31-a246965e3de8.png) for user recognition.
8. Configure the new web app integration as follows:
   1. For `Grant type` check the following: `Authorization Code`, `Refresh Token`, and `Implicit (hybrid)`.

      > **Important:** `Refresh Token` must be checked. Without it, the Quilt registry cannot complete the sign-in flow and will return a 401 error.
   2. To the `Sign-in redirect URIs` add `<QuiltWebHost>/oauth-callback` URL.
   3. Leave the `Allow wildcard * in the login URI redirect` checkbox **unchecked** unless you need wildcard redirect URIs (e.g., for multiple subdomains). Note that wildcards only match a single subdomain level: `*.example.com` matches `app.example.com` but NOT `app.dev.example.com`.
   4. Optionally add to the `Sign-out redirect URIs` (if desired by your organization).
   5. **Uncheck "Enable immediate access"** (also called Federation Broker Mode). This setting is on by default and will cause SSO to fail with `access_denied — Identity Provider: Unknown`. When unchecked, you can assign users directly.
   6. For the `Assignments > Controlled Access` selection, choose the option desired by your organization. Ensure at least one user or group is assigned to the app.
9. Once you click the `Save` button you will have a new application integration to review.
   1. If it's undefined, update the `Initiate login URI` to your `<QuiltWebHost>` URL.
   2. Copy the `Client ID`, `Secret`, and `Base URL` to a safe place
10. **Configure the authentication policy.** Go to **Security > Authentication Policies** and check which policy your new app is assigned to. The default policy ("Any two factors") requires MFA, which will block sign-in unless all users have MFA enrolled. Create or use a policy that allows password-only authentication. When creating a new policy, also edit the **Catch-all Rule** — it defaults to 2 factor types.
11. Go to **Okta > Security > API > Authorization servers**
    1. You should see a `default` entry with the `Audience` value set to `api://default`, and an `Issuer URI` that looks like the following:

       ```
       <MY_COMPANY>.okta.com/oauth2/default
       ```
    2. Click on the `default` authorization server. Go to the **Access Policies** tab and ensure there is at least one **rule** that grants tokens for Authorization Code flow. A policy with no rules will silently deny all token requests, causing sign-in to fail.
    3. See [Okta authorization servers](https://developer.okta.com/docs/concepts/auth-servers/#which-authorization-server-should-you-use) for more.
12. Proceed to [Enabling SSO](#enabling-sso)

### OneLogin

1. Go to Administration > Applications > Custom Connectors
2. Click `New Connector`
   1. Name the connector *Quilt Connector* or something similar
   2. Set `Sign on method` to `OpenID Connect`
   3. Set `Login URL` to `<QuiltWebHost>/oauth-callback`
   4. Click "Save"
3. Go back to Applications > Custom Connectors
4. Click `Add App to Connector`
5. Save the app (be sure to save it for the Organization)
6. Go to Applications > Applications > *Your new app* > SSO

   1. Click SSO. Copy the `Client ID`, `ClientSecret` and `Issuer URL` to a safe place.
   2. "Application Type" should be set to `Web`.
   3. "Token Endpoint" should be set to `POST`.

   ![](/files/M8k0WwnBUOGzZPCnpX2x)
7. Add *Your new app* to the users who need to access Quilt:

   ![](/files/-MK0AF_OHVC7UYi8uNmz)
8. Proceed to [Enabling SSO](#enabling-sso).

### Enabling SSO

The SSO parameter names in your stack depend on how the CloudFormation template was built — not on whether you use the Console, CLI, or Terraform to deploy it. To determine which applies, look at your stack's parameters in CloudFormation:

* **`SingleSignOnProvider` dropdown present** → your stack uses single-provider SSO; follow the [single-provider instructions](#single-provider-sso) below.
* **`GoogleAuth`, `AzureAuth`, etc. present** → your stack uses multi-provider SSO; follow the [multi-provider instructions](#multi-provider-sso) below.

#### Single-provider SSO

Set `PasswordAuth` to `Enabled` in the Quilt template (AWS Console > CloudFormation > *Quilt stack* > Update > Use current template > Next > Specify stack details), then select your provider from the `SingleSignOnProvider` dropdown (Google, Okta, OneLogin, or Azure).

![](/files/Byqh2QrtyldSxeg0yKTt)

Use the following settings for the remaining parameters:

| CFT Parameter              | Google SSO    | Okta SSO  | OneLogin SSO | Azure SSO               |
| -------------------------- | ------------- | --------- | ------------ | ----------------------- |
| `SingleSignOnClientId`     | Client ID     | Client ID | Client ID    | Application (client) ID |
| `SingleSignOnClientSecret` | Client secret | Secret    | ClientSecret | Client secret Value     |
| `SingleSignOnBaseUrl`      | N/A           | Base URL  | Issuer URL   | AzureBaseUrl            |

> Be sure to set the [default role](#setting-the-default-role) as indicated above.

#### Multi-provider SSO

Stacks built with multi-provider SSO use per-provider parameters instead of the shared `SingleSignOnProvider` dropdown, allowing multiple providers to be enabled simultaneously. These parameters are passed the same way regardless of deployment method — Console, CLI, or Terraform. See [Authentication Examples](https://github.com/quiltdata/iac/blob/main/EXAMPLES.md#authentication-examples) for examples using the [Quilt IAC Terraform module](https://github.com/quiltdata/iac).

| Function      | Google               | Okta               | OneLogin               | Azure               |
| ------------- | -------------------- | ------------------ | ---------------------- | ------------------- |
| Enable        | `GoogleAuth`         | `OktaAuth`         | `OneLoginAuth`         | `AzureAuth`         |
| Client ID     | `GoogleClientId`     | `OktaClientId`     | `OneLoginClientId`     | `AzureClientId`     |
| Client Secret | `GoogleClientSecret` | `OktaClientSecret` | `OneLoginClientSecret` | `AzureClientSecret` |
| Base URL      | N/A                  | `OktaBaseUrl`      | `OneLoginBaseUrl`      | `AzureBaseUrl`      |

### Preparing an AWS Role for use with Quilt

These instructions document how to set up an existing role for use with Quilt. If the role you want to use doesn't exist yet, create it now. For guidance creating IAM roles, see: [IAM best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html), and the [Principle of Least Privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege)

Go to your Quilt stack in CloudFormation. Go to `Outputs`, then find `RegistryRoleARN` and copy its value. It should look something like this: `arn:aws:iam::000000000000:role/stackname-ecsTaskExecutionRole`.

Go to the IAM console and navigate to `Roles`. Select the role you want to use. Go to the `Trust Relationships` tab for the role, and select `Edit Trust Relationship`. The statement might look something like this:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    "... one or more statements"
  ]
}
```

Add an object to the beginning of the Statement array with the following contents:

```json
{
  "Effect": "Allow",
  "Principal": {
    "AWS": "$YOUR_REGISTRY_ROLE_ARN"
  },
  "Action": "sts:AssumeRole"
},
```

Note the comma after the object. Your trust relationship should now look something like this:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "$YOUR_REGISTRY_ROLE_ARN"
      },
      "Action": "sts:AssumeRole"
    },
    "... whatever was here before"
  ]
}
```

You can now configure a Quilt Role with this role (using the Catalog's Admin panel, or `quilt3.admin.create_role`).

### ManagedUserRoleExtraPolicies

The `ManagedUserRoleExtraPolicies` parameter allows you to add additional IAM policies to the managed user role. This is useful for granting additional permissions to users in your Quilt instance.

This parameter works in conjunction with Quilt policy configuration in the Admin panel. You need to:

1. Create the appropriate IAM policy
2. Add its ARN to `ManagedUserRoleExtraPolicies` (this step)
3. In the Admin panel, create a Quilt policy with the "Manually set ARN instead of configuring per-bucket permissions" option enabled, and enter the same policy ARN
4. Attach that Quilt policy to the managed (Source=Quilt) roles that should receive the additional permissions

In the AWS Console, go to CloudFormation > Your Quilt Stack -> Update -> Parameters and add the ARN of that IAM policy to `ManagedUserRoleExtraPolicies` at the bottom of the page:

![](/files/UHAU0KvehEnaWzCbgInO)

This parameter accepts a comma-separated list of policy ARNs.

**Note:** This parameter is needed only for managed (Source=Quilt) roles. Quilt scopes each managed role's session to the policies attached to it, so an added permission must be granted in both places: on the role (this parameter) and as an attached Quilt policy (steps 3-4). For custom (Source=Custom) roles, attach policies directly in IAM without this parameter.

### S3 buckets with Service-Side Encryption using Key Management Service (SSE-KMS)

In order for Quilt to access and index buckets encrypted with SSE-KMS, you must do three things:

1. Add KMS Key Usage to Quilt Permission Boundary
2. Add Quilt Principals to KMS Key Policy
3. Add KMS Key Access to a Source=Quilt Role

NOTE: This will not work with the default Source=Custom Roles.

#### 1. Add KMS Key Usage to Quilt Permission Boundary

By default, AWS does not allow anything in your account to access KMS. If you haven't done so already, create an IAM policy that explicitly enables KMS access.

```json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "kms:Decrypt",
      "kms:GenerateDataKey"
    ],
    "Resource": "arn:aws:kms:us-west-2:111122223333:key/*"
  }
}
```

Then add this policy to the `ManagedUserRoleExtraPolicies` as described above.

#### 2. Add Quilt Principals to KMS Key Policy

In order for Quilt to index buckets with SSE-KMS, you must add certain principals to the corresponding key policy. Go to CloudFormation > Your Quilt Stack > Resources and look for IAM roles with the following logical IDs:

* `AmazonECSTaskExecutionRole`
* `PkgEventsRole`
* `PkgSelectLambdaRole`
* `SearchHandlerRole`
* `T4BucketReadRole`
* `T4BucketWriteRole`

Note the ARN for each of the above logical IDs and add an Allow statement similar to the following to the KMS key policy:

```json
{
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "<RoleARN-1>",
            ...
            "<RoleARN-N>"
        ]
    },
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
    ],
    "Resource": "*"
}
```

#### 3. Add KMS Key Access to Quilt Role

Finally, you need create a restricted policy that gives a Quilt role access to the keys for specific buckets, e.g:

```json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "kms:Decrypt",
      "kms:GenerateDataKey"
    ],
    "Resource": [
      "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
      "arn:aws:kms:us-west-2:111122223333:key/0987dcba-09fe-87dc-65ba-ab0987654321"
    ]
  }
}
```

You can now create a Quilt Policy from this policy using the Catalog's Admin panel. Afterwards, you can attach that Policy to a user-defined Quilt Role (which has Source=Quilt in the Roles panel, as opposed to system-defined Source=Custom Roles).

## Backup and Recovery

All data and metadata in Quilt is stored in S3. S3 data is automatically backed up (replicated across multiple available zones). To protect against accidental deletion or overwriting of data, we strongly recommend enabling object versioning for all S3 buckets connected to Quilt.

No data will be lost if a Quilt stack goes down. The Quilt search indexes will be automatically rebuilt when buckets are added to a new stack.

### Region Failure

To protect against data loss in the event of a region failure, enable [S3 Bucket Replication](https://aws.amazon.com/s3/features/replication/) on all S3 buckets.

The time to restore varies with storage needs, but a <2-hour recovery time objective (RTO) and <15 minute recovery point objective (RPO) are generally possible.

To restore Quilt in your backup region:

1. Create a new Quilt stack from the same CloudFormation template in the backup region.
2. Connect the replica buckets (in the backup region) to your Quilt stack. In the Quilt catalog, select "Users and Buckets"->"Buckets" and enter the bucket information.

## Emergency Maintenance

See [Troubleshooting](/quilt-python-sdk/more/troubleshooting)

## Support

Support is available to all Quilt customers by:

* online chat (in the Quilt catalog)
* email to <support@quilt.bio>
* [Slack](https://slack.quilt.bio)

Quilt guarantees response to support issues according to the following SLAs for Quilt Business and Quilt Enterprise customers.

### Quilt Business

|            | Initial Response | Temporary Resolution |
| ---------- | ---------------- | -------------------- |
| Priority 1 | 1 business day   | 3 business days      |
| Priority 2 | 2 business days  | 5 business days      |
| Priority 3 | 3 business days  | N/A                  |

### Quilt Enterprise

|            | Initial Response | Temporary Resolution |
| ---------- | ---------------- | -------------------- |
| Priority 1 | 4 business hours | 1 business day       |
| Priority 2 | 1 business day   | 2 business days      |
| Priority 3 | 1 business days  | N/A                  |

### Definitions

* *Business Day* means Monday through Friday (PST), excluding holidays observed by Quilt Data.
* *Business Hours* means 8:00 a.m. to 7:00 p.m. (PST) on Business Days.
* *Priority 1* means a critical problem with the Software in which the Software inoperable;
* *Priority 2* means a problem with the Software in which the Software is severely limited or degraded, major functions are not performing properly, and the situation is causing a significant impact to Customer's operations or productivity;
* *Priority 3* means a minor or cosmetic problem with the Software in which any of the following occur: the problem is an irritant, affects nonessential functions, or has minimal impact to business operations; the problem is localized or has isolated impact; the problem is an operational nuisance; the problem results in documentation errors; or the problem is any other problem that is not a Priority 1 or a Priority 2, but is otherwise a failure of the Software to conform to the Documentation or Specifications;
* *Temporary Resolution* means a temporary fix or patch that has been implemented and incorporated into the Software by Quilt Data to restore Software functionality.


# quilt3.admin Python API

## quilt3.admin.types

### Permission(bucket: str, level: quilt3.\_graphql\_client.enums.BucketPermissionLevel) -> None <a href="#permission" id="permission"></a>

### PolicySummary(id: str, title: str, arn: str, managed: bool, permissions: list\[quilt3.admin.types.Permission]) -> None <a href="#policysummary" id="policysummary"></a>

Policy without back-references to roles (avoids circular nesting).

### ManagedRole(id: str, name: str, arn: str, policies: list\[quilt3.admin.types.PolicySummary], permissions: list\[quilt3.admin.types.Permission], typename\_\_: Literal\['ManagedRole']) -> None <a href="#managedrole" id="managedrole"></a>

### UnmanagedRole(id: str, name: str, arn: str, typename\_\_: Literal\['UnmanagedRole']) -> None <a href="#unmanagedrole" id="unmanagedrole"></a>

### Policy(id: str, title: str, arn: str, managed: bool, permissions: list\[quilt3.admin.types.Permission], roles: list\[quilt3.admin.types.ManagedRole]) -> None <a href="#policy" id="policy"></a>

### User(name: str, email: str, date\_joined: datetime.datetime, last\_login: datetime.datetime, is\_active: bool, is\_admin: bool, is\_sso\_only: bool, is\_service: bool, role: Annotated\[quilt3.admin.types.ManagedRole | quilt3.admin.types.UnmanagedRole, FieldInfo(annotation=NoneType, required=True, discriminator='typename\_\_')] | None, extra\_roles: list\[typing.Annotated\[quilt3.admin.types.ManagedRole | quilt3.admin.types.UnmanagedRole, FieldInfo(annotation=NoneType, required=True, discriminator='typename\_\_')]]) -> None <a href="#user" id="user"></a>

### SSOConfig(text: str, timestamp: datetime.datetime, uploader: quilt3.admin.types.User) -> None <a href="#ssoconfig" id="ssoconfig"></a>

### TabulatorTable(name: str, config: str) -> None <a href="#tabulatortable" id="tabulatortable"></a>

### Bucket(name: str, title: str, icon\_url: str | None, description: str | None, overview\_url: str | None, tags: list\[str] | None, relevance\_score: int, last\_indexed: datetime.datetime | None, sns\_notification\_arn: str | None, scanner\_parallel\_shards\_depth: int | None, skip\_meta\_data\_indexing: bool | None, file\_extensions\_to\_index: list\[str] | None, index\_content\_bytes: int | None, prefixes: list\[str]) -> None <a href="#bucket" id="bucket"></a>

## quilt3.admin.api\_keys

Admin API for managing API keys.

### list(email: str | None = None, key\_name: str | None = None, fingerprint: str | None = None, status: Literal\['ACTIVE', 'EXPIRED'] | None = None) -> List\[quilt3.api\_keys.APIKey] <a href="#list" id="list"></a>

List API keys. Optionally filter by user email, key name, fingerprint, or status.

**Arguments**

* **email**: Filter by user email.
* **key\_name**: Filter by key name.
* **fingerprint**: Filter by key fingerprint.
* **status**: Filter by "ACTIVE" or "EXPIRED". None returns all.

**Returns**

List of API keys matching the filters.

### get(id: str) -> quilt3.api\_keys.APIKey | None <a href="#get" id="get"></a>

Get a specific API key by ID.

**Arguments**

* **id**: The API key ID.

**Returns**

The API key, or None if not found.

### revoke(id: str) -> None <a href="#revoke" id="revoke"></a>

Revoke an API key.

**Arguments**

* **id**: The API key ID to revoke.

**Raises**

* `Quilt3AdminError`: If the operation fails.

## quilt3.admin.buckets

### get(name: str) -> quilt3.admin.types.Bucket | None <a href="#get" id="get"></a>

Get a specific bucket configuration from the registry. Returns `None` if the bucket does not exist.

**Arguments**

* **name**: Name of the bucket to get.

### list() -> list\[quilt3.admin.types.Bucket] <a href="#list" id="list"></a>

List all bucket configurations in the registry.

### add(name: str, title: str, \*, description: str | None = None, icon\_url: str | None = None, overview\_url: str | None = None, tags: List\[str] | None = None, relevance\_score: int | None = None, sns\_notification\_arn: str | None = None, scanner\_parallel\_shards\_depth: int | None = None, skip\_meta\_data\_indexing: bool | None = None, file\_extensions\_to\_index: List\[str] | None = None, index\_content\_bytes: int | None = None, delay\_scan: bool | None = None, browsable: bool | None = None, prefixes: List\[str] | None = None) -> quilt3.admin.types.Bucket <a href="#add" id="add"></a>

Add a new bucket to the registry.

**Arguments**

* **name**: S3 bucket name.
* **title**: Display title for the bucket.
* **description**: Optional description.
* **icon\_url**: Optional URL for bucket icon.
* **overview\_url**: Optional URL for bucket overview page.
* **tags**: Optional list of tags.
* **relevance\_score**: Optional relevance score for bucket ordering.
* **sns\_notification\_arn**: Optional SNS topic ARN for notifications.
* **scanner\_parallel\_shards\_depth**: Optional depth for parallel scanning.
* **skip\_meta\_data\_indexing**: If True, skip metadata indexing.
* **file\_extensions\_to\_index**: Optional list of file extensions to index content.
* **index\_content\_bytes**: Optional max bytes of content to index.
* **delay\_scan**: If True, delay initial bucket scan.
* **browsable**: If True, bucket is browsable.
* **prefixes**: Optional list of S3 prefixes to scope bucket access to. If provided, only these prefixes will be indexed and verified for access.

### update(name: str, title: str, \*, description: str | None = None, icon\_url: str | None = None, overview\_url: str | None = None, tags: List\[str] | None = None, relevance\_score: int | None = None, sns\_notification\_arn: str | None = None, scanner\_parallel\_shards\_depth: int | None = None, skip\_meta\_data\_indexing: bool | None = None, file\_extensions\_to\_index: List\[str] | None = None, index\_content\_bytes: int | None = None, browsable: bool | None = None, prefixes: List\[str] | None = None) -> quilt3.admin.types.Bucket <a href="#update" id="update"></a>

Update an existing bucket configuration.

**Arguments**

* **name**: S3 bucket name.
* **title**: Display title for the bucket.
* **description**: Optional description.
* **icon\_url**: Optional URL for bucket icon.
* **overview\_url**: Optional URL for bucket overview page.
* **tags**: Optional list of tags.
* **relevance\_score**: Optional relevance score for bucket ordering.
* **sns\_notification\_arn**: Optional SNS topic ARN for notifications.
* **scanner\_parallel\_shards\_depth**: Optional depth for parallel scanning.
* **skip\_meta\_data\_indexing**: If True, skip metadata indexing.
* **file\_extensions\_to\_index**: Optional list of file extensions to index content.
* **index\_content\_bytes**: Optional max bytes of content to index.
* **browsable**: If True, bucket is browsable.
* **prefixes**: Optional list of S3 prefixes to scope bucket access to. If provided, only these prefixes will be indexed and verified for access. Changing prefixes will trigger permission re-verification.

### remove(name: str) -> None <a href="#remove" id="remove"></a>

Remove a bucket from the registry.

**Arguments**

* **name**: Name of the bucket to remove.

## quilt3.admin.roles

### get(id\_or\_name: str) -> quilt3.admin.types.ManagedRole | quilt3.admin.types.UnmanagedRole | None <a href="#get" id="get"></a>

Get a role by ID or name. Return `None` if the role does not exist.

**Arguments**

* **id\_or\_name**: Role ID or name.

### get\_default() -> quilt3.admin.types.ManagedRole | quilt3.admin.types.UnmanagedRole | None <a href="#get_default" id="get_default"></a>

Get the default role from the registry. Return `None` if no default role is set.

### list() -> List\[quilt3.admin.types.ManagedRole | quilt3.admin.types.UnmanagedRole] <a href="#list" id="list"></a>

Get a list of all roles in the registry.

### create\_managed(name: str, policies: List\[str] = ()) -> quilt3.admin.types.ManagedRole <a href="#create_managed" id="create_managed"></a>

Create a managed role in the registry.

**Arguments**

* **name**: Role name.
* **policies**: Policy IDs to attach to the role.

### create\_unmanaged(name: str, arn: str) -> quilt3.admin.types.UnmanagedRole <a href="#create_unmanaged" id="create_unmanaged"></a>

Create an unmanaged role in the registry.

**Arguments**

* **name**: Role name.
* **arn**: Existing IAM role ARN.

### update\_managed(id\_or\_name: str, \*, name: str, policies: List\[str]) -> quilt3.admin.types.ManagedRole <a href="#update_managed" id="update_managed"></a>

Update a managed role in the registry (full replacement).

**Arguments**

* **id\_or\_name**: Role ID or name.
* **name**: New role name.
* **policies**: Policy IDs to attach to the role.

### update\_unmanaged(id\_or\_name: str, \*, name: str, arn: str) -> quilt3.admin.types.UnmanagedRole <a href="#update_unmanaged" id="update_unmanaged"></a>

Update an unmanaged role in the registry (full replacement).

**Arguments**

* **id\_or\_name**: Role ID or name.
* **name**: New role name.
* **arn**: Existing IAM role ARN.

### patch\_managed(id\_or\_name: str, \*, name: str | None = None, policies: List\[str] | None = None) -> quilt3.admin.types.ManagedRole <a href="#patch_managed" id="patch_managed"></a>

Partially update a managed role — only specified fields are changed.

**Arguments**

* **id\_or\_name**: Role ID or name.
* **name**: New role name (keeps current if not specified).
* **policies**: Policy IDs to attach (keeps current if not specified).

### patch\_unmanaged(id\_or\_name: str, \*, name: str | None = None, arn: str | None = None) -> quilt3.admin.types.UnmanagedRole <a href="#patch_unmanaged" id="patch_unmanaged"></a>

Partially update an unmanaged role — only specified fields are changed.

**Arguments**

* **id\_or\_name**: Role ID or name.
* **name**: New role name (keeps current if not specified).
* **arn**: New IAM role ARN (keeps current if not specified).

### delete(id\_or\_name: str) -> None <a href="#delete" id="delete"></a>

Delete a role from the registry.

**Arguments**

* **id\_or\_name**: Role ID or name.

### set\_default(id\_or\_name: str) -> quilt3.admin.types.ManagedRole | quilt3.admin.types.UnmanagedRole <a href="#set_default" id="set_default"></a>

Set the default role in the registry.

**Arguments**

* **id\_or\_name**: Role ID or name.

## quilt3.admin.users

### get(name: str) -> quilt3.admin.types.User | None <a href="#get" id="get"></a>

Get a specific user from the registry. Return `None` if the user does not exist.

**Arguments**

* **name**: Username of user to get.

### list() -> List\[quilt3.admin.types.User] <a href="#list" id="list"></a>

Get a list of all users in the registry.

### create(name: str, email: str, role: str, extra\_roles: List\[str] | None = None) -> quilt3.admin.types.User <a href="#create" id="create"></a>

Create a new user in the registry.

**Arguments**

* **name**: Username of user to create.
* **email**: Email of user to create.
* **role**: Active role of the user.
* **extra\_roles**: Additional roles to assign to the user.

### delete(name: str) -> None <a href="#delete" id="delete"></a>

Delete user from the registry.

**Arguments**

* **name**: Username of user to delete.

### set\_email(name: str, email: str) -> quilt3.admin.types.User <a href="#set_email" id="set_email"></a>

Set the email for a user.

**Arguments**

* **name**: Username of user to update.
* **email**: Email to set for the user.

### set\_admin(name: str, admin: bool) -> quilt3.admin.types.User <a href="#set_admin" id="set_admin"></a>

Set the admin status for a user.

**Arguments**

* **name**: Username of user to update.
* **admin**: Admin status to set for the user.

### set\_active(name: str, active: bool) -> quilt3.admin.types.User <a href="#set_active" id="set_active"></a>

Set the active status for a user.

**Arguments**

* **name**: Username of user to update.
* **active**: Active status to set for the user.

### reset\_password(name: str) -> None <a href="#reset_password" id="reset_password"></a>

Reset the password for a user.

**Arguments**

* **name**: Username of user to update.

### set\_role(name: str, role: str, extra\_roles: List\[str] | None = None, \*, append: bool = False) -> quilt3.admin.types.User <a href="#set_role" id="set_role"></a>

Set the active and extra roles for a user.

**Arguments**

* **name**: Username of user to update.
* **role**: Role to be set as the active role.
* **extra\_roles**: Additional roles to assign to the user.
* **append**: If True, append the extra roles to the existing roles. If False, replace the existing roles.

### add\_roles(name: str, roles: List\[str]) -> quilt3.admin.types.User <a href="#add_roles" id="add_roles"></a>

Add roles to a user.

**Arguments**

* **name**: Username of user to update.
* **roles**: Roles to add to the user.

### remove\_roles(name: str, roles: List\[str], fallback: str | None = None) -> quilt3.admin.types.User <a href="#remove_roles" id="remove_roles"></a>

Remove roles from a user.

**Arguments**

* **name**: Username of user to update.
* **roles**: Roles to remove from the user.
* **fallback**: If set, the role to assign to the user if the active role is removed.

## quilt3.admin.sso\_config

### get() -> quilt3.admin.types.SSOConfig | None <a href="#get" id="get"></a>

Get the current SSO configuration.

### set(config: str | None) -> quilt3.admin.types.SSOConfig | None <a href="#set" id="set"></a>

Set the SSO configuration. Pass `None` to remove SSO configuration.

## quilt3.admin.tabulator

### list\_tables(bucket\_name: str) -> list\[quilt3.admin.types.TabulatorTable] <a href="#list_tables" id="list_tables"></a>

List all tabulator tables in a bucket.

### set\_table(bucket\_name: str, table\_name: str, config: str | None) -> None <a href="#set_table" id="set_table"></a>

Set the tabulator table configuration. Pass `None` to remove the table.

### rename\_table(bucket\_name: str, table\_name: str, new\_table\_name: str) -> None <a href="#rename_table" id="rename_table"></a>

Rename tabulator table.

### get\_open\_query() -> bool <a href="#get_open_query" id="get_open_query"></a>

Get the **open query** status.

### set\_open\_query(enabled: bool) -> None <a href="#set_open_query" id="set_open_query"></a>

Set the **open query** status.


# Advanced


# Package Events

When a package is created or revised in a Quilt stack bucket, the stack emits a `package-revision` event on the default [EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) bus. These events have the following structure:

```json
{
    "version": "0",
    "id": "6425eb6a-9627-e6a1-2ae8-9d2d8883dc74",
    "detail-type": "package-revision",
    "source": "com.quiltdata",
    "account": "012345678901",
    "time": "2024-04-25T14:46:51Z",
    "region": "us-east-1",
    "resources": [],
    "detail": {
        "version": "0.1",
        "type": "created",
        "bucket": "example",
        "handle": "some/package",
        "topHash": "a0fddace2eb2fd91faa697d237a5dbdcfa77f0fd38ca8b4c850dbd93d142ee69"
    }
}
```

You can create an [EventBridge rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) similar to the following to respond to `package-revision` events:

## Example: send email on package creation

```yaml
Description: >
  Demonstrate how to respond to package events.

  Modify `EventBridgeRule` to customize event processing.

  See https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html
  for event pattern syntax.

  See https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html
  for rule targets.
Parameters:
  PackageBucket:
    Type: String
  PackageNamePrefix:
    Type: String
    Description: Leave empty to match every package.
  EmailToSubscribe:
    Type: String
    Description: >
      Confirm subscription over email to receive a copy of package events
      that occur under `PackageNamePrefix` in `PackageBucket`.

Resources:
  EventBridgeRule:
    Type: AWS::Events::Rule
    Properties:
      EventPattern:
        source:
          - "com.quiltdata"
        detail-type:
          - "package-revision"
        detail:
          type:
            - "created"
          handle:
            - prefix: !Ref PackageNamePrefix
          bucket:
            - !Ref PackageBucket
      Targets:
        - Arn: !Ref SNSTopic
          Id: "OpsTopic"

  SNSTopic:
    Type: AWS::SNS::Topic

  SNSTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      PolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sns:Publish'
            Resource: '*'
      Topics:
        - !Ref SNSTopic

  SNSTopicSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Endpoint: !Ref EmailToSubscribe
      Protocol: email
      TopicArn: !Ref SNSTopic
```


# Private Endpoints

> This page describes a feature that is not enabled by default. You can ask your Quilt account manager to enable it.

## Data perimeters

A **data perimeter** ensures that only **trusted principals** on **expected networks** can access **trusted resources**.

For example, you may wish to ensure that only private IPs can access data in Amazon S3, Quilt's primary data store. Such a data perimeter strengthens your security by ensuring that S3 credentials alone are not sufficient to access data in Amazon S3.

In order for Quilt to function properly with expected private networks, your Quilt account manager must configure your CloudFormation stack to run its services (e.g. Lambda, API Gateway) on private IPs.

Additionally you will need to create and configure the following AWS resources, or equivalents depending on your network architecture:

1. Create an [interface VPC endpoint](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-apis.html) for Amazon API Gateway.

   This interface endpoint is used by Quilt's backend services to keep network traffic private to your VPC. Enter the VPC endpoint ID in your CloudFormation template as the `ApiGatewayVPCEndpointId` template parameter.

   > Note that, even if you do not use private endpoints for Quilt services, traffic between your VPC and AWS services [does not leave the AWS network backbone](https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/centralized-access-to-vpc-private-endpoints.html).
2. Create an Amazon S3 Gateway endpoint.

   [S3 gateway endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) facilitate access to S3 from the VPC that you run Quilt in.

   > AWS permits one [S3 gateway endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) per VPC per region If you wish to connect buckets from multiple stacks to Quilt, a transit VPC or similar design is required.
3. Provide a NAT gateway (or similar).

   Quilt's private endpoints require access to public Internet services like Amazon ECR and Amazon SNS.

   > See [Amazon's guide on NAT gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html#nat-gateway-creating).
4. Test and apply policies to enforce your data perimeter.

   We recommend that you test an individual bucket policy on a clean bucket to prevent inadvertent loss of access to your data. Once Quilt and other services are able to access this experimental bucket as expected, you can graduate to a more comprehensive [Service Control Policy (SCP)](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html) to implement your data perimeter at the organization level. SCPs define **guardrails** on any action that the account's administrator delegates to the IAM users and roles in the account.

   > See ["Enabling and disabling policy types"](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_enable-disable.html) for more on SCPs.

## Example Service Control Policy

The following SCP establishes a data perimeter around all in-organization Amazon S3 buckets prefixed with the string "quilt" such that only principals with *one or more* of the following characteristics can access data in Amazon S3.

1. The source VPC is either `vpc-LOCAL` or `vpc-VPN`.
2. The principal on the request has the `NetworkRestrictedExempt` tag.

   > Use this tag as a failsafe entry point when testing and debugging your SCP
3. The request comes from a specific IP range (e.g. `192.0.2.0 - 192.0.2.255`).
4. The principal is an [AWS service principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#principal-services).

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PreventUnexpectedNetworksButAllowAWSServices",
            "Effect": "Deny",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::quilt*",
                "arn:aws:s3:::quilt*/*"
            ],
            "Condition": {
                "StringNotEqualsIfExists": {
                    "aws:SourceVpc": [
                        "vpc-LOCAL",
                        "vpc-VPN"
                    ]
                },
                "Null": {
                    "aws:PrincipalTag/NetworkRestrictedExempt": "true"
                },
                "NotIpAddressIfExists": {
                    "aws:SourceIp": [
                        "192.0.2.0/24",
                        "203.0.113.0/24"
                    ]
                },
                "Bool": {
                    "aws:PrincipalIsAWSService": "false",
                    "aws:ViaAWSService": "false"
                }
            }
        }
    ]
}
```

## Verifying your setup

If you have [Quilt canaries](/quilt-platform-administrator/best-practices/good-practice) enabled, check the catalog admin panel to ensure that they are functioning.

## Considerations

1. There can only be one S3 gateway endpoint per VPC.
2. Your S3 buckets must be in the same region as the gateway endpoint.
3. Routing traffic on private networks may incur Transit Gateway, inter-VPC, and Interface Endpoint charges.
4. The DNS of any VPN clients must assign AWS global and regional S3 service names to the Interface Endpoint IP addresses.

## References

* [Choosing your VPC Endpoint Strategy for Amazon S3](https://aws.amazon.com/blogs/architecture/choosing-your-vpc-endpoint-strategy-for-amazon-s3/)
* [Secure hybrid access to Amazon S3 using AWS PrivateLink](https://aws.amazon.com/blogs/networking-and-content-delivery/secure-hybrid-access-to-amazon-s3-using-aws-privatelink/)
* [Establishing a Data Perimeter](https://aws.amazon.com/blogs/security/establishing-a-data-perimeter-on-aws/)
* [Building a Data Perimeter Whitepaper](https://docs.aws.amazon.com/whitepapers/latest/building-a-data-perimeter-on-aws/building-a-data-perimeter-on-aws.html)


# Restrict Access by Bucket Prefix

It is possible to use custom roles and policies in the Quilt Catalog to limit access to specific folders in an S3 bucket. Nevertheless, this approach has limitations and is therefore not recommended for high-security data where not just the contents *but the names of files and folders are sensitive*.

## Pre-requisites

* IAM policies and roles (see below for an example)
* Quilt Catalog Admin status
* For cross-account roles, you must have a recent version of the Quilt Stack (1-Feb-2023 or later) so that the Quilt registry has sufficient permissions to assume cross-account roles on behalf of users
* Ask your Quilt Account Manager to **enable "prefix-aware search"** for your stack in order for search to hide the objects from unauthorized users in the search results.

## Limitations and workarounds

* Roles for users of the Quilt Catalog's Bucket tab must have **full ListBucket permissions**, whether or not they are allowed to access all folders and objects. Catalog users who click on a prefix or object that they are not permitted to access will see an *Access Denied* message.

  * Alternatively, you can [hide the Bucket tab completely](/quilt-platform-administrator/admin#show-and-hide-features-in-the-quilt-catalog) and leave users access to the Package tab.

  > IAM is not designed as a filter for browsing S3. ListBucket will return a 403 error for the root of bucket if users do not have full permissions (currently incompatible with the Quilt Catalog)
* Similar to prefixes (above), Quilt Packages that reference prefixes that users cannot access via IAM will reveal package-relative file names and object-level *Quilt* metadata, but will neither reveal S3 object metadata nor S3 object contents. Clicking on a package entry in the Catalog that has a physical key that the user is not allowed to access will display an *Access Denied* message.
* Prefix-aware search performs a head request on every object result; this may slow search performance in the Catalog
* Number of search results is not available with prefix-aware search

## Requirements and recommendations

* Manage all Quilt roles and policies for prefixes [with the Quilt Catalog Admin Panel](/quilt-platform-administrator/admin#users-and-roles)
* Provided that you use roles created in the Quilt Catalog Admin Panel **you do not need to, and should not,** insert a trust relationship into your roles by hand.

> In certain rare circumstances (e.g. upgrading from an older Quilt stack for cross-account role assumption) if Quilt is not working as expected **you can rename, remove, or add a Quilt managed role** in order to force the Quilt stack to update its permissions state.

## Example

You can attach a "custom" policy to a role with the Quilt Catalog Admin Panel similar to the following:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectTagging",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:ListBucket",
                "s3:ListBucketVersions",
                "s3:RestoreObject",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:PutObject",
                "s3:PutObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET>",
                "arn:aws:s3:::<BUCKET>/.quilt/*",
                "arn:aws:s3:::<BUCKET>/<PREFIX>/*"
            ]
        }
    ]
}
```

> The `.quilt` folder is where Quilt Package Manifests are stored for all packages in a bucket registry. Users must have access to this directory to view Packages.


# S3 Events via EventBridge

When using Quilt alongside other AWS services that consume S3 events (like FSx, Lambda triggers, or custom applications), you may encounter conflicts because **S3 only allows one event notification configuration per bucket**. This guide shows you how to resolve these conflicts using AWS EventBridge.

## 🎯 Understanding the Problem

### How Quilt Uses S3 Events

By default, Quilt automatically creates S3 Event Notifications to:

* Keep its managed Elasticsearch index up-to-date
* Track changes to bucket contents in real-time
* Maintain package metadata and search functionality

**Default Quilt Event Flow:**

```
S3 Bucket → S3 Event Notification → SNS Topic → SQS Queue → Lambda → Elasticsearch
```

### The S3 Event Limitation

**AWS S3 Limitation**: Each S3 bucket can only have **one event notification configuration**. This means:

❌ **This Won't Work:**

```
S3 Bucket ──┬── Quilt Event Notification
            └── FSx Event Notification     ← CONFLICT!
```

✅ **This Will Work:**

```
S3 Bucket → EventBridge → ┬── Quilt SNS Topic
                          └── FSx Event Handler
```

## 🛠️ Solution Options

### Option 1: SNS Fanout (Recommended)

Use SNS to distribute events to multiple consumers:

* **Best for**: Multiple AWS services needing S3 events
* **Complexity**: Medium
* **Reliability**: High
* **Guide**: [AWS Fanout Pattern](https://aws.amazon.com/blogs/compute/fanout-s3-event-notifications-to-multiple-endpoints/)

### Option 2: EventBridge Routing (This Guide)

Use EventBridge to create synthetic S3 events:

* **Best for**: Complex event routing and transformation
* **Complexity**: Medium-High
* **Reliability**: High
* **Flexibility**: Highest

### Option 3: Just-in-Time Resources

Spin up resources only when needed:

* **Best for**: Batch processing workloads
* **Complexity**: Low
* **Cost**: Lowest
* **Limitation**: Not suitable for real-time use cases

## 🚀 EventBridge Implementation Guide

This section provides a complete step-by-step guide to set up EventBridge routing for S3 events to resolve conflicts between Quilt and other services.

### Prerequisites

Before starting, ensure you have:

* ✅ AWS CLI or Console access with appropriate permissions
* ✅ A Quilt deployment already running
* ✅ The S3 bucket you want to add to Quilt
* ✅ CloudTrail enabled for the bucket (Quilt requirement)

### Step-by-Step Implementation

#### Step 1: Create SNS Topic

Create an SNS topic in the **same region** as your S3 bucket:

```bash
# Using AWS CLI
aws sns create-topic \
    --name quilt-eventbridge-notifications \
    --region us-east-1

# Note the TopicArn from the response
```

**Console Steps:**

1. Navigate to **SNS Console** → **Topics** → **Create topic**
2. **Type**: Standard
3. **Name**: `quilt-eventbridge-notifications`
4. **Region**: Same as your S3 bucket
5. Click **Create topic** and note the ARN

#### Step 2: Verify CloudTrail Configuration

Quilt requires CloudTrail for S3 data events. Check your CloudFormation stack:

**Option A: Quilt-Managed Trail**

* Go to **CloudFormation** → **Your Quilt Stack** → **Resources**
* Look for a CloudTrail resource
* Quilt will automatically add your bucket to this trail

**Option B: Existing Trail**

* Go to **CloudFormation** → **Your Quilt Stack** → **Parameters**
* Find the CloudTrail bucket parameter
* Manually add your bucket to the existing trail in CloudTrail console

#### Step 3: Create EventBridge Rule

Create an EventBridge rule to capture S3 events:

**Console Steps:**

1. Navigate to **EventBridge Console** → **Rules** → **Create rule**
2. **Name**: `quilt-s3-events-rule`
3. **Event bus**: default
4. **Rule type**: Rule with an event pattern

#### Step 4: Configure Event Pattern

Set up the event pattern to capture S3 operations:

**Event source**: AWS services **AWS service**: Simple Storage Service (S3) **Event type**: Specific operation(s)

**Select these operations:**

* ✅ `PutObject`
* ✅ `CopyObject`
* ✅ `CompleteMultipartUpload`
* ✅ `DeleteObject`
* ✅ `DeleteObjects`

**Bucket specification:**

* Select **Specific bucket(s) by name**
* Enter your bucket name: `your-bucket-name`

**Example Event Pattern JSON:**

```json
{
  "source": ["aws.s3"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["s3.amazonaws.com"],
    "eventName": [
      "PutObject",
      "CopyObject", 
      "CompleteMultipartUpload",
      "DeleteObject",
      "DeleteObjects"
    ],
    "requestParameters": {
      "bucketName": ["your-bucket-name"]
    }
  }
}
```

![Event Pattern Configuration](/files/-MRHLywKHZ-gt1xVZitF)

#### Step 5: Configure Event Target

Set the SNS topic as the target for EventBridge events:

1. **Target type**: AWS service
2. **Select a target**: SNS topic
3. **Topic**: Select the SNS topic created in Step 1

![Event Target Configuration](/files/-MRHLywLh6S6ol_ptXyU)

#### Step 6: Set Up Input Transformer

Configure the input transformer to convert EventBridge events to S3 event format:

**Input Path:**

```json
{
  "awsRegion": "$.detail.awsRegion",
  "bucketName": "$.detail.requestParameters.bucketName", 
  "eventName": "$.detail.eventName",
  "eventTime": "$.detail.eventTime",
  "isDeleteMarker": "$.detail.responseElements.x-amz-delete-marker",
  "key": "$.detail.requestParameters.key",
  "versionId": "$.detail.responseElements.x-amz-version-id"
}
```

**Input Template:**

```json
{
  "Records": [
    {
      "awsRegion": <awsRegion>,
      "eventName": <eventName>, 
      "eventTime": <eventTime>,
      "s3": {
        "bucket": {
          "name": <bucketName>
        },
        "object": {
          "eTag": "",
          "isDeleteMarker": <isDeleteMarker>,
          "key": <key>,
          "versionId": <versionId>
        }
      }
    }
  ]
}
```

#### Step 7: Save and Test the Rule

1. Click **Create rule** to save the EventBridge configuration
2. Test by uploading a file to your S3 bucket
3. Check CloudWatch Logs for the EventBridge rule to verify events are being processed

#### Step 8: Configure Quilt

Add the bucket to Quilt using the SNS topic:

1. Open **Quilt Admin Panel** → **Buckets**
2. Click **Add Bucket** or edit existing bucket
3. **Bucket Name**: `your-bucket-name`
4. **SNS Topic ARN**: Paste the ARN from Step 1
5. **Important**: Leave S3 Event Notifications **disabled**

![Quilt EventBridge Configuration](/files/-MRHLywQTkhzijiQm8sB)

#### Step 9: Initial Indexing

Perform initial bucket indexing:

1. In Quilt Admin Panel, find your bucket
2. Click **Re-Index and Repair**
3. **⚠️ IMPORTANT**: Do **NOT** check the "Repair" checkbox
   * Repair would attempt to create S3 event notifications
   * This would conflict with your existing service (FSx, etc.)
4. Click **Start Re-Index**

### 🧪 Testing Your Setup

#### Verify Event Flow

Test that events are flowing correctly:

```bash
# Upload a test file
aws s3 cp test.txt s3://your-bucket-name/test.txt

# Check EventBridge metrics
aws events describe-rule --name quilt-s3-events-rule

# Check SNS topic metrics  
aws sns get-topic-attributes --topic-arn YOUR_SNS_TOPIC_ARN
```

#### Validate Quilt Integration

1. Upload a file to your S3 bucket
2. Wait 1-2 minutes for processing
3. Check Quilt catalog to see if the file appears
4. Search for the file in Quilt's search interface

## 🔧 Troubleshooting

### Common Issues and Solutions

#### Issue 1: Events Not Appearing in Quilt

**Symptoms:**

* Files uploaded to S3 don't appear in Quilt catalog
* Search doesn't find recently uploaded files

**Troubleshooting Steps:**

1. **Check EventBridge Rule Status**

```bash
aws events describe-rule --name quilt-s3-events-rule
```

* Ensure `State` is `ENABLED`

2. **Verify CloudTrail is Logging S3 Events**
   * Go to CloudTrail Console → Event history
   * Filter by Event source: `s3.amazonaws.com`
   * Confirm events are being logged
3. **Check SNS Topic Metrics**
   * Go to SNS Console → Your topic → Monitoring
   * Look for "Messages published" metrics
4. **Validate Input Transformer**
   * Test the EventBridge rule with a sample event
   * Check CloudWatch Logs for transformation errors

#### Issue 2: Permission Errors

**Symptoms:**

* EventBridge rule shows errors in CloudWatch
* SNS topic not receiving messages

**Solution:** Ensure EventBridge has permission to publish to SNS:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:region:account:quilt-eventbridge-notifications"
    }
  ]
}
```

#### Issue 3: Duplicate Events

**Symptoms:**

* Files appear multiple times in Quilt
* Excessive processing in Quilt logs

**Solution:**

* Check for multiple EventBridge rules targeting the same bucket
* Ensure you haven't enabled both S3 Event Notifications AND EventBridge

### Performance Considerations

#### Event Latency

* **EventBridge Latency**: \~1-5 seconds additional delay vs direct S3 events
* **CloudTrail Dependency**: Events only trigger after CloudTrail processes them
* **Batch Processing**: Consider batching for high-volume buckets

#### Cost Optimization

```bash
# Monitor EventBridge usage
aws events describe-rule --name quilt-s3-events-rule --query 'EventPattern'

# Check SNS costs
aws sns get-topic-attributes --topic-arn YOUR_TOPIC_ARN --attribute-names All
```

### Known Limitations

#### EventBridge-Specific Limitations

1. **Bulk Delete Operations**
   * The `delete-objects` API (used by AWS Console bulk delete) doesn't generate individual `delete-object` events
   * **Workaround**: Use individual delete operations or manual re-indexing
   * **Impact**: Bulk deletes may not be reflected in Quilt immediately
2. **Event Transformation Complexity**
   * EventBridge events have different structure than native S3 events
   * Input transformer may not capture all S3 event metadata
   * **Mitigation**: Test thoroughly with your specific use cases

#### General S3 Event Limitations

1. **Lifecycle Policy Deletions**

   * S3 lifecycle deletions are **not** captured by CloudTrail or S3 Events

   * **AWS Documentation**: [Supported Event Types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.title.html)

   > You do not receive event notifications from automatic deletes from lifecycle policies or from failed operations.
2. **CloudTrail Dependency**

   * EventBridge S3 events require CloudTrail data events

   * **AWS Documentation**: [Lifecycle and Logging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-and-other-bucket-config.html#lifecycle-general-considerations-logging)

   > Amazon S3 Lifecycle actions are not captured by AWS CloudTrail object level logging. CloudTrail captures API requests made to external Amazon S3 endpoints, whereas S3 Lifecycle actions are performed using internal Amazon S3 endpoints.

### Best Practices

#### Security

* ✅ Use least-privilege IAM policies
* ✅ Enable SNS topic encryption
* ✅ Monitor EventBridge rule metrics
* ✅ Set up CloudWatch alarms for failed events

#### Reliability

* ✅ Test event flow end-to-end before production
* ✅ Set up dead letter queues for failed events
* ✅ Monitor CloudWatch metrics for all components
* ✅ Have a rollback plan to direct S3 events if needed

#### Cost Management

* ✅ Monitor EventBridge and SNS costs
* ✅ Consider event filtering to reduce volume
* ✅ Use appropriate SNS delivery retry policies
* ✅ Clean up test resources after implementation

## 📚 Additional Resources

* [**AWS EventBridge Documentation**](https://docs.aws.amazon.com/eventbridge/)
* [**S3 Event Notifications**](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html)
* [**SNS Fanout Pattern**](https://aws.amazon.com/blogs/compute/fanout-s3-event-notifications-to-multiple-endpoints/)
* [**CloudTrail S3 Data Events**](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html)

***

**Need help?** Contact Quilt support or join our [Slack community](https://slack.quilt.bio) for assistance with EventBridge integration.


# SSO Permissions Mapping

> This feature requires Quilt Platform version 1.54.0 or higher

## Overview

This feature allows Quilt admin to configure what roles and admin flag value are assigned to the user who logs in via SSO based on the user's ID token claims.

The configuration can be set with `quilt3.admin.sso_config.set()` or with admin UI.

![admin UI for setting SSO permissions mapping](/files/55ak1nc94cTYSisKjO92)

> Note: Roles used by configuration can't be removed or renamed.

> Note: The user who sets the configuration will never have their admin flag revoked.

> Note: After configuration is set, any user who logs in via SSO can't be manually assigned roles or admin permissions.

## Configuration

The configuration file is to be written in YAML and is defined by [this JSON Schema](https://github.com/quiltdata/quilt/blob/master/shared/schemas/sso-config-1.0.json) which includes descriptions of all the fields.

> Warning: In schemas don't forget to add claims you want to check to `required`, because otherwise the schema will match any ID token even if these claims are missing.

### Which claims are matched

Each mapping's `schema` is validated against the **decoded ID token** only. Quilt does **not** read the access token, call the provider's userinfo endpoint, or resolve OIDC distributed/overage claims (the `_claim_sources` pointer Entra emits when a user belongs to more than \~200 groups) — if a value is not present directly in the ID token, a mapping cannot match on it.

Quilt also requests only the `openid` and `email` scopes at login, so group or role claims are never pulled in via a scope — they must be configured to be emitted into the ID token directly (e.g. via the provider's token/claim configuration), or the mapping will have nothing to match.

A mapping can match on **any** claim in the ID token, not just `email` or `groups`. For example, to map on an Entra **app-role** `roles` claim (often more reliable than `groups` for guest/cross-tenant users, since app roles are defined on the resource application itself):

```yaml
version: "1.0"
default_role: ReadQuiltBucket
mappings:
  - schema:
      type: object
      properties:
        roles:
          type: array
          contains:
            const: QuiltReadWrite
      required:
        - roles
    roles:
      - ReadWriteQuiltBucket
```

> Tip: To confirm exactly which claims arrive in the token, decode it at [jwt.ms](https://jwt.ms).

> Note: By default, mappings are evaluated in order and **only the first matching mapping is applied** — to assign multiple roles to a user this way, include all roles in the `roles` array of a single mapping. Alternatively, set `union_roles: true` at the top level of the config (Quilt Platform 1.69+) to grant the union of roles from **all** matching mappings; users can switch between the assigned roles via the role switcher, and any role no longer in the match set is revoked on next login.

> Note: Under `union_roles: true` (Quilt Platform 1.69+), the `admin` flag is **tri-state** and is **not** simply unioned the way `roles` are:
>
> * omitted (or `null`) — the mapping does not vote on admin,
> * `true` — the mapping grants admin,
> * `false` — the mapping **vetoes** admin.
>
> A user is made admin only if **at least one** matching mapping sets `admin: true` **and no** matching mapping sets `admin: false`. An explicit `admin: false` on any matching mapping therefore blocks admin even when another matching mapping sets `admin: true` (the user who sets the configuration is exempt — see the note above). The admin flag is **recomputed and written on every login** for any user who matches at least one mapping: if **every** matching mapping omits `admin`, the user receives no admin vote and is demoted, so omitting `admin` on a catch-all protects only users who *also* match an `admin: true` mapping. This matters for broad catch-all mappings (e.g. a domain-wide `pattern`) that a privileged user also matches: to keep such users admin, ensure they also match a mapping that sets `admin: true` rather than relying on omission alone, and reserve `admin: false` for when you intend to actively deny admin.
>
> With `union_roles: false`, only the first matching mapping applies and its `admin` value alone is used. The default is **no admin**: omitting `admin` (or setting it to `null`) on that mapping grants no admin permissions, and because the flag is written on every login, a matched user who would otherwise be admin has it removed. Use `admin: true` to grant admin in this mode.
>
> Platforms before 1.69 silently ignore `union_roles` (staying in first-match mode) and reject an explicit `admin: null` at upload.

### Example

```yaml
version: "1.0"
default_role: ReadQuiltBucket
union_roles: true
mappings:
  - schema:
      type: object
      properties:
        email:
          const: admin@example.com
      required:
        - email
    roles:
      - AdminTools
    admin: true
  - schema:
      type: object
      properties:
        groups:
          type: array
          contains:
            const: rw
      required:
        - groups
    roles:
      - ReadWriteQuiltBucket
```

By default (or with `union_roles: false`), only the first matching mapping applies — the `admin@example.com` user above would receive `AdminTools` only. With `union_roles: true`, that same user is granted both `AdminTools` and `ReadWriteQuiltBucket` **only if their token also carries group `rw`** (the second mapping requires it) — otherwise they match the first mapping alone and receive `AdminTools` only. When granted both, they can switch between the roles via the role switcher; a user with group `rw` only is granted `ReadWriteQuiltBucket` in either mode. The `admin@example.com` user remains admin because the first mapping sets `admin: true`, and — when they also match the second mapping — that mapping **omits** `admin` (a non-vote) rather than setting `admin: false`; had it set `admin: false`, admin would be vetoed (see the tri-state note above).

> Note: Users matching no mapping receive the `default_role` (`ReadQuiltBucket` in this example). Their admin flag is unchanged.

## Configuring your IdP to emit the required claims

Mappings only match against claims that actually appear in the user's ID token. If your mapping checks `groups` but the IdP doesn't include a `groups` claim, **no mapping will match** and users will silently fall through to the `default_role` — which (per the note above) leaves their admin flag unchanged from whatever it was before.

A common symptom is users receiving the `default_role` even though they belong to the groups referenced in the mappings. Verify by pasting their ID token into [jwt.io](https://jwt.io/) and confirming the expected claims are present — or, if the stack has `store_last_login_context: true` set in the SSO config, by reading the `idTokenPayload` field of `UserLastLoginContext` via the admin GraphQL API.

### Okta

Okta does **not** emit a `groups` claim by default. Which Okta screen you use to add it depends on **which authorization server your Quilt stack is configured against**. Check the `iss` claim in an existing ID token:

#### Custom authorization server (`iss` ends in `/oauth2/<id>`)

**This is the typical Quilt configuration.** The legacy app-level Group Claims filter does **not** apply here; you must add the claim on the authorization server itself:

1. **Security → API → Authorization Servers → `default` → Claims**.
2. **Add Claim**:
   * **Name:** `groups`
   * **Include in token type:** `ID Token`, `Always`
   * **Value type:** `Groups`
   * **Filter:** `Matches regex` · `.*` (or a narrower expression covering the groups your mappings reference — note that bare `*` is not a valid regex; use `.*`)
   * **Include in:** `Any scope`
3. Create.

#### Okta Org authorization server (`iss` has no `/oauth2/...` path)

Use the legacy app-level filter. Note that the claim name `groups` is reserved, so it cannot be added via **Token claims → Add expression** on the **Sign On** tab — that restriction is app-level and does not apply to the authorization server Claims tab above.

1. Open **Applications → \[Your App] → Sign On** tab.
2. Under **OpenID Connect ID Token**, click **Show legacy configuration**.
3. Set **Groups claim type** to `Filter`.
4. Set **Groups claim filter** to `groups` | **Matches regex** | `.*` (or a narrower expression that includes every group your mappings reference).
5. Save.

After saving, affected users must log out and back in for the new ID token to include the `groups` claim. Confirm with jwt.io that the token now contains something like `"groups": ["Everyone", "Employees", ...]`.


# Tabulator

> NOTE: This feature requires Quilt Platform version 1.55.0 or higher

Tabulator aggregates tabular data objects across multiple packages using AWS Athena. Admins define schemas and data sources for CSV, TSV, or Parquet files, enabling users to run SQL queries directly on the contents of Quilt packages. You can even use named capture groups to extract additional columns from the logical key and package name.

> **Faster, cheaper as of Quilt Platform version 1.70.** Tabulator now resolves the packages it needs from the per-bucket [Iceberg package index](/quilt-platform-catalog-user/advanced/iceberg-tables) instead of doing a full S3 scan through Glue/Athena SerDe tables on every call. Queries are cheaper and faster end-to-end. Permissions are unchanged: each caller queries under their own bucket-scoped credentials, and existing role and bucket permissions apply automatically.

The configuration is written in YAML and managed using the `quilt3.admin.tabulator` [APIs](https://docs.quilt.bio/quilt-platform-administrator/advanced/pages/0BNiVuuLjuVeFk5cONa7#quilt3.admin.tabulator) or via the Quilt Admin UI:

![Admin UI for setting Tabulator configuration](/files/9QXk7EDIzQycCKoPI5dP)

### Configuration

Each Tabulator configuration is written in YAML, following the structure outlined below.

#### Example

```yaml
schema:
  - name: name  # must match ^[a-z_][a-z0-9_]*$
    type: STRING  # usually BOOLEAN, INT, FLOAT, DOUBLE, STRING, DATE, TIMESTAMP
  - name: length
    type: INT
  - name: effective_length
    type: FLOAT
  - name: tpm
    type: FLOAT
  - name: num_reads
    type: FLOAT
source:
  type: quilt-packages  # currently the only supported type
  package_name: "^ccle/(?<date>[^_]+)_(?<study_id>[^_]+)_nfcore_rnaseq$"
  logical_key: "salmon/(?<sample_id>[^/]+)/quant*\\.genes\\.sf$"
parser:
  format: csv  # or `parquet`
  delimiter: "\t"
  header: true
continue_on_error: true
```

1. **Schema**: The schema defines the columns in the table. Each column must have a name and a type. The name must match the regular expression `^[a-z_][a-z0-9_]*$`. For CSV/TSVs, these names do not need to match the column names in the document. For Parquet, they must match except for case. However, if column names are present in a CSV/TSV, you must set `header` to `true` in the parser configuration.
2. **Types**: Must be uppercase and match the [Apache Arrow Data Types](https://docs.aws.amazon.com/athena/latest/ug/data-types.html) used by Amazon Athena. Valid types are BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, STRING, BINARY, DATE, TIMESTAMP.
3. **Source**: The source defines the packages and objects to query. The `type` must be `quilt-packages`. The `package_name` is a regular expression that matches the package names to include. The `logical_key` is a regular expression that matches the keys of the objects to include. The regular expression may include named capture groups that will be added as columns to the table.
4. **Parser**: The parser defines how to read the files. The `format` must be one of `csv` or `parquet`. The optional `delimiter` (defaults to ',') is the character used to separate fields in the CSV file. The optional `header` field (defaults to 'false') is a boolean that indicates whether the first row of the CSV file contains column names.

#### Added columns

In addition to the columns defined in the schema, Tabulator will add:

* any named capture groups from the `logical_key` or `package_name` regular expression
* `$pkg_name` for the package name
* `$logical_key` for the object as referenced by the package
* `$physical_key` for the underlying S3 URI
* `$top_hash` for the revision of the package containing the object (currently we query only the `latest` package revision)
* `$issue` for any problems encountered while parsing the file (new in Quilt Platform version 1.58)

#### Using Athena to Access Tabulator

The primary way of accessing Tabulator is using the Quilt stack to query those tables. This can be done by users via the per-bucket "Queries" tab in the Quilt Catalog, or programmatically via `quilt3`. See "Usage" below for more details.

As of Quilt Platform version 1.57, admins can enable [open query](#open-query) (below) to allow external users to access Tabulator tables directly from the AWS Console, Athena views, or JDBC connectors. This is especially useful for customers who want to access Tabulator from external services, such as Tableau and Spotfire.

#### Continuing After Errors

> Available since Quilt Platform version 1.58

If Tabulator encounters an error while processing a file, it normally stops and returns an error. As of Quilt Platform version 1.58:

* If a file is missing a `nullable` column (the default), Tabulator will automatically fill that column with null, record that fact in the `$issue` column and continue processing that and other files.
* If `continue_on_error` is set to true, Tabulator will also skip files with other errors (e.g. non-nullable columns, malformed CSVs with non-numeric strings in numeric columns). It will add a single line with non-nullable columns set to a type-appropriate default value (e.g., 0 or "") and record that fact in the `$issue` column.

![Tabulator Issue](/files/4FLGu0b1eUor2i9s6cl1)

#### Caveats

1. **Schema Consistency**: All files in the package that match the logical key must have the exact same schema as defined in the configuration (unless using Quilt Platform version 1.58 or higher, and using `continue_on_error: true`).
2. **Memory Usage**: Tabulator may fail on large files (> 10 GB), files with large rows (> 100 KB), and large numbers of files (> 10000). Additionally, Athena has a 16 MB limit per row.
3. **Cost Management**: Querying very large datasets can be expensive (approximately dollars per terabyte). Be sure to set up appropriate cost controls and monitoring.
4. **Concurrency**: Tabulator will attempt to process each file concurrently, but may be limited by the concurrency of Athena or the federation lambda in the [region](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/discover) where the query is running. If you are experiencing slow performance, it may be because the concurrency is too low. You can increase the concurrency in [that region](https://us-east-1.console.aws.amazon.com/servicequotas/home/services/lambda/quotas/L-B99A9384)'s AWS Service Quotas console.
5. **Athena VPC**: If you are using a VPC endpoint for Athena, you must ensure it is accessible from the Quilt stack and Tabulator lambda.

### Usage

Once the configuration is set, users can query the tables using the Athena tab from the Quilt Catalog. Note that because Tabulator runs with elevated permissions, it cannot be accessed from the AWS Console by default (unless [open query](#open-query) is enabled).

For example, to query the `ccle_tsv` table from the appropriate workgroup in the `quilt-tf-stable` stack, where the database (bucket name) is `udp-spec`:

```sql
SELECT * FROM "quilt-tf-stable-tabulator"."udp-spec"."ccle_tsv"
```

You can join this with the per-bucket [Iceberg package tables](/quilt-platform-catalog-user/advanced/iceberg-tables) that Quilt maintains automatically. (As of Quilt Platform version 1.70 these per-bucket Iceberg tables replace the previous global `*_packages-view` tables, which have been removed.) For example, `udp-spec_package_manifest` holds package-level metadata keyed by `top_hash`. These tables live in the Iceberg Glue database (the `IcebergDatabase` resource in your stack), which is a different database from the Tabulator one used above, so qualify them with that database name — e.g. `"<IcebergDatabase>"."udp-spec_package_manifest"` — when joining across the two.

Tabulator exposes the revision of each row's package as the `$top_hash` column, so you can join on it to add package-level `metadata` to the tabulated results:

```sql
SELECT
  "ccle_tsv".*,
  m.metadata
FROM "quilt-tf-stable-tabulator"."udp-spec"."ccle_tsv"
JOIN "<IcebergDatabase>"."udp-spec_package_manifest" m
  ON "ccle_tsv"."$top_hash" = m.top_hash
```

#### From Outside the Quilt Catalog

To call Tabulator from outside the Queries tab, you must use `quilt3` to authenticate against the stack using `config()` and `login()`, which opens a web page from which you must paste in the appropriate access token. Use `get_boto3_session()` to get a session with the same permissions as your Quilt Catalog user, then use the `boto3` Athena client to run queries.

> If [open query](#open-query) is enabled, you can use any AWS credentials providing access to Athena resources associated with Tabulator.

Here is a complete example:

```python
import quilt3
import time

DOMAIN = 'stable'
WORKGROUP = f'QuiltUserAthena-tf-{DOMAIN}-NonManagedRoleWorkgroup'
FULL_TABLE = f'"quilt-tf-{DOMAIN}-tabulator"."udp-spec"."ccle_tsv"'
QUERY = f'SELECT * FROM {FULL_TABLE} LIMIT 10'

quilt3.config(f'https://{DOMAIN}.quilttest.com/')
quilt3.login()
session = quilt3.get_boto3_session()
athena_client = session.client('athena')

response = athena_client.start_query_execution(
    QueryString=QUERY,
    WorkGroup=WORKGROUP
)
query_execution_id = response['QueryExecutionId']
print(f'Query execution ID: {query_execution_id}')

while True:
    execution_response = athena_client.get_query_execution(QueryExecutionId=query_execution_id)
    state = execution_response['QueryExecution']['Status']['State']
    if state in ('SUCCEEDED', 'FAILED', 'CANCELLED'):
        break
    print(f'\tQuery state: {state}')
    time.sleep(1)
print(f'Query finished with state: {state}')

if state == 'SUCCEEDED':
    results = athena_client.get_query_results(QueryExecutionId=query_execution_id)
    for row in results['ResultSet']['Rows']:
        print([field.get('VarCharValue') for field in row['Data']])
else:
    print(f'Query did not succeed. Final state: {state}')
```

### Open Query

> Available since Quilt Platform version 1.57

By default, Tabulator is only accessible via a session provided by the Quilt Catalog, and the access is scoped to the permissions of the Catalog user associated with that session. However, admins can choose to enable **open query** to Tabulator tables, deferring all access control to AWS, thus enabling access from external services. This allows querying Tabulator from the AWS Console, Athena views or JDBC connectors -- as long as the caller has been granted the necessary permissions to access Athena resources associated with Tabulator.

#### 1. Enable Open Query

An admin can enable open query via the `quilt3.admin.tabulator.set_open_query()` [API](/quilt-platform-administrator/admin-1#set_open_query) or Admin UI:

![Tabulator Settings](/files/aeqkWb4hGZS1HsXcFktb)

#### 2. Configure Permissions

In order to access Tabulator in open query mode, the caller must use a special workgroup, and have permissions to use that workgroup and access tabulator resources. For convenience, Quilt Stack provides a pre-configured workgroup and policy for open query -- they can be found in the stack outputs:

1. `TabulatorOpenQueryPolicyArn`: attach this managed policy to a relevant IAM role (or copy the statements directly to your own role/policy).
2. `TabulatorOpenQueryWorkGroup`: configure your Athena client or connector to use this workgroup (or create your own with the same results output configuration).

![Tabulator Resources](/files/DR8XlwEO4aqvRdEQEZrx)


# Troubleshooting

## Overview stats (objects, packages) seem incorrect or aren't updating

## Packages tab doesn't work

## Packages or stats are missing or are not updating

These are all symptoms of the same underlying issue: the Elasticsearch index is out of sync. If any of the following are true, please wait a few minutes and try again:

* you recently added the bucket or upgraded the stack
* search volume is high, or
* read/write volume is high

If that doesn't work, try the following steps:

### Re-index the bucket

If you have less than 1 million objects in the bucket, you should re-index the bucket:

1. Open the bucket overview in the Quilt catalog and click the gear icon (upper right), or navigate to Admin settings > Buckets and inspect the settings of the bucket in question.
2. Under "Indexing and notifications", click "Re-index and Repair".

> Optionally: **if and only if** bucket notifications are not working and you are certain that there are no other subscribers to the S3 Events of the bucket in question, check "Repair S3 notifications".

Bucket packages, stats, and the search index will repopulate in the next few minutes.

However, if you have more than 1 million objects in the bucket, re-indexing will take much longer and potentially become expensive. In that case, please try the below steps. If those do not work, please contact [Quilt support](mailto:support@quilt.bio).

### Inspect the Elasticsearch domain

1. Determine your Quilt instance's ElasticSearch domain from Amazon Console > OpenSearch or `aws opensearch list-domain-names`. Note the domain name (hereafter `QUILT_DOMAIN`).
2. Run the following command and save the output file:

   ```sh
   aws es describe-elasticsearch-domain --domain-name "$QUILT_DOMAIN"\
     > quilt-es-domain.json
   ```
3. Visit Amazon Console > OpenSearch > `QUILT_DOMAIN` > Cluster health.
4. Set the time range as long as possible to fully overlap with your observed issues.
5. Screenshot the Summary, Overall Health, and Key Performance Indicator sections
6. Send the JSON output file and screenshots to [Quilt support](mailto:support@quilt.bio).

> As a rule you should **not** reconfigure your Elasticsearch domain directly as this will result in stack drift that will be lost the next time you update your Quilt instance.

## "Session expired" notice

There are two reasons for encountering the "Session expired" notice after clicking the `RELOAD` button in the Quilt Catalog.

1. Your browser cache is out of date, in which case you need to:
   1. Delete session storage
   2. Delete local storage
   3. Delete cookies
2. Your Quilt user Role has been corrupted. You will need a Quilt Admin user to reset your Quilt user Role to a default (**and valid**) Role.

## Browser Network and Console Logs

To help diagnose Catalog issues:

1. Go to the affected page in your Quilt Catalog.
2. Open the browser Developer tools:
   * Google Chrome: Press **F12**, **Ctrl+Shift+I** or from the Chrome menu select **More tools > Developer tools**.
3. Select the **Network** tab.
   1. Ensure the session is recorded:
      * Google Chrome: Check the red button in the upper left corner is set to **Record**.
   2. Ensure **Preserve Log** is enabled.
   3. Perform the action that triggers the error (e.g. clicking the `Download package` button).
   4. Export the logs as HAR format.
      * Google Chrome: **Ctrl + Click** anywhere on the grid of network requests and select **Save all as HAR with content**.
   5. Save the HAR-formatted file to your localhost.

      ![Save browser Network error logs as HAR
      content](/files/FtkATJQoDQrcotnDIsHS)
4. Select the **Console** tab.
   1. Perform the action that triggers the error (e.g. clicking the `Download package` button).
   2. Export the logs.
      * Google Chrome: **Ctrl + Click** anywhere on the grid of network requests and select **Save as...**.
   3. Save the log file to your localhost.


# SSO Redirect Loop

## Overview

This guide helps customers diagnose and resolve **SSO login redirect loop** issues when using **Microsoft Azure, Google Workspace, or Okta** for authentication.

## Step 1: Verify Redirect URIs

Incorrect redirect URIs are a common cause of SSO loops.

* **Azure**: Go to **Azure AD > Enterprise Applications > \[Your Application] > Authentication**.
* **Google**: Go to **Google Admin Console > Security > Set up single sign-on (SSO)**.
* **Okta**: Go to **Okta Admin > Applications > \[Your App] > Sign On**.

Ensure the **redirect URI** matches exactly what is expected, as documented in the Quilt [technical reference](https://docs.quilt.bio/quilt-platform-administrator/technical-reference#single-sign-on-sso). This includes:

* Case sensitivity (Azure and Okta are case-sensitive!)
* Proper `https://` scheme
* Trailing slashes (if required)

Double-check the **Logout URL**, as incorrect values can cause infinite redirects.

## Step 2: Review SSO Provider Sign-In Logs

SSO provider logs can help pinpoint misconfigurations.

1. **Azure**: Navigate to **Azure AD > Enterprise Applications > \[Your Application] > Sign-In Logs**.
2. **Google**: Go to **Google Admin Console > Reports > Audit > SAML**.
3. **Okta**: Go to **Okta Admin > System Log**.

Look for failed sign-ins and error codes:

* **AADSTS50011 (Azure)**: Redirect URI mismatch.
* **AADSTS50008 (Azure)**: Invalid token signature.
* **AADSTS50105 (Azure)**: User is not assigned to the app.
* **`access_denied — Identity Provider: Unknown` (Okta)**: Federation Broker Mode ("Enable immediate access") is enabled on the app. Disable it in the app settings and assign users directly.
* **`redirect_uri` mismatch (Okta)**: The redirect URI in the app doesn't match. Note that wildcards only match one subdomain level (`*.example.com` does not match `app.dev.example.com`).
* **"You are not allowed to access this app" (Okta)**: The user is not assigned to the app, or the authorization server's access policy has no rules. Check both the app Assignments tab and Security > API > Authorization Servers > default > Access Policies.
* **"Unable to sign in with Okta" in the Quilt catalog**: The Okta login succeeded but the registry's token exchange failed. Check that `Refresh Token` is enabled under Grant type in the app's General settings, and that the authorization server's access policy has a rule granting tokens.
* **SSO permissions mapping assigns everyone the `default_role`**: The ID token is likely missing the `groups` claim. See [SSO permissions mapping → Configuring your IdP to emit the required claims](/quilt-platform-administrator/advanced/sso-permissions#configuring-your-idp-to-emit-the-required-claims).
* **403 or 400 errors (Google/Okta)**: Often indicate incorrect redirect URIs or token issues.

## Step 3: Inspect Browser Network Requests

Use browser DevTools (F12) to examine the authentication flow:

* Open the **Network tab** before attempting to log in.
* Filter requests by `sso`, `redirect`, or `login`.
* Look for repeated requests to the same URL, indicating a loop.
* Click on the **Request/Response Headers** to check error messages.

## Step 4: Verify Token Claims (SAML or OIDC)

Incorrect claims or missing attributes can cause authentication failures.

* If using **SAML**, use **SAML-tracer (Firefox)** or **Fiddler** to inspect assertions.
* If using **OIDC**, paste the ID token into [jwt.io](https://jwt.io/) or [jwt.ms](https://jwt.ms/) to check claims.
* Ensure the **issuer (iss)** and **audience (aud)** claims match what the application expects.

## Step 5: Collect CloudWatch Logs for ECS

Follow the usual [troubleshooting steps](https://docs.quilt.bio/quilt-python-sdk/more/troubleshooting#elastic-container-service-ecs) to collect the ECS logs for the registry service:

* Look for authentication-related errors or unexpected redirects.
* Verify that the expected redirect URIs are being returned.

## Common Fixes

| Issue                         | Possible Fix                                             |
| ----------------------------- | -------------------------------------------------------- |
| Redirect loop after login     | Verify redirect URIs and logout URL in Azure/Google/Okta |
| Authentication fails silently | Check SSO provider logs for error codes                  |
| Incorrect token claims        | Ensure the ID token contains the expected claims         |
| User not authorized           | Assign the user to the SSO Enterprise App                |
| Invalid signature error       | Confirm the correct token signing algorithm is used      |

## Next Steps

If the issue persists after these checks, provide the following information to your support team:

1. SSO Provider Sign-In Logs with error codes.
2. Browser Network logs (HAR file) showing redirects.
3. SAML assertion (if applicable) or decoded OIDC token.
4. ECS logs from the registry.

Following these steps should help you diagnose and resolve most SSO redirect loop issues efficiently across Azure, Google, and Okta.


# Best Practices


# GxP for Security & Compliance

## Overview

GxP environments for labs, manufacturing, documentation, and clinical practice require what we call a "data chain of custody" that enables Quilt users to understand where data came from, who produced it, when it was produced, and why it should be trusted.

The GxP module for Quilt includes the following key features:

1. Strong cryptographic checksums for data at the object level and at the collection (or package) level
2. Integrated IQ (installation qualification) and OQ (operational qualification) testing to help you automate tedious manual qualification cycles into tests run by machines and reported to you.

## Status monitoring

### Architectural overview

The Quilt status monitoring system consists of four parts:

1. *Canaries:* A collection of end-to-end quality tests for key operational functionality provided by Quilt. They are run on a schedule in your AWS infrastructure.
2. *Status reports:* Simple HTML files generated on a daily schedule by an AWS Lambda function (`status_reports`) and stored in a dedicated Amazon S3 bucket. The HTML files contain the most recent canary run results (Operational Qualification) and Quilt instance Cloud Formation parameters and outputs (Infrastructure Qualification).
   1. Operational Qualification report: all canary test results and status (historical and current)
   2. Infrastructure Qualification report: all CloudFormation parameters and status
3. *Administrative User Interface:* An HTML page in the Quilt catalog that displays the current operational status of a Quilt instance and provides access to current and historical status reports.
4. *Amazon SNS topic:* Users can subscribe to the topic and receive canary error notifications (available as `CanaryNotificationsTopic` stack output).

![Example status report](/files/iohUhLTooDDs6PvWcFoo)

### Setting up status monitoring for a Quilt instance

In order to enable status monitoring for a Quilt instance, the following conditions must be met:

* The Quilt instance CloudFormation template must have canaries enabled
* The correct template parameters must be provided during the Quilt instance CloudFormation deployment

The required Quilt and AWS resources are provisioned automatically by the Quilt deployment.

#### List of canaries

There are currently four end-to-end quality tests:

* `BucketAccessControl`: Test that Users can only access specifically allowed Amazon S3 buckets
* `Immutable URIs`: Test to resolve immutable Quilt URIs
* `PackagePushUi`: Test package `push` functionality via Quilt catalog package creation dialog
* `Search`: Search S3 objects and Quilt packages in the Quilt catalog

#### Quilt instance CloudFormation deployment parameters

When deploying a Quilt instance using a CloudFormation template with canaries enabled, you will have to provide the following parameters:

* `CanaryNotificationsEmail` (if enabled): A valid email address to send canary failure and error notifications

### Accessing current operational status and reports archive

Go to your Quilt web-based catalog administration panel, "Status" tab (under the /admin/status URL). There you can see the stack's current operational status (as well as the chart with historic data for the last month) and the reports table (which can be sorted / filtered) where you can preview or download stored reports (HTML files).

![Status overview](/files/d55q6JhNT8LJ4Vu6r3rV)

### Getting notifications about canary failures

For convenience, the following canary-related events are forwarded to an SNS topic (available as `CanaryNotificationsTopic` in the Quilt instance CloudFormation deployment Outputs tab):

1. A canary entering error state
2. A canary run failure

Quilt administrators can subscribe to this SNS topic to receive these event notifications and process them appropriately (e.g. notifying first responders or sending to incident management systems).

If enabled (see set-up instructions above), these events will be also sent as emails to the configured address (`CanaryNotificationsEmail` CloudFormation template parameter).

## Audit Trail

Audit trails enable you to track which users had which permissions at which time. While data access to S3 is logged via CloudTrail, certain "admin plane" events such as the following are logged to a managed S3 bucket and exposed via an Athena table:

* A user logs into the Catalog
* A user role is changed
* A user is added, deleted, or inactivated in the Catalog

### Audit events

Audit events are stored as JSON records in JSONL files in a managed audit trail S3 bucket.

#### Event structure

* `eventVersion: str`, (required, since 1.0)

  Version of the event record.
* `eventTime: datetime` (required, since 1.0)

  When the action was **completed** (in UTC).
* `eventID: str` (required, since 1.0)

  A unique ID (UUID) of the event.
* `eventSource: "QuiltServer" | "QuiltScript"` (required, since 1.0)

  The service or part of the system that this event originates from.
* `eventType: "QuiltApiCall" | "QuiltScriptInvocation"` (required, since 1.0)

  The type of event that generated the event record.
* `eventName: str` (required, since 1.0)

  The name of action performed in the form `${namespace}.${operationName}`, e.g. `Users.Create`. See [Event Taxonomy](#event-taxonomy) for details.
* `userAgent: str` (optional, since 1.0)

  The agent through which the request was made, such as a web browser, a Quilt Stack or Quilt Python Client.
* `sourceIPAddress: str` (optional, since 1.0)

  The IP address that the request was made from.
* `userIdentity: UserIdentity` (required, since 1.0)

  Information about the identity that performed the action. Refer to [UserIdentity](#user-identity) section below for details.
* `requestID: str` (optional, since 1.0)

  Request ID based on [AWS ALB request tracing capabilities](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-request-tracing.html).
* `requestParameters: object` (required, since 1.0)

  The parameters, if any, that were sent with the request. These parameters are documented under [Event Taxonomy](#event-taxonomy) section.
* `responseElements: any` (optional, since 1.0)

  The response data, if any.
* `errorCode: str` (optional, since 1.0)

  Only present if an error occurred while trying to perform an action, `null` when the action succeeds.
* `errorMessage: str` (optional, since 1.0)

  Description of the error.
* `additionalEventData: object` (optional, since 1.0)

  Additional data about the event that was not part of the request or response.

#### Event source and type

When a Quilt Server API is called (GraphQL query or HTTP endpoint), `eventSource` is set to `QuiltServer` and `eventType` -- to `QuiltApiCall`.

When an admin script is invoked (this usually happens on stack bring-up / upgrade), `eventSource` is set to `QuiltScript` and `eventType` -- to `QuiltScriptInvocation`.

#### Event schema versioning

Event records are versioned using SemVer-like versioning scheme (`MAJOR.MINOR`).

`MAJOR` version is incremented on backwards-incompatible changes to the schema, e.g. removing a JSON field that already exists, or changing how the contents of a field are represented (for example, a date format).

`MINOR` version is incremented on backwards-compatible changes, such as adding new fields to the event structure.

#### GraphQL requests

For GraphQL requests, query (mutation) parameters are recorded as `requestParameters`, and response data (respecting the selection set) are recorded as `responseElements`. Errors are inferred and recorded to `errorCode` / `errorMessage`.

#### User Identity

All `UserIdentity` variants are stored as JSON objects with a required `type` field. Other fields vary based on the type of the recorded user identity.

**`QuiltUser`**

Represents an authenticated Quilt User.

**Attributes**:

* `type: "QuiltUser"`
* `id: str`
* `userName: str`
* `email: str`
* `isAdmin: bool`
* `isActive: bool`
* `isSsoOnly: bool`
* `isService: bool`
* `lastLogin: datetime`
* `dateJoined: datetime`
* `roleId: str` (optional) - An ID of the associated Quilt Role.
* `sessionContext: SessionContext` (optional, since Platform version 1.67) - Authentication context for the session.

**`SessionContext`**

Contains information about how the user authenticated.

**Attributes**:

* `auth: AuthContext` - Authentication method details.

**`AuthContext`**

Discriminated union with `type` field:

* **`InteractiveAuth`**: User authenticated via SSO/OAuth.
  * `type: "interactive"`
* **`APIKeyAuth`**: User authenticated via API key.
  * `type: "api_key"`
  * `keyId: str` - The API key ID.
  * `keyName: str` - User-provided name of the key.
  * `keyFingerprint: str` - Display fingerprint (e.g., `qk_abc...xyz`).

**`Unidentified`**

Represents a user we were unable to identify.

In `QuiltApiCall` actions, this means an anonymous (unauthenticated) user, e.g. we can't identify a user trying to sign-in with a non-registered username.

In `QuiltScript` actions, this means the execution environment doesn't have AWS credentials available -- this shouldn't happen in our production installations.

**Attributes**:

* `type: "Unidentified"`

**`IAMUser`**

Represents an AWS/IAM user (e.g. when ECS invokes an admin script). Records the data returned by [`sts:GetCallerIdentity`](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html).

**Attributes**:

* `type: "IAMUser"`
* `account: str` - The AWS account ID number of the account that owns or contains the calling entity.
* `id: str` - The unique identifier of the calling entity.
* `arn: str` - The AWS ARN associated with the calling entity.

#### Event Taxonomy

See the GraphQL schema for GraphQL type reference.

**`Auth` namespace**

All the authentication-related operations.

**`Auth.RefreshToken`**

Authentication token refreshed.

* `request_parameters`
  * `refresh_token: "***"`
* `responseElements`
  * `access_token: "***"`
  * `refresh_token: "***"`
  * `expires_at: datetime`
* `additionalEventData`
  * `method: "code" | "refresh"`

**`Auth.Login`**

User logged in.

* `requestParameters`
  * `provider: str` (when `method` is `oauth`)
  * `code: "***"` (when `method` is `oauth`)
  * `username: str` (when `method` is `password`)
  * `password: "***"` (when `method` is `password`)
* `additionalEventData`
  * `method: "oauth" | "password"`
  * `account_id: str` (when `method` is `oauth`)
  * `email: str` (when `method` is `oauth`)
  * `user_created: true` (when `method` is `oauth` and new user was created)
  * `account_linked: true` (when `method` is `oauth` and existing user was linked with a new OAuth identity)
* `responseElements`
  * `access_token: "***"`
  * `refresh_token: "***"`
  * `exp: datetime`

**`Auth.ServiceLogin`**

A service user (Canary) logged in.

* `requestParameters`
  * `provider: str`
  * `token: "***"`
* `additionalEventData`
  * `account_id: str`
* `responseElements`
  * `access_token: "***"`
  * `refresh_token: "***"`
  * `exp: datetime`

**`Auth.Activate`**

A user was activated.

**`Auth.PasswordResetRequest`**

Password reset requested.

* `requestParameters`
  * `email: str`

**`Auth.PasswordChange`**

User password changed.

* `requestParameters`
  * `password: "***"`
  * `link: "***"`

**`Auth.Register`**

User signed up.

* `requestParameters`
  * `username: str`
  * `email: str`
  * `password: "***"`
* `additionalEventData`
  * `default_role_id: str`

**`Auth.Logout`**

User signed out.

**`Auth.IssueCode`**

OAuth code issued.

* `responseElements`
  * `code`
    * `user_id: str`
    * `code: str`
    * `expires: datetime`
    * `sso_provider: str`
    * `sso_access_token: "***"`
    * `sso_refresh_token: "***"`
    * `sso_expires: datetime`

**`Auth.GetAWSCredentials`**

AWS credentials issued for a Quilt user.

* `responseElements`
  * `AccessKeyId: str`
  * `SecretKey: "***"`
  * `SessionToken: "***"`

**`Auth.APIKey`**

API key authentication attempt (logged on failure only).

* `requestParameters`
  * `fingerprint: str` - Truncated key fingerprint (e.g., `qk_abc...xyz`), never the full secret
* `errorCode`
  * `INVALID` - Key not found or invalid format
  * `EXPIRED` - Key has expired
  * `USER_INACTIVE` - Key owner is inactive

**`APIKeys` namespace**

API key management operations. Available to all authenticated users for their own keys.

**`APIKeys.Create` (GraphQL: `Mutation.apiKeyCreate`)**

API key created.

* `requestParameters`
  * `input`
    * `name: str`
    * `expiresInDays: int`
* `responseElements`
  * `apiKey`
    * `id: str`
    * `name: str`
    * `fingerprint: str`
    * `status: str`
    * `createdAt: datetime`
    * `expiresAt: datetime`
  * `secret: "***"` - Always redacted in audit logs

**`APIKeys.Revoke` (GraphQL: `Mutation.apiKeyRevoke`)**

API key revoked (deleted).

* `requestParameters`
  * `id: str` (when revoking by ID)
  * `secret: "***"` (when revoking by secret, always redacted)

**`APIKeys.Admin.Revoke` (GraphQL: `Mutation.admin.apiKeys.revoke`)**

Admin revoked another user's API key.

* `requestParameters`
  * `id: str`

**`Users` namespace**

User management operations. Only accessible by the admin users.

**`Users.List`**

List users. Response data is not logged.

**`Users.Create`**

User created.

* `requestParameters`
  * `username: str`
  * `email: str`

**`Users.Disable`**

User disabled.

* `requestParameters`
  * `username: str`

**`Users.Enable`**

User enabled.

* `requestParameters`
  * `username: str`

**`Users.EditEmail`**

User's email changed.

* `requestParameters`
  * `username: str`
  * `email: str`

**`Users.GrantAdmin`**

User is granted with admin rights.

* `requestParameters`
  * `username: str`

**`Users.RevokeAdmin`**

User is revoked admin rights.

* `requestParameters`
  * `username: str`

**`Users.Delete`**

User deleted.

* `requestParameters`
  * `username: str`

**`Users.ResetPassword`**

User's password reset.

* `requestParameters`
  * `username: str`

**`Users.SetRole`**

User's role updated.

* `requestParameters`
  * `username: str`
  * `role: str` - Role name

**`Buckets` namespace**

Bucket management operations. Only accessible by the admin users.

**`Buckets.Add` (GraphQL: `Mutation.bucketAdd`)**

Bucket added to the stack.

**`Buckets.Update` (GraphQL: `Mutation.bucketUpdate`)**

Bucket settings updated.

**`Buckets.Remove` (GraphQL: `Mutation.bucketRemove`)**

Bucket removed from the stack.

**`Policies` namespace**

Quilt Policy management operations. Only accessible by the admin users.

**`Policies.Create` (GraphQL: `Mutation.policyCreateManaged` / `Mutation.policyCreateUnmanaged`)**

Quilt Policy created.

`requestParameters.managed` is set accordingly.

**`Policies.Update` (GraphQL: `Mutation.policyUpdateManaged` / `Mutation.policyUpdateUnmanaged`)**

Quilt Policy updated.

`requestParameters.managed` is set accordingly.

**`Policies.Delete` (GraphQL: `Mutation.policyDelete`)**

Quilt Policy deleted.

**`Roles` namespace**

Quilt Role management operations. Only accessible by the admin users.

**`Roles.Create` (GraphQL: `Mutation.roleCreateManaged` / `Mutation.roleCreateUnmanaged`)**

Quilt Role created.

`requestParameters.managed` is set accordingly.

**`Roles.Update` (GraphQL: `Mutation.roleUpdateManaged` / `Mutation.roleUpdateUnmanaged`)**

Quilt Role updated.

`requestParameters.managed` is set accordingly.

**`Roles.Delete` (GraphQL: `Mutation.roleDelete`)**

Quilt Role deleted.

**`Roles.SetDefault` (GraphQL: `Mutation.roleSetDefault`)**

Quilt Role set as default.

**`Scripts` namespace**

Admin scripts. Usually invoked by CloudFormation on stack bring-up / upgrade.

All `Scripts.*` events contain the following data:

* `additionalRequestData`
  * `script_name: str` - Script filename
  * `script_args: str[]` - List of arguments
  * `script_command: str` - The whole unparsed command string
* `userIdentity: IAMUser` - AWS identity of the user executing the script
* `requestParameters: object` - Parsed named script arguments

**`Scripts.CreateAdmin`**

Create an admin user account. Succeeds once on stack bring-up, fails on subsequent stack upgrades.

* `requestParameters`
  * `env: bool` - Pass account info in environment variables
  * `role_name: str` (optional) - Name for Quilt T4 Role
  * `email: str`
  * `password: "***"` (optional)
* `additionalEventData`
  * `role_id: str` (optional) - ID of the Quilt Role, if found by the name `role_name` (when provided).

**`Scripts.CreateRole`**

Add a named role to the stack, or set the ARN of a role if a role with the specified name already exists.

* `requestParameters`
  * `name: str`
  * `arn: str` (optional) - ARN for AWS Role that is associated with this Quilt Role
  * `default: bool` - Set this role as default if default role is not already set

**`Scripts.UpdateBucketResources`**

Update all bucket-related resources: policies, indexes, SNS, cloudtrail, ...

* `additionalEventData`
  * `subscriptions_created: list` - A mapping (list of tuples) of bucket names to ARNs of associated subscriptions created by the script
  * `subscription_errors: list` - A mapping (list of tuples) of bucket names to error messages received while trying to create associated subscriptions

**`Scripts.SetupCanaries`**

Create a canary user (`_canary <canary@quiltdata.io>`) and set up resources required for running continuous integration testing (OQ monitoring).

* `requestParameters`
  * `bucket_allowed: str`
  * `bucket_restricted: str`

#### Example event records

Admin user creation (failed because the user was created earlier):

```json
{
  "eventVersion": "1.0",
  "eventTime": "2023-06-08T13:55:27Z",
  "eventID": "3d59b43a-3c31-48b1-bd5c-a039af87fdc5",
  "eventSource": "QuiltScript",
  "type": "QuiltScriptInvocation",
  "eventName": "Scripts.CreateAdmin",
  "userAgent": "quilt-stack quilt-registry (Linux 5.10.179-166.674.amzn2.x86_64) CPython/3.8.16",
  "sourceIPAddress": null,
  "userIdentity": {
    "type": "IAMUser",
    "id": "*REDACTED*",
    "account": "*REDACTED*",
    "arn": "arn:aws:sts::*REDACTED*:assumed-role/*REDACTED*"
  },
  "requestParameters": {
    "env": true,
    "role_name": "ReadWriteQuiltBucket",
    "email": "example@quiltdata.io",
    "password": null
  },
  "responseElements": null,
  "errorCode": "Conflict",
  "errorMessage": "Email already taken.",
  "additionalEventData": {
    "script_name": "./scripts/create_admin.py",
    "script_args": ["-e", "-r", "ReadWriteQuiltBucket"],
    "script_command": "./scripts/create_admin.py -e -r ReadWriteQuiltBucket",
    "role_id": "*REDACTED*"
  }
}
```

A user with admin rights authenticated using password:

```json
{
  "eventVersion": "1.0",
  "eventTime": "2023-06-08T13:59:36Z",
  "eventID": "92e66ee3-42fd-4142-990d-9d54059c583b",
  "eventSource": "QuiltServer",
  "type": "QuiltApiCall",
  "eventName": "Auth.Login",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0",
  "sourceIPAddress": "*REDACTED*",
  "userIdentity": {
    "type": "QuiltUser",
    "id": "*REDACTED*",
    "userName": "example_user",
    "email": "example_user@quiltdata.io",
    "isAdmin": true,
    "lastLogin": "2023-03-23T12:37:53Z",
    "isActive": true,
    "isSsoOnly": false,
    "isService": false,
    "dateJoined": "2020-11-16T03:32:01Z",
    "roleId": "*REDACTED*"
  },
  "requestParameters": { "username": "example_user", "password": "***" },
  "responseElements": {
    "access_token": "***",
    "refresh_token": "***",
    "exp": "2023-09-06T13:59:36Z"
  },
  "errorCode": null,
  "errorMessage": null,
  "additionalEventData": { "method": "password" }
}
```

A service user (Canary) authenticated:

```json
{
  "eventVersion": "1.0",
  "eventTime": "2023-06-12T00:37:21Z",
  "eventID": "10b0901a-74cd-4e62-8954-33f4fba6e8ad",
  "eventSource": "QuiltServer",
  "type": "QuiltApiCall",
  "eventName": "Auth.ServiceLogin",
  "userAgent": "",
  "sourceIPAddress": "*REDACTED*",
  "userIdentity": {
    "type": "QuiltUser",
    "id": "*REDACTED*",
    "userName": "_canary",
    "email": "canary@quiltdata.io",
    "isAdmin": false,
    "lastLogin": "2022-11-24T13:55:31Z",
    "isActive": true,
    "isSsoOnly": false,
    "isService": true,
    "dateJoined": "2022-11-24T13:55:31Z",
    "roleId": "*REDACTED*"
  },
  "requestParameters": { "provider": "quilt-service-auth", "token": "***" },
  "responseElements": {
    "access_token": "***",
    "refresh_token": "***",
    "exp": "2023-09-10T00:37:21Z"
  },
  "errorCode": null,
  "errorMessage": null,
  "additionalEventData": { "account_id": "_canary" }
}
```

### Querying With Athena

Audit events can be queried with [AWS Athena](https://aws.amazon.com/athena/). Quilt Stack provisions the following resources:

* **Audit Trail Database**: a Glue database named `audittraildatabase-${random_string}`, exposed as `AuditTrailDatabase` stack resource.
* **Audit Trail Table**: a Glue table named `audit_trail` in **Audit Trail Database**.
* **Audit Workgroup**: an Athena workgroup named `${AWS::StackName}-audit`, exposed as `AuditTrailWorkgroup` stack resource.
* **Audit Trail Bucket**: an S3 bucket storing all the audit trail data and Athena query results for **Audit Workgroup**.
* **Audit Trail Query Policy**: a managed IAM policy allowing to query audit trail data via Athena, exposed as `AuditTrailAthenaQueryPolicy` stack resource.

In order to query audit trail data via AWS Athena Console, you should:

1. Ensure you have adequate permissions to query the data (e.g. use the Quilt-managed **Audit Trail Query Policy**).
2. Select `AwsDataCatalog` from the "Data source" dropdown.
3. Select **Audit Trail Database** from the "Database" dropdown.
4. Select **Audit Workgroup** from the "Workgroup" dropdown.

All the events are available in the `audit_trail` table, which has the following fields (schema version 1.0):

* `eventversion: string`
* `eventtime: timestamp`
* `eventid: string`
* `eventsource: string`
* `eventtype: string`
* `eventname: string`
* `useragent: string`
* `sourceipaddress: string`
* `useridentity: string`
* `requestparameters: string`
* `responseelements: string`
* `errorcode: string`
* `errormessage: string`
* `additionaleventdata: string`
* `requestid: string`
* `date: string` (partition)

The data is partitioned by `date`, which has `YYYY/mm/dd` format. [Partitioning](https://docs.aws.amazon.com/athena/latest/ug/ctas-partitioning-and-bucketing.html#ctas-partitioning-and-bucketing-what-is-partitioning) helps to make the queries faster and cheaper by reducing the amount of data scanned, so it's advised to always filter by the partition key when possible.

All the JSON objects from an event record are exposed to Athena as strings, so you can leverage athena JSON querying capabilities.

#### Example queries

When did a user with the specified email last log in?

```sql
SELECT
eventtime,
useragent,
sourceipaddress,
useridentity,
requestparameters,
responseelements,
additionaleventdata
FROM audit_trail
WHERE eventname = 'Auth.Login'
AND errorcode IS NULL
AND json_extract_scalar(useridentity, '$.type') = 'QuiltUser'
AND json_extract_scalar(useridentity, '$.email') = 'example@quiltdata.io'
ORDER BY eventtime DESC
LIMIT 1;
```

Example query result:

| `eventtime`               | `useragent`                                                              | `sourceipaddress` | `useridentity`                                                                                                                                                                                                                                               | `requestparameters`                       | `responseelements`                                                          | `additionaleventdata`   |
| ------------------------- | ------------------------------------------------------------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | --------------------------------------------------------------------------- | ----------------------- |
| `2023-06-08 13:59:36.000` | `Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0` | *REDACTED*        | `{"isssoonly":false,"isadmin":true,"lastlogin":"2023-03-23T12:37:53Z","roleid":"*REDACTED*","isactive":true,"isservice":false,"id":"*REDACTED*","type":"QuiltUser","datejoined":"2020-11-16T03:32:01Z","email":"example@quiltdata.io","username":"example"}` | `{"password":"***","username":"example"}` | `{"access_token":"***","refresh_token":"***","exp":"2023-09-06T13:59:36Z"}` | `{"method":"password"}` |

What are all the actions performed by a user with the given email today and which IP did they come from?

```sql
SELECT
eventtime,
eventname,
useragent,
sourceipaddress,
requestparameters,
responseelements,
additionaleventdata,
errorcode
FROM audit_trail
WHERE date = date_format(current_date, '%Y/%m/%d')
AND json_extract_scalar(useridentity, '$.type') = 'QuiltUser'
AND json_extract_scalar(useridentity, '$.email') = 'example@quiltdata.io'
ORDER BY eventtime
```

Example query result:

| `eventtime`               | `eventname`              | `useragent`                                                                            | `sourceipaddress` | `requestparameters`                                                                       | `responseelements`                                                              | `additionaleventdata`           | `errorcode` |
| ------------------------- | ------------------------ | -------------------------------------------------------------------------------------- | ----------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------- | ----------- |
| `2023-06-14 11:33:00.000` | `Auth.Login`             | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0` | *REDACTED*        | `{"password":"***","username":"nl0"}`                                                     | `{"access_token":"***","refresh_token":"***","exp":"2023-09-12T11:33:00Z"}`     | `{"method":"password"}`         |             |
| `2023-06-14 11:33:02.000` | `Auth.GetAWSCredentials` | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0` | *REDACTED*        | `{}`                                                                                      | `{"secretkey":"***","sessiontoken":"***","accesskeyid":"ASIA2LR7MUT63B7CE5WE"}` | `{}`                            |             |
| `2023-06-14 11:33:07.000` | `Users.List`             | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0` | *REDACTED*        | `{}`                                                                                      | `{"results":[*REDACTED*]}`                                                      | `{}`                            |             |
| `2023-06-14 11:33:16.000` | `Users.Disable`          | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0` | *REDACTED*        | `{"username":"*REDACTED*"}`                                                               |                                                                                 | `{}`                            |             |
| `2023-06-14 11:33:42.000` | `Users.GrantAdmin`       | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0` | *REDACTED*        | `{"username":"*REDACTED*"}`                                                               |                                                                                 | `{}`                            |             |
| `2023-06-14 11:34:16.000` | `Roles.Update`           | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0` | *REDACTED*        | `{"input":{"name":"*REDACTED*","policies":[*REDACTED*]},"managed":true,"id":"*REDACTED"}` | `{"role":{*REDACTED*},"__typename":"RoleUpdateSuccess"}`                        | `{"graphql_path":"roleUpdate"}` |             |
| `2023-06-14 11:34:26.000` | `Auth.Logout`            | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0` | *REDACTED*        | `{}`                                                                                      |                                                                                 | `{}`                            |             |

Which users were active this month and what actions they performed?

```sql
SELECT
json_extract_scalar(useridentity, '$.id') as userid,
array_agg(DISTINCT json_extract_scalar(useridentity, '$.username')) as usernames,
array_agg(DISTINCT json_extract_scalar(useridentity, '$.email')) as emails,
array_agg(DISTINCT json_extract_scalar(useridentity, '$.isadmin')) as isadmin_values,
array_agg(DISTINCT json_extract_scalar(useridentity, '$.roleid')) as roles,
array_agg(DISTINCT sourceipaddress) as ips,
min(eventtime) as time_first,
max(eventtime) as time_last,
array_agg(DISTINCT eventname) as actions
FROM audit_trail
WHERE date BETWEEN date_format(current_date, '%Y/%m/01') AND date_format(current_date, '%Y/%m/31')
AND json_extract_scalar(useridentity, '$.type') = 'QuiltUser'
GROUP BY json_extract_scalar(useridentity, '$.id')
```

Example query result:

| `userid`   | `usernames` | `emails`                | `isadmin_values` | `roles`        | `ips`          | `time_first`              | `time_last`               | `actions`                                                                                                                                       |
| ---------- | ----------- | ----------------------- | ---------------- | -------------- | -------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| *REDACTED* | `[admin]`   | `[admin@example.com]`   | `[true]`         | `[*REDACTED*]` | `[*REDACTED*]` | `2023-06-08 00:16:48.000` | `2023-06-14 11:34:26.000` | `[Users.Disable, Users.Enable, Users.GrantAdmin, Roles.Update, Auth.Logout, Users.RevokeAdmin, Auth.GetAWSCredentials, Auth.Login, Users.List]` |
| *REDACTED* | `[user]`    | `[user@example.com]`    | `[true, false]`  | `[*REDACTED*]` | `[*REDACTED*]` | `2023-06-08 00:16:26.000` | `2023-06-14 13:10:02.000` | `[Users.List, Auth.GetAWSCredentials]`                                                                                                          |
| *REDACTED* | `[_canary]` | `[canary@quiltdata.io]` | `[false]`        | `[*REDACTED*]` | `[*REDACTED*]` | `2023-06-08 00:37:21.000` | `2023-06-14 12:38:20.000` | `[Auth.GetAWSCredentials, Auth.ServiceLogin]`                                                                                                   |

Which unique users logged in this month and which role did they have and were they an admin?

```sql
SELECT
json_extract_scalar(useridentity, '$.id') as userid,
array_agg(DISTINCT json_extract_scalar(useridentity, '$.username')) as usernames,
array_agg(DISTINCT json_extract_scalar(useridentity, '$.email')) as emails,
array_agg(DISTINCT json_extract_scalar(useridentity, '$.isadmin')) as isadmin_values,
array_agg(DISTINCT json_extract_scalar(useridentity, '$.roleid')) as roles
FROM audit_trail
WHERE date BETWEEN date_format(current_date, '%Y/%m/01') AND date_format(current_date, '%Y/%m/31')
AND eventname = 'Auth.Login'
AND errorcode IS NULL
AND json_extract_scalar(useridentity, '$.type') = 'QuiltUser'
GROUP BY json_extract_scalar(useridentity, '$.id')
```

Example query result:

| `userid`   | `usernames` | `emails`              | `isadmin_values` | `roles`        |
| ---------- | ----------- | --------------------- | ---------------- | -------------- |
| *REDACTED* | `[admin]`   | `[admin@example.com]` | `[true]`         | `[*REDACTED*]` |
| *REDACTED* | `[user]`    | `[user@example.com]`  | `[false]`        | `[*REDACTED*]` |


# Organizing S3 Buckets

## Organizing data in S3 buckets

You can think of your data buckets as a grid. The horizontal dimension reflects the data lifecycle phase (raw, refined, curated). The vertical dimension reflects topic or domain.

As discussed under [mental model](/mentalmodel#buckets-are-branches), S3 buckets in Quilt are like branches in git; they reflect how "done" data are.

### Don't rely on folder structure

There's no one folder structure that works for everyone. For instance, if you organize experiments by `department/date` then it's tedious for users to look at everything that happened on a given date. And vice versa. Since there's no one folder structure that works for everyone in the business, files are copied so that they can be found through multiple folder paths. But copies reduce *uniqueness*. As a result users don't know which copy of the file to trust. These copies then diverge, further reducing trust in data.

#### Folder names are not metadata

As users try to keep files organized they put metadata in the file names and paths. This is fragile. First, files and folders can be moved, thus severing their connection to metadata stored in the path. Second, file names with metadata can only get so long before they become unusable. Finally, metadata and folder-naming conventions vary across users, making metadata in the path of low consistency, low quality, and low trust.

We call file paths *physical* views because they are fixed addresses for data on disk or in blob storage.

## Package logical views are more flexible than fixed physical views

For the above reasons reason, packages offer both metadata tags and infinite *logical* views atop *physical* locations in S3. Metadata tags prevent file names from getting longer and longer to hackily include metadata.

With Quilt packages, you can include a given S3 object in as many packages without ever copying that object. Or you can organize multiple S3 buckets into single packages under any logical folder structure that you wish (by using the Quilt catalog or the `.set*()` APIs).

Since packages separate metadata from logical view paths, but include metadata and data in the package manifest, data and metadata can never be accidentally separated. File paths also become simpler and easier to trust.

### S3 data lifecycle

Below is an example of how you might organize three data domains according to a three-phase data lifecycle.

| Domain / Lifecycle phase | Raw              | Refined              | Curated              |
| ------------------------ | ---------------- | -------------------- | -------------------- |
| Domain1                  | s3://domain1-raw | s3://domain1-refined | s3://domain1-curated |
| Domain2                  | s3://domain2-raw | s3://domain2-refined | s3://domain2-curated |
| Domain3                  | s3://domain3-raw | s3://domain3-refined | s3://domain3-curated |

In addition to the three lifecycle phases above it is useful to have on or more "sandbox" buckets where users can create experimental packages without fear of overwriting or disrupting business data.

### How enterprises organize instrument, scientist, ELN data

See [Bringing Scientific Data Management Systems Into the Cloud](https://blog.quiltdata.com/bringing-scientific-data-management-systems-into-the-cloud-video-41be228a41b9) from Bio-IT world for a in-depth view of organizing scientific data in the cloud from instrument, to scientist, to ELN.

![](/files/0rwg7fgAc51GiDOZlzOI)

### How many buckets do I need, and how should I divide data?

Generally, you'll need three S3 buckets for each data domain.

Below are a few questions you can use to determine how many data domains and lifecycle phases you'll need:

1. Are these data related in a meaningful way?

   If not, favor separate buckets.
2. Are these data all a the same level of "doneness"; are they at the same stage in the data quality lifecycle (raw, refined, curated)?
3. Do the data have related access patterns in terms of users and groups?

   If not, favor separate buckets for reasons of IAM simplicity and search speed in Quilt.
4. Do humans or pipelines have delete permissions against these buckets?

   If so, data should be duplicated to another bucket on push to ensure durability.
5. Does this data need to be sealed and linked to an IND filing or ELN?

   If so, favor curated or "production" buckets where all data are self-contained and delete access is exceptional or non-existent.
6. Are we mixing validated and non-validated data?

   If so, favor segregation at the bucket level.
7. Is this data part of a staging or dev environment?

   If so, consider how this data will be copied to more stable buckets, or why it is OK to lose this data at any time.
8. Do you want to be able to trace the data provenance back to its source, for example from instrument, to scientist, to ELN entry?

   If so, consider isolated, only-instruments-can-write buckets for the early data lifecycle, and read-only or write-only buckets for the curated (ELN) phase.


# Installation

Quilt has two user-facing components:

* A Python client
* A web catalog

## Python client

Python 3.10 or higher is required.

```bash
$ pip install 'quilt3[pyarrow]'
```

If you do not need to serialize and deserialize dataframes with Quilt, you can obtain a smaller install, useful in disk-constrained environments like AWS Lambda, with `pip install quilt3`.

If you plan to use [Quilt Catalog Local Development Mode](/quilt-python-sdk/api-reference/localmode), add `catalog` extra while installing `quilt3`, e.g.:

```bash
$ pip install 'quilt3[catalog,pyarrow]'
```

If you wish to use AWS resources, such as S3 buckets, you will need valid AWS credentials. If this is your first time using the AWS CLI, run the following:

```bash
$ aws configure
```

For AWS CLI v2 installation, use the [official AWS CLI install guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

If you are already using the AWS CLI, you may use your existing profile, or [create a new profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html).

### Developer

Install the current Quilt client from `master`:

```bash
$ pip install git+https://github.com/quiltdata/quilt.git#subdirectory=api/python
```

## Enterprise Installation

For installing the Quilt catalog and stack (enterprise deployment on AWS), see the [Catalog Installation Guide](/quilt-platform-administrator/installation).


# Quick Start

Get up and running with Quilt in minutes! This guide provides multiple learning paths based on your experience level and preferred learning style.

## 🚀 Choose Your Learning Path

### 👨‍💻 **For Developers** - Hands-on Python Tutorial

Start coding immediately with our interactive Python tutorial:

* [**Interactive Python Tutorial**](https://open.quiltdata.com/b/quilt-example/packages/examples/quickstart/tree/latest/QuickStart.ipynb) - Learn `quilt3` through practical examples

### 📺 **For Visual Learners** - Video Tutorials

Watch comprehensive video guides:

* [**Complete Video Series**](https://www.youtube.com/playlist?list=PLmXfD6KoA_vBtgGgt0X4ui4cRlEkdJKp9) - How to work with S3 datasets using Quilt
* **Duration**: \~30 minutes total
* **Topics**: Installation, basic operations, data versioning, collaboration

### 📊 **For Data Scientists** - Real Dataset Exploration

Explore production datasets with guided examples:

* [**CORD-19 Dataset Analysis**](https://open.quiltdata.com/b/quilt-example/packages/akarve/cord19) - Real-world COVID research data exploration
* [**Machine Learning with PyTorch**](https://medium.com/pytorch/how-to-iterate-faster-in-machine-learning-by-versioning-data-and-models-featuring-detectron2-4fd2f9338df5) - Versioning data and models for rapid ML experimentation

## ⚡ 5-Minute Quick Start

### 1. **Install Quilt**

```bash
pip install quilt3
```

### 2. **Authenticate (Optional for Public Data)**

For public datasets like `s3://quilt-example`, no authentication is needed. For private buckets or catalogs, choose your authentication method:

```python
import quilt3

# Interactive login (for local development, notebooks)
quilt3.login()  # Opens browser for OAuth/SSO

# OR use an API key (for automation, CI/CD, scripts)
import os
quilt3.login_with_api_key(os.environ["QUILT_API_KEY"])
```

**📚 Learn more**: See the [Authentication Guide](/quilt-python-sdk/api-reference/authentication) for detailed setup instructions, best practices, and use cases.

### 3. **Browse Public Data**

```python
import quilt3

# Browse available datasets (no auth needed for public data)
packages = list(quilt3.list_packages("s3://quilt-example"))
print(f"Found {len(packages)} public datasets")

# Load a sample dataset
pkg = quilt3.Package.browse("examples/hurdat", "s3://quilt-example")
print(pkg)
```

### 4. **Access Your First File**

```python
# Download and read a file (using pkg from previous step)
data_file = pkg["README_NF_QUILT.md"]
content = data_file.get()
print(content)
```

### 5. **Create Your First Package**

```python
import quilt3
import tempfile
import os

# Create a temporary file
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt') as f:
    f.write("Hello, Quilt!")
    temp_file = f.name

# Create a new package
new_pkg = quilt3.Package()
new_pkg.set("my_data.txt", temp_file)
new_pkg.set_meta({"description": "My first Quilt package"})

# Clean up
os.unlink(temp_file)

# Note: Pushing requires S3 credentials, so we'll just show the package
print(f"Package created with {len(new_pkg)} files")
```

## 🎯 Next Steps

### **Beginner Path**

1. ✅ Complete the 5-minute quick start above
2. 📖 Read the [Mental Model](/mentalmodel) to understand Quilt concepts
3. 🔧 Follow the [Installation Guide](/quilt-python-sdk/installation) for your environment
4. 📝 Try the [Basic Workflows](/quilt-python-sdk/uploading-a-package)

### **Intermediate Path**

1. 🏗️ Set up your [AWS Integration](/quilt-platform-administrator/crossaccount)
2. 👥 Configure [Team Collaboration](/quilt-platform-administrator/admin)
3. 🔍 Learn [Advanced Search](/quilt-platform-catalog-user/advanced/working-with-elasticsearch)
4. 📊 Explore [Data Visualization](/quilt-platform-catalog-user/visualizationdashboards)

### **Advanced Path**

1. 🔐 Configure [Cross-Account Access](/quilt-platform-administrator/crossaccount)
2. ⚡ Set up [EventBridge Integration](/quilt-platform-administrator/advanced/eventbridge)
3. 🤖 Implement [Automated Workflows](/workflows)
4. 🔧 Use the [Admin API](/quilt-platform-administrator/admin-1)

## 🌐 Explore Open Data

Discover publicly available datasets:

* [**Open Quilt Data Portal**](https://open.quiltdata.com/) - Browse hundreds of public datasets
* **Featured Collections**: COVID-19 research, climate data, genomics, financial datasets
* **No registration required** - Start exploring immediately

## 💡 Common Use Cases

### **Data Science Teams**

* Version control for datasets and models
* Reproducible research and experiments
* Collaborative data exploration

### **ML/AI Development**

* Dataset versioning for model training
* Experiment tracking and comparison
* Model artifact management

### **Enterprise Data Management**

* Centralized data catalog
* Data governance and compliance
* Cross-team data sharing

### **Research Organizations**

* Research data management
* Publication-ready data packages
* Long-term data preservation

## 🆘 Need Help?

* 📖 **Documentation**: Browse the full [Quilt Documentation](https://github.com/quiltdata/quilt/blob/master/README.md)
* 💬 **Community**: Join our [Slack Community](https://slack.quilt.bio)
* 🐛 **Issues**: Report bugs on [GitHub](https://github.com/quiltdata/quilt/issues)
* 📧 **Support**: Contact <support@quilt.bio>

***

**Ready to dive deeper?** Continue with the [Mental Model](/mentalmodel) to understand how Quilt organizes and manages your data.


# Editing a Package

Data in Quilt is organized in terms of **data packages**. A data package is a logical group of files, directories, and metadata.

### Initializing a package

To edit a new empty package, use the package constructor:

```python
import quilt3
p = quilt3.Package()
```

To edit a preexisting package, we need to first make sure to install the package:

```python
import quilt3
quilt3.Package.install(
    "examples/hurdat",
    "s3://quilt-example",
)
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 5902.48entries/s]

Successfully installed package 'examples/hurdat', tophash=f8d1478 from s3://quilt-example


```

Use `browse` to edit the package:

```python
p = quilt3.Package.browse('examples/hurdat')
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 9920.30entries/s]
```

For more information on accessing existing packages see the section "[Installing a Package](/quilt-python-sdk/installing-a-package)".

### Adding data to a package

Use the `set` and `set_dir` commands to add individual files and whole directories, respectively, to a `Package`:

```python
# Create test directories
import quilt3
from pathlib import Path
from os import chdir
TEST_DIR = "test_workflow"
SUB_DIR = "subdir"

# create test directories
Path(TEST_DIR).mkdir(exist_ok=True)
Path(TEST_DIR, SUB_DIR).mkdir(exist_ok=True)
chdir(TEST_DIR) # %cd TEST_DIR/ if in Jupyter

# add entries individually using `set`
# ie p.set("foo.csv", "/local/path/foo.csv"),
# p.set("bar.csv", "s3://bucket/path/bar.csv")

# create test data
with open("data.csv", "w") as f:
    f.write("id, value\na, 42")

p = quilt3.Package()
p.set("data.csv", "data.csv")
p.set("banner.png", "s3://quilt-example/imgs/banner.png")

# or grab everything in a directory at once using `set_dir`
# ie p.set_dir("stuff/", "/path/to/stuff/"),
# p.set_dir("things/", "s3://path/to/things/")

# create logical directory in package
p.set_dir("stuff/", SUB_DIR)
p.set_dir("imgs/", "s3://quilt-example/imgs/")
```

```
(remote Package)
 └─banner.png
 └─data.csv
 └─imgs/
   └─banner.png
 └─stuff/
```

The first parameter to these functions is the *logical key*, which will determine where the file lives within the package. So after running the commands above our package will look like this:

```python
p
```

```
(remote Package)
 └─banner.png
 └─data.csv
 └─imgs/
   └─banner.png
 └─stuff/
```

The second parameter is the *physical key*, which states the file's actual location. The physical key may point to either a local file or a remote object (with an `s3://` path).

If the physical key and the logical key are the same, you may omit the second argument:

```python
import quilt3
p = quilt3.Package()
p.set("data.csv")
```

```
(local Package)
 └─data.csv
```

Another useful trick. Use `"."` to set the contents of the package to that of the current directory:

```python
# create a test file in test directory
with open("new_data.csv", "w") as f:
    f.write("id, value\na, 42")

# set the contents of the package to that of the current directory
p.set_dir(".", ".")
```

```
(local Package)
 └─data.csv
 └─new_data.csv
```

### Deleting data in a package

Use `delete` to remove entries from a package:

```python
p.delete("data.csv")
```

```
(local Package)
 └─new_data.csv
```

Note that this will only remove this piece of data from the package. It will not delete the actual data itself.

### Adding metadata to a package

Packages support metadata anywhere in the package. To set metadata on package entries or directories, use the `meta` argument:

```python
import quilt3
p = quilt3.Package()
p.set("data.csv", "new_data.csv", meta={"type": "csv"})
p.set_dir("subdir/", "subdir/", meta={"origin": "unknown"})
```

```
(local Package)
 └─data.csv
 └─subdir/
```

You can also set metadata on the package as a whole using `set_meta`.

```python
# set metadata on a package
p.set_meta({"package-type": "demo"})
```

```
(local Package)
 └─data.csv
 └─subdir/
```


# Uploading a Package

Once your package is ready it's time to save and distribute it.

### Saving a package manifest locally

To save a package to your local disk use `build`.

```python
import quilt3
p = quilt3.Package()

top_hash = p.build("aneesh/test_data")
```

Building a package requires providing it with a name. Packages names must follow the `"${namespace}/${packagename}"` format. For small teams, we recommend using the package author's name as the namespace.

### Authenticating to a remote registry

To share a package with others via a remote registry you will first need to authenticate against, if you haven't done so already:

```python
# only need to run this once
# ie quilt3.config('https://your-catalog-homepage/')
quilt3.config('https://open.quiltdata.com/')

# follow the instructions to finish login
quilt3.login()
```

### Pushing a package to a remote registry

To share a package with others via a remote registry, use `push`:

```python
p = quilt3.Package()
p.push(
    "aneesh/test_data",
    "s3://quilt-example",
    message="Updated version my package"
)
```

`s3://quilt-example` is the *registry*—the storage backend that the package is available from.

You can omit the registry argument if you configure a `default_remote_registry` (this setting persists between sessions):

```python
quilt3.config(default_remote_registry='s3://quilt-example')
p = quilt3.Package()
p.push("aneesh/test_data")
```

You can control where files land using `dest`:

```python
p = quilt3.Package()
p.push(
    "aneesh/test_data",
    dest="s3://quilt-example/foo/bar"
)
```

> For even more fine-grained control of object landing paths see [Materialization](/quilt-python-sdk/advanced/materialization).

### Saving a package on a remote registry

`push` will send both a package manifest and its data to a remote registry. This will involve copying your data to S3. To save just the package manifest to S3 without any data copying, use `build`:

```python
p = quilt3.Package()
p.build("aneesh/test_data", "s3://quilt-example")
```

This will create a new version of your package with all of its physical keys preserved.

### Delete a package from a registry

To delete a package from a registry:

```python
# delete a package in the local registry
quilt3.delete_package("aneesh/test_data")

# delete a package in a remote registry
quilt3.delete_package("aneesh/test_data", "s3://quilt-example")
```

Note that this will not delete any package data, only the package manifest.


# Installing a Package

### Searching for packages

As explained in ["Uploading a Package"](/quilt-python-sdk/uploading-a-package), packages are managed using *registries*. There is a one local registry on your machine, and potentially many remote registries elsewhere "in the world". Use `list_packages` to see the packages available on a registry:

```python
import quilt3 # list local packages
list(quilt3.list_packages())
```

```
['aneesh/cli-push',
 'examples/hurdat',
 'aleksey/hurdat']
```

```python
import quilt3 # list remote packages
list(quilt3.list_packages("s3://quilt-example"))
```

```
['aleksey/hurdat',
 'examples/hurdat',
 'quilt/altair',
 'quilt/hurdat',
 'quilt/open_fruit',
 'quilt/open_images']
```

### Installing a package

To make a remote package and all of its data available locally, `install` it.

The examples in this section use the `examples/hurdat` [demo package](https://open.quiltdata.com/b/quilt-example/tree/examples/hurdat/):

```python
import quilt3 # install package
quilt3.Package.install(
    "examples/hurdat",
    "s3://quilt-example",
)
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 7049.25entries/s]

Successfully installed package 'examples/hurdat', tophash=f8d1478 from s3://quilt-example


```

Note that unless this registry is public, you will need to be logged into a user who has read access to this registry in order to install from it:

```python
# only need to run this once
# ie quilt3.config('https://your-catalog-homepage/')
quilt3.config('https://open.quiltdata.com/')

# follow the instructions to finish login
quilt3.login()
```

Data files that you download are written to a folder in your local registry by default. You can specify an alternative destination using dest:

```python
import quilt3  # install package into a specific folder
quilt3.Package.install(
    "examples/hurdat",
    "s3://quilt-example",
    dest="./"
)
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 9027.77entries/s]
Copying objects: 100%|██████████| 3.62M/3.62M [00:00<00:00, 303MB/s]

Successfully installed package 'examples/hurdat', tophash=f8d1478 from s3://quilt-example


```

Finally, you can install a specific version of a package by specifying the corresponding top hash:

```python
import quilt3  # install specific version of package
quilt3.Package.install(
    "examples/hurdat",
    "s3://quilt-example",
    top_hash="058e62c"
)
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 11491.24entries/s]
Copying objects: 100%|██████████| 35.4k/35.4k [00:02<00:00, 14.3kB/s]

Successfully installed package 'examples/hurdat', tophash=058e62c from s3://quilt-example


```

### Browsing a package manifest

An alternative to `install` is `browse`. `browse` downloads a package manifest without also downloading the data in the package.

```python
import quilt3 # load a package manifest from a remote registry
p = quilt3.Package.browse("examples/hurdat", "s3://quilt-example")

# load a package manifest from the default remote registry
quilt3.config(default_remote_registry="s3://quilt-example")
p = quilt3.Package.browse("examples/hurdat")
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 7541.00entries/s]
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 10710.68entries/s]
```

`browse` is advantageous when you don't want to download everything in a package at once. For example if you just want to look at a package's metadata.

### Importing a package

You can import a local package from within Python:

```python
from quilt3.data.examples import hurdat
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 9637.65entries/s]
```

This allows you to manage your data and code dependencies all in one place in your Python scripts or Jupyter notebooks.


# Getting Data from a Package

The examples in this section use the `aleksey/hurdat` [demo package](https://open.quiltdata.com/b/quilt-example/tree/aleksey/hurdat/):

```python
import quilt3
p = quilt3.Package.browse('aleksey/hurdat', 's3://quilt-example')
p
```

```
Loading manifest: 100%|██████████| 7/7 [00:00<00:00, 8393.40entries/s]





(remote Package)
 └─.gitignore
 └─.quiltignore
 └─notebooks/
   └─QuickStart.ipynb
 └─quilt_summarize.json
 └─requirements.txt
 └─scripts/
   └─build.py
```

### Slicing through a package

Use `dict` key selection to slice into a package tree:

```python
# returns PackageEntry("requirements.txt")
p["requirements.txt"]
```

```
PackageEntry('s3://quilt-example/aleksey/hurdat/requirements.txt?versionId=bQtxuZlaylNVHi0GmxkSMofT5qXJvP95')
<!--pytest-codeblocks:cont-->
```

```python
# returns (remote Package)
p["notebooks"]
```

```
(remote Package)
 └─QuickStart.ipynb
```

Slicing into a `Package` directory returns another `Package` rooted at that subdirectory. Slicing into a package entry returns an individual `PackageEntry`.

### Downloading package data to disk

To download a subset of files from a package directory to a `dest`, use `fetch`:

```python
# download a subfolder
p["notebooks"].fetch()

# download a single file
p["notebooks"]["QuickStart.ipynb"].fetch()

# download everything
p.fetch()
```

```
Copying objects: 100%|██████████| 36.7k/36.7k [00:01<00:00, 22.7kB/s]
100%|██████████| 36.7k/36.7k [00:01<00:00, 24.1kB/s]
Copying objects: 100%|██████████| 39.9k/39.9k [00:02<00:00, 16.5kB/s]





(local Package)
 └─.gitignore
 └─.quiltignore
 └─notebooks/
   └─QuickStart.ipynb
 └─quilt_summarize.json
 └─requirements.txt
 └─scripts/
   └─build.py
```

`fetch` will default to downloading the files to the current directory, but you can also specify an alternative path:

```python
p["notebooks"]["QuickStart.ipynb"].fetch("./references/")
```

```
100%|██████████| 36.7k/36.7k [00:01<00:00, 22.5kB/s]





PackageEntry('file:///Users/gregezema/Documents/programs/quilt/docs/Walkthrough/references/')
```

### Downloading package data into memory

Alternatively, you can download data directly into memory:

```python
p["quilt_summarize.json"].deserialize()
```

```
['notebooks/QuickStart.ipynb']
```

To apply a custom deserializer to your data, pass the function as a parameter to the function. For example, to load a hypothetical `yaml` file using `yaml.safe_load`:

```python
import yaml
# returns a dict
p["quilt_summarize.json"].deserialize(yaml.safe_load)
```

```
['notebooks/QuickStart.ipynb']
```

The deserializer should accept a byte stream as input.

### Getting entry locations

You can get the path to a package entry or directory using `get`:

```python
# returns /path/to/pkg/root/notebooks/QuickStart.ipynb
p["notebooks"]["QuickStart.ipynb"].get()
```

```
's3://quilt-example/aleksey/hurdat/notebooks/QuickStart.ipynb?versionId=PH.9gsCH6LM9RQIqsy1U4X6H6s.VoQ_B'
```

### Getting metadata

Metadata is available using the `meta` property.

```python
# get entry metadata
p["notebooks"]["QuickStart.ipynb"].meta

# get directory metadata
p["notebooks"].meta

# get package metadata
p.meta
```


# Example: Git-like Operations

`quilt3` provides a simple command-line for versioning large datasets and storing them in Amazon S3. There are only two commands you need to know:

* `push` creates a new package revision in an S3 bucket that you designate
* `install` downloads data from a remote package to disk

## Why not use Git?

In short, neither Git nor Git LFS have the capacity or performance to function as a repository for data. S3, on the other hand, is widely used, fast, supports versioning, and currently stores some trillions of data objects.

Similar concerns apply when baking datasets into Docker containers: images bloat and slow container operations down.

## Pre-requisites

You will need either an AWS account, credentials, and an S3 bucket, OR a Quilt enterprise stack with at least one bucket. In order to read from and write to S3 with `quilt3`, you must first do one of the following:

* [Configure your AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
* OR, if and only if your company runs a Quilt enterprise stack, run the following:

  ```bash
  pip install quilt3
  quilt3 config https://yourquilt.yourcompany.com
  quilt3 login
  ```

## Install a package

A Quilt package contains any collection of data (usually as files), metadata, and documentation that you specify.

Let's get a data package from S3 and write it `quilt-hurdat/data` .

```bash
mkdir reef-check
cd reef-check
quilt3 install \
    "akarve/reef-check" \
    --registry s3://quilt-example \
    --dest .
```

Now you've got data in the current working directory.

```
ls
CA-06-california-counties.json	quilt_summarize.json  urchins-interactive.json
README.md			reef-check.ipynb      urchins2006-2019.parquet
```

## Creating your first package

Now let's imagine that we've modified this data locally. We save our Jupyter notebook and push the results back to Quilt:

```bash
# Be sure to substitute YOUR_NAME and YOUR_BUCKET with the desired strings
quilt3 push \
    YOUR_NAME/reef-check \
    --dir . \
    --registry s3://YOUR_BUCKET \
    --message "Initial commit of reef data"
```

Quilt will then print out something like the following:

```bash
Package YOUR_NAME/reef-check@ea334b7 pushed to s3://YOUR_BUCKET
Successfully pushed the new package to https://yourquilt.yourocmpany.com/b/YOUR_NAME/packages/akarve/reef-check
```

## List the packages in a bucket

```bash
quilt3 list-packages s3://YOUR_BUCKET
```

In the Quilt catalog, you will now see a new package revision, complete with a README, datagrid preview, and an interactive visualization in Altair.

![](/files/EkBN8iCx46m2NNyqZUfb) ![](/files/Enm37TckzsVKwGcvLRxc)

You can see an example of this package live [here](https://open.quiltdata.com/b/quilt-example/packages/akarve/reef-check/tree/8a0d875d5f46355a770143d66c4a1cfea43c7ffa215db600b9ff88fafe687676/).

## Learn more

Those are the basics of reading and writing Quilt packages with the CLI. See the [CLI reference](/quilt-python-sdk/api-reference/cli) for more.


# API Reference


# quilt3

## quilt3

Quilt API

### clear\_api\_key() <a href="#clear_api_key" id="clear_api_key"></a>

Clear the API key and fall back to interactive session (if available).

### config(\*catalog\_url, \*\*config\_values) <a href="#config" id="config"></a>

Set or read the QUILT configuration.

To retrieve the current config, call directly, without arguments:

```
import quilt3
quilt3.config()
```

To trigger autoconfiguration, call with just the navigator URL:

```
import quilt3
quilt3.config('https://YOUR-CATALOG-URL.com')
```

To set config values, call with one or more key=value pairs:

```
import quilt3
quilt3.config(navigator_url='http://example.com')
```

Default config values can be found in `quilt3.util.CONFIG_TEMPLATE`.

**Arguments**

* **catalog\_url**: A (single) URL indicating a location to configure from
* **\*\*config\_values**: `key=value` pairs to set in the config

**Returns**

`QuiltConfig`: (an ordered Mapping)

### delete\_package(name, registry=None, top\_hash=None) <a href="#delete_package" id="delete_package"></a>

Delete a package. Deletes only the manifest entries and not the underlying files.

**Arguments**

* **name (str)**: Name of the package
* **registry (str)**: The registry the package will be removed from
* **top\_hash (str)**: Optional. A package hash to delete, instead of the whole package.

### get\_boto3\_session(\*, fallback: bool = True) -> boto3.session.Session <a href="#get_boto3_session" id="get_boto3_session"></a>

Return a Boto3 session with Quilt stack credentials and AWS region. In case of no Quilt credentials found, return a "normal" Boto3 session if `fallback` is `True`, otherwise raise a `QuiltException`.

> Note: you need to call `quilt3.config("https://your-catalog-homepage/")` to have region set on the session, if you previously called it in quilt3 < 6.1.0.

### list\_package\_versions(name, registry=None) <a href="#list_package_versions" id="list_package_versions"></a>

Lists versions of a given package.

Returns an iterable of (latest\_or\_unix\_ts, hash) of package revisions. If the registry is None, default to the local registry.

**Arguments**

* **name (str)**: Name of the package
* **registry (str)**: location of registry to load package from.

**Returns**

An iterable of tuples containing the version and hash for the package.

### list\_packages(registry=None) <a href="#list_packages" id="list_packages"></a>

Lists Packages in the registry.

Returns an iterable of all named packages in a registry. If the registry is None, default to the local registry.

**Arguments**

* **registry (str)**: location of registry to load package from.

**Returns**

An iterable of strings containing the names of the packages

### logged\_in() <a href="#logged_in" id="logged_in"></a>

Return catalog URL if Quilt client is authenticated, `None` otherwise.

### login() <a href="#login" id="login"></a>

Authenticate to your Quilt stack and assume the role assigned to you by your stack administrator. Not required if you have existing AWS credentials.

Launches a web browser and asks the user for a token.

### login\_with\_api\_key(key: str) <a href="#login_with_api_key" id="login_with_api_key"></a>

Authenticate using an API key.

The API key is stored in memory only (no disk persistence). While set, the API key overrides any interactive session. Use clear\_api\_key() to revert to interactive session.

**Arguments**

* **key**: API key string (starts with 'qk\_')

**Raises**

* `ValueError`: If the key doesn't start with 'qk\_' prefix.

### logout() <a href="#logout" id="logout"></a>

Do not use Quilt credentials. Useful if you have existing AWS credentials.

### search(query: str | dict, limit: int = 10) -> List\[dict] <a href="#search" id="search"></a>

Execute a search against the configured search endpoint.

**Arguments**

* **query**: query string to query if passed as `str`, DSL query body if passed as `dict`
* **limit**: maximum number of results to return. Defaults to 10

Query Syntax: [Query String Query](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl-query-string-query.html) [Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl.html)

Index schemas and search examples can be found in the [Quilt Search documentation](https://docs.quilt.bio/quilt-platform-catalog-user/search).

**Returns**

search results

## quilt3.api\_keys

API for managing your own API keys.

### APIKey(id: str, name: str, fingerprint: str, created\_at: datetime.datetime, expires\_at: datetime.datetime, last\_used\_at: datetime.datetime | None, status: Literal\['ACTIVE', 'EXPIRED'], user\_email: str) -> None <a href="#apikey" id="apikey"></a>

An API key for programmatic access.

### APIKeyError(result) <a href="#apikeyerror" id="apikeyerror"></a>

Error during API key operation.

### list(name: str | None = None, fingerprint: str | None = None, status: Literal\['ACTIVE', 'EXPIRED'] | None = None) -> List\[quilt3.api\_keys.APIKey] <a href="#list" id="list"></a>

List your API keys. Optionally filter by name, fingerprint, or status.

**Arguments**

* **name**: Filter by key name.
* **fingerprint**: Filter by key fingerprint.
* **status**: Filter by "ACTIVE" or "EXPIRED". None returns all.

**Returns**

List of your API keys matching the filters.

### get(id: str) -> quilt3.api\_keys.APIKey | None <a href="#get" id="get"></a>

Get a specific API key by ID.

**Arguments**

* **id**: The API key ID.

**Returns**

The API key, or None if not found.

### create(name: str, expires\_in\_days: int = 90) -> Tuple\[quilt3.api\_keys.APIKey, str] <a href="#create" id="create"></a>

Create a new API key for yourself.

**Arguments**

* **name**: Name for the API key.
* **expires\_in\_days**: Days until expiration (1-365, default 90).

**Returns**

Tuple of (APIKey, secret). The secret is only returned once - save it securely!

**Raises**

* `APIKeyError`: If the operation fails.

### revoke(id: str | None = None, secret: str | None = None) -> None <a href="#revoke" id="revoke"></a>

Revoke an API key. Provide either the key ID or the secret.

**Arguments**

* **id**: The API key ID to revoke.
* **secret**: The API key secret to revoke.

**Raises**

* `ValueError`: If neither id nor secret is provided.
* `APIKeyError`: If the operation fails.


# quilt3.Package

## Package() <a href="#package" id="package"></a>

In-memory representation of a package

### manifest

Provides a generator of the dicts that make up the serialized package.

### top\_hash

Returns the top hash of the package.

Note that physical keys are not hashed because the package has the same semantics regardless of where the bytes come from.

**Returns**

A string that represents the top hash of the package

### Package.\_\_repr\_\_(self, max\_lines=20) <a href="#package.__repr" id="package.__repr"></a>

String representation of the Package.

### Package.install(name, registry=None, top\_hash=None, dest=None, dest\_registry=None, \*, path=None) <a href="#package.install" id="package.install"></a>

Installs a named package to the local registry and downloads its files.

**Arguments**

* **name(str)**: Name of package to install.
* **registry(str)**: Registry where package is located. Defaults to the default remote registry.
* **top\_hash(str)**: Hash of package to install. Defaults to latest.
* **dest(str)**: Local path to download files to.
* **dest\_registry(str)**: Registry to install package to. Defaults to local registry.
* **path(str)**: If specified, downloads only `path` or its children.

### Package.resolve\_hash(name, registry, hash\_prefix) <a href="#package.resolve_hash" id="package.resolve_hash"></a>

Find a hash that starts with a given prefix.

**Arguments**

* **name (str)**: name of package
* **registry (str)**: location of registry
* **hash\_prefix (str)**: hash prefix with length between 6 and 64 characters

### Package.browse(name, registry=None, top\_hash=None) <a href="#package.browse" id="package.browse"></a>

Load a package into memory from a registry without making a local copy of the manifest.

**Arguments**

* **name(string)**: name of package to load
* **registry(string)**: location of registry to load package from
* **top\_hash(string)**: top hash of package version to load

### Package.\_\_contains\_\_(self, logical\_key) <a href="#package.__contains" id="package.__contains"></a>

Checks whether the package contains a specified logical\_key.

**Returns**

True or False

### Package.\_\_getitem\_\_(self, logical\_key) <a href="#package.__getitem" id="package.__getitem"></a>

Filters the package based on prefix, and returns either a new Package or a PackageEntry.

**Arguments**

* **prefix(str)**: prefix to filter on

**Returns**

PackageEntry if prefix matches a logical\_key exactly otherwise Package

### Package.fetch(self, dest='./') <a href="#package.fetch" id="package.fetch"></a>

Copy all descendants to `dest`. Descendants are written under their logical names *relative* to self.

**Arguments**

* **dest**: where to put the files (locally)

**Returns**

A new Package object with entries from self, but with physical keys pointing to files in `dest`.

### Package.keys(self) <a href="#package.keys" id="package.keys"></a>

Returns logical keys in the package.

### Package.walk(self) <a href="#package.walk" id="package.walk"></a>

Generator that traverses all entries in the package tree and returns tuples of (key, entry), with keys in alphabetical order.

### Package.load(readable\_file) <a href="#package.load" id="package.load"></a>

Loads a package from a readable file-like object.

**Arguments**

* **readable\_file**: readable file-like object to deserialize package from

**Returns**

A new Package object

**Raises**

file not found json decode error invalid package exception

### Package.set\_dir(self, lkey, path=None, meta=None, update\_policy='incoming', unversioned: bool = False) <a href="#package.set_dir" id="package.set_dir"></a>

Adds all files from `path` to the package.

Recursively enumerates every file in `path`, and adds them to the package according to their relative location to `path`.

**Arguments**

* **lkey(string)**: prefix to add to every logical key, use '/' for the root of the package.
* **path(string)**: path to scan for files to add to package. If None, lkey will be substituted in as the path.
* **meta(dict)**: user level metadata dict to attach to lkey directory entry.
* **update\_policy(str)**: can be either 'incoming' (default) or 'existing'. If 'incoming', whenever logical keys match, always take the new entry from set\_dir. If 'existing', whenever logical keys match, retain existing entries and ignore new entries from set\_dir.
* **unversioned(bool)**: when True, do not retrieve VersionId for S3 physical keys.

**Returns**

self

**Raises**

* `PackageException`: When `path` doesn't exist.
* `ValueError`: When `update_policy` is invalid.

### Package.get(self, logical\_key) <a href="#package.get" id="package.get"></a>

Gets object from logical\_key and returns its physical path. Equivalent to self\[logical\_key].get().

**Arguments**

* **logical\_key(string)**: logical key of the object to get

**Returns**

Physical path as a string.

**Raises**

* `KeyError`: when logical\_key is not present in the package
* `ValueError`: if the logical\_key points to a Package rather than PackageEntry.

### Package.readme(self) <a href="#package.readme" id="package.readme"></a>

Returns the README PackageEntry

The README is the entry with the logical key 'README.md' (case-sensitive). Will raise a QuiltException if no such entry exists.

### Package.set\_meta(self, meta) <a href="#package.set_meta" id="package.set_meta"></a>

Sets user metadata on this Package.

### Package.build(self, name, registry=None, message=None, \*, workflow=Ellipsis) <a href="#package.build" id="package.build"></a>

Serializes this package to a registry.

**Arguments**

* **name**: optional name for package
* **registry**: registry to build to defaults to local registry
* **message**: the commit message of the package
* **workflow**: workflow ID or `None` to skip workflow validation. If not specified, the default workflow will be used.
* **For details see**: <https://docs.quilt.bio/advanced-usage/workflows>

**Returns**

The top hash as a string.

### Package.dump(self, writable\_file) <a href="#package.dump" id="package.dump"></a>

Serializes this package to a writable file-like object.

**Arguments**

* **writable\_file**: file-like object to write serialized package.

**Returns**

None

**Raises**

fail to create file fail to finish write

### Package.set(self, logical\_key, entry=None, meta=None, serialization\_location=None, serialization\_format\_opts=None, unversioned: bool = False) <a href="#package.set" id="package.set"></a>

Returns self with the object at logical\_key set to entry.

**Arguments**

* **logical\_key(string)**: logical key to update
* **entry(PackageEntry OR string OR object)**: new entry to place at logical\_key in the package. If entry is a string, it is treated as a URL, and an entry is created based on it. If entry is None, the logical key string will be substituted as the entry value. If entry is an object and quilt knows how to serialize it, it will immediately be serialized and written to disk, either to serialization\_location or to a location managed by quilt. List of types that Quilt can serialize is available by calling `quilt3.formats.FormatRegistry.all_supported_formats()`
* **meta(dict)**: user level metadata dict to attach to entry
* **serialization\_format\_opts(dict)**: Optional. If passed in, only used if entry is an object. Options to help Quilt understand how the object should be serialized. Useful for underspecified file formats like csv when content contains confusing characters. Will be passed as kwargs to the FormatHandler.serialize() function. See docstrings for individual FormatHandlers for full list of options -
* **https**: //github.com/quiltdata/quilt/blob/master/api/python/quilt3/formats.py
* **serialization\_location(string)**: Optional. If passed in, only used if entry is an object. Where the serialized object should be written, e.g. "./mydataframe.parquet"
* **unversioned(bool)**: when True, do not retrieve VersionId for S3 physical keys.

**Returns**

self

### Package.delete(self, logical\_key) <a href="#package.delete" id="package.delete"></a>

Returns self with logical\_key removed.

**Returns**

self

**Raises**

* `KeyError`: when logical\_key is not present to be deleted

### Package.push(self, name, registry=None, dest=None, message=None, selector\_fn=None, \*, workflow=Ellipsis, force: bool = False, dedupe: bool = False) <a href="#package.push" id="package.push"></a>

Creates a new package, or a new revision of an existing package in a package registry in Amazon S3.

By default, any files not currently in the destination bucket are copied to the destination S3 bucket at a path matching logical key structure. Files in the destination bucket are not copied even if they are not located in in the location matching the logical key. After objects are copied, a new package manifest is package manifest is created that points to the objects in their new locations.

The optional parameter `selector_fn` allows callers to choose which files are copied to the destination bucket, and which retain their existing physical key. When using selector functions, it is important to always copy local files to S3, otherwise the resulting package will be inaccessible to users accessing it from Amazon S3.

The Package class includes two additional built-in selector functions:

* `Package.selector_fn_copy_all` copies all files to the destination path regardless of their current location.
* `Package.selector_fn_copy_local` copies only local files to the destination path. Any PackageEntry's with physical keys pointing to objects in other buckets will retain their existing physical keys in the resulting package.

If we have a package with entries:

* `pkg["entry_1"].physical_key = s3://bucket1/folder1/entry_1`
* `pkg["entry_2"].physical_key = s3://bucket2/folder2/entry_2`

And, we call `pkg.push("user/pkg_name", registry="s3://bucket2")`, the file referenced by `entry_1` will be copied, while the file referenced by `entry_2` will not. The resulting package will have the following entries:

* `pkg["entry_1"].physical_key = s3://bucket2/user/pkg_name/entry_1`
* `pkg["entry_2"].physical_key = s3://bucket2/folder1/entry_2`

Quilt3 Versions 6.3.1 and earlier copied all files to the destination path by default. To match this behavior in later versions, callers should use `selector_fn=Package.selector_fn_copy_all`.

Using the same initial package and push, but adding `selector_fn=Package.selector_fn_copy_all` will result in both files being copied to the destination path, producing the following package:

* `pkg["entry_1"].physical_key = s3://bucket2/user/pkg_name/entry_1`
* `pkg["entry_2"].physical_key = s3://bucket2/user/pkg_name/entry_2`

Note that push is careful to not push data unnecessarily. To illustrate, imagine you have a PackageEntry: `pkg["entry_1"].physical_key = "/tmp/package_entry_1.json"`

If that entry would be pushed to `s3://bucket/prefix/entry_1.json`, but `s3://bucket/prefix/entry_1.json` already contains the exact same bytes as '/tmp/package\_entry\_1.json', `quilt3` will not push the bytes to S3, no matter what `selector_fn('entry_1', pkg["entry_1"])` returns.

By default, push will not overwrite an existing package if its top hash does not match the parent hash of the package being pushed. Use `force=True` to skip the check.

**Arguments**

* **name**: name for package in registry
* **dest**: where to copy the objects in the package. Must be either an S3 URI prefix (e.g., s3://$bucket/$key) in the registry bucket, or a callable that takes logical\_key and package\_entry, and returns an S3 URI. (Changed in 6.0.0a1) previously top\_hash was passed to the callable dest as a third argument.
* **registry**: registry where to create the new package
* **message**: the commit message for the new package
* **selector\_fn**: An optional function that determines which package entries should be copied to S3. The function takes in two arguments, logical\_key and package\_entry, and should return False if that PackageEntry should not be copied to the destination registry during push. If for example you have a package where the files are spread over multiple buckets and you add a single local file, you can use selector\_fn to only push the local file to S3 (instead of pushing all data to the destination bucket).
* **workflow**: workflow ID or `None` to skip workflow validation. If not specified, the default workflow will be used.
* **For details see**: <https://docs.quilt.bio/advanced-usage/workflows>
* **force**: skip the top hash check and overwrite any existing package
* **dedupe**: don't push if the top hash matches the existing package top hash; return the current package

**Returns**

A new package that points to the copied objects.

### Package.rollback(name, registry, top\_hash) <a href="#package.rollback" id="package.rollback"></a>

Set the "latest" version to the given hash.

**Arguments**

* **name(str)**: Name of package to rollback.
* **registry(str)**: Registry where package is located.
* **top\_hash(str)**: Hash to rollback to.

### Package.diff(self, other\_pkg) <a href="#package.diff" id="package.diff"></a>

Returns three lists -- added, modified, deleted.

Added: present in other\_pkg but not in self. Modified: present in both, but different. Deleted: present in self, but not other\_pkg.

**Arguments**

* **other\_pkg**: Package to diff

**Returns**

added, modified, deleted (all lists of logical keys)

### Package.map(self, f, include\_directories=False) <a href="#package.map" id="package.map"></a>

Performs a user-specified operation on each entry in the package.

**Arguments**

* **f(x, y)**: function The function to be applied to each package entry. It should take two inputs, a logical key and a PackageEntry.
* **include\_directories**: bool Whether or not to include directory entries in the map.

Returns: list The list of results generated by the map.

### Package.filter(self, f, include\_directories=False) <a href="#package.filter" id="package.filter"></a>

Applies a user-specified operation to each entry in the package, removing results that evaluate to False from the output.

**Arguments**

* **f(x, y)**: function The function to be applied to each package entry. It should take two inputs, a logical key and a PackageEntry. This function should return a boolean.
* **include\_directories**: bool Whether or not to include directory entries in the map.

**Returns**

A new package with entries that evaluated to False removed

### Package.verify(self, src, extra\_files\_ok=False) <a href="#package.verify" id="package.verify"></a>

Check if the contents of the given directory matches the package manifest.

**Arguments**

* **src(str)**: URL of the directory
* **extra\_files\_ok(bool)**: Whether extra files in the directory should cause a failure.

**Returns**

True if the package matches the directory; False otherwise.

## PackageEntry(physical\_key, size, hash\_obj, meta) <a href="#packageentry" id="packageentry"></a>

Represents an entry at a logical key inside a package.

**\_\_init\_\_**

Creates an entry.

**Arguments**

* **physical\_key**: a URI (either `s3://` or `file://`)
* **size(number)**: size of object in bytes
* **hash({'type'**: string, 'value': string}): hash object
* **for example**: {'type': 'SHA256', 'value': 'bb08a...'}
* **meta(dict)**: metadata dictionary

**Returns**

a PackageEntry

### PackageEntry.as\_dict(self) <a href="#packageentry.as_dict" id="packageentry.as_dict"></a>

Returns dict representation of entry.

### PackageEntry.set\_meta(self, meta) <a href="#packageentry.set_meta" id="packageentry.set_meta"></a>

Sets the user\_meta for this PackageEntry.

### PackageEntry.set(self, path=None, meta=None) <a href="#packageentry.set" id="packageentry.set"></a>

Returns self with the physical key set to path.

**Arguments**

* **path(string)**: new path to place at logical\_key in the package Currently only supports a path on local disk
* **meta(dict)**: metadata dict to attach to entry. If meta is provided, set just updates the meta attached to logical\_key without changing anything else in the entry

**Returns**

self

### PackageEntry.get(self) <a href="#packageentry.get" id="packageentry.get"></a>

Returns the physical key of this PackageEntry.

### PackageEntry.get\_cached\_path(self) <a href="#packageentry.get_cached_path" id="packageentry.get_cached_path"></a>

Returns a locally cached physical key, if available.

### PackageEntry.get\_bytes(self, use\_cache\_if\_available=True) <a href="#packageentry.get_bytes" id="packageentry.get_bytes"></a>

Returns the bytes of the object this entry corresponds to. If 'use\_cache\_if\_available'=True, will first try to retrieve the bytes from cache.

### PackageEntry.get\_as\_json(self, use\_cache\_if\_available=True) <a href="#packageentry.get_as_json" id="packageentry.get_as_json"></a>

Returns a JSON file as a `dict`. Assumes that the file is encoded using utf-8.

If 'use\_cache\_if\_available'=True, will first try to retrieve the object from cache.

### PackageEntry.get\_as\_string(self, use\_cache\_if\_available=True) <a href="#packageentry.get_as_string" id="packageentry.get_as_string"></a>

Return the object as a string. Assumes that the file is encoded using utf-8.

If 'use\_cache\_if\_available'=True, will first try to retrieve the object from cache.

### PackageEntry.deserialize(self, func=None, \*\*format\_opts) <a href="#packageentry.deserialize" id="packageentry.deserialize"></a>

Returns the object this entry corresponds to.

**Arguments**

* **func**: Skip normal deserialization process, and call func(bytes), returning the result directly.
* **\*\*format\_opts**: Some data formats may take options. Though normally handled by metadata, these can be overridden here.

**Returns**

The deserialized object from the logical\_key

**Raises**

physical key failure hash verification fail when deserialization metadata is not present

### PackageEntry.fetch(self, dest=None) <a href="#packageentry.fetch" id="packageentry.fetch"></a>

Gets objects from entry and saves them to dest.

**Arguments**

* **dest**: where to put the files Defaults to the entry name

**Returns**

None

### PackageEntry.\_\_call\_\_(self, func=None, \*\*kwargs) <a href="#packageentry.__call" id="packageentry.__call"></a>

Shorthand for self.deserialize()


# quilt3.Bucket

Bucket interface for Quilt.

**\_\_init\_\_**

Creates a Bucket object.

**Arguments**

* **bucket\_uri(str)**: URI of bucket to target. Must start with 's3://'

**Returns**

A new Bucket

## Bucket.search(self, query: str | dict, limit: int = 10) -> List\[dict] <a href="#bucket.search" id="bucket.search"></a>

Execute a search against the configured search endpoint.

**Arguments**

* **query**: query string to query if passed as `str`, DSL query body if passed as `dict`
* **limit**: maximum number of results to return. Defaults to 10

Query Syntax: [Query String Query](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl-query-string-query.html) [Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl.html)

Index schemas and search examples can be found in the [Quilt Search documentation](https://docs.quilt.bio/quilt-platform-catalog-user/search).

**Returns**

search results

## Bucket.put\_file(self, key, path) <a href="#bucket.put_file" id="bucket.put_file"></a>

Stores file at path to key in bucket.

**Arguments**

* **key(str)**: key in bucket to store file at
* **path(str)**: string representing local path to file

**Returns**

None

**Raises**

* if no file exists at path
* if copy fails

## Bucket.put\_dir(self, key, directory) <a href="#bucket.put_dir" id="bucket.put_dir"></a>

Stores all files in the `directory` under the prefix `key`.

**Arguments**

* **key(str)**: prefix to store files under in bucket
* **directory(str)**: path to directory to grab files from

**Returns**

None

**Raises**

* if writing to bucket fails

## Bucket.keys(self) <a href="#bucket.keys" id="bucket.keys"></a>

Lists all keys in the bucket.

**Returns**

List of strings

## Bucket.delete(self, key) <a href="#bucket.delete" id="bucket.delete"></a>

Deletes a key from the bucket.

**Arguments**

* **key(str)**: key to delete

**Returns**

None

**Raises**

* if delete fails

## Bucket.delete\_dir(self, path) <a href="#bucket.delete_dir" id="bucket.delete_dir"></a>

Delete a directory and all of its contents from the bucket.

**Arguments**

* **path (str)**: path to the directory to delete

## Bucket.ls(self, path=None, recursive=False) <a href="#bucket.ls" id="bucket.ls"></a>

List data from the specified path.

**Arguments**

* **path (str)**: bucket path to list
* **recursive (bool)**: show subdirectories and their contents as well

**Returns**

`list`: Return value structure has not yet been permanently decided Currently, it's a `tuple` of `list` objects, containing the `following`: (directory info, file/object info, delete markers).

## Bucket.fetch(self, key, path) <a href="#bucket.fetch" id="bucket.fetch"></a>

Fetches file (or files) at `key` to `path`.

If `key` ends in '/', then all files with the prefix `key` will match and will be stored in a directory at `path`.

Otherwise, only one file will be fetched and it will be stored at `path`.

**Arguments**

* **key(str)**: key in bucket to fetch
* **path(str)**: path in local filesystem to store file or files fetched

**Returns**

None

**Raises**

* if path doesn't exist
* if download fails

## Bucket.select(self, key, query, raw=False) <a href="#bucket.select" id="bucket.select"></a>

Selects data from an S3 object.

**Arguments**

* **key(str)**: key to query in bucket
* **query(str)**: query to execute (SQL by default)
* **query\_type(str)**: other query type accepted by S3 service
* **raw(bool)**: return the raw (but parsed) response

**Returns**

`pandas.DataFrame`: results of query


# quilt3.hooks

## get\_build\_s3\_client\_hook() -> quilt3.hooks.BuildClientHook | None <a href="#get_build_s3_client_hook" id="get_build_s3_client_hook"></a>

Return build S3 client hook.

## set\_build\_s3\_client\_hook(hook: quilt3.hooks.BuildClientHook | None) -> quilt3.hooks.BuildClientHook | None <a href="#set_build_s3_client_hook" id="set_build_s3_client_hook"></a>

Set build S3 client hook.

Example for overriding `ServerSideEncryption` parameter for certain S3 operations:

```python
from quilt3.hooks import set_build_s3_client_hook

def event_handler(params, **kwargs):
    # Be mindful with parameters you set here.
    # Specifically it's not recommended to override/delete already set parameters
    # because that can break quilt3 logic.
    params.setdefault("ServerSideEncryption", "AES256")

def hook(build_client_base, session, client_kwargs, **kwargs):
    client = build_client_base(session, client_kwargs, **kwargs)
    # Docs for boto3 events system we use below:
    # https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
    for op in (
        "CreateMultipartUpload",
        "CopyObject",
        "PutObject",
    ):
        client.meta.events.register(f"before-parameter-build.s3.{op}", event_handler)
    return client

old_hook = set_build_s3_client_hook(hook)
```

**Arguments**

* **hook**: Build client hook.

**Returns**

Old build client hook.


# Local Catalog

You can securely and privately run the Quilt catalog in "single-player mode" on your machine. `quilt3 catalog` launches a Python webserver and local services that communicate with S3 using temporary AWS credentials, derived from your default AWS credentials (or active `AWS_PROFILE`) with `boto3.sts.get_session_token`. Data and credentials remain local and private to your machine and AWS account.

For more details about configuring and using AWS credentials in `boto3`, see the [AWS documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

### Installation

```bash
$ pip install 'quilt3[catalog]'
```

### Invocation

```bash
$ quilt3 catalog
```

See the [CLI API reference](/quilt-python-sdk/api-reference/cli#catalog) for details.


# CLI, Environment

## `catalog`

```
usage: quilt3 catalog [-h] [--detailed-help] [--host HOST] [--port PORT]
                      [--no-browser]
                      [navigation_target]

Run Quilt catalog locally

positional arguments:
  navigation_target     Which page in the local catalog to open. Leave blank
                        to go to the catalog landing page, pass in an S3 url
                        (e.g. 's3://bucket/myfile.txt') to go to file viewer,
                        or pass in a package name in the form
                        'BUCKET:USER/PKG' to go to the package viewer.

options:
  -h, --help            show this help message and exit
  --detailed-help, --detailed_help
                        Display detailed information about this command and
                        then exit
  --host HOST           Bind socket to this host
  --port PORT           Bind to a socket with this port
  --no-browser          Don't open catalog in a browser after startup
```

Run the Quilt catalog on your machine. Running `quilt3 catalog` launches a Python webserver on your local machine that serves a catalog web app and provides required backend services using temporary AWS credentials. Temporary credentials are derived from your default AWS credentials (or active `AWS_PROFILE`) using `boto3.sts.get_session_token`. For more details about configuring and using AWS credentials in `boto3`, see the AWS documentation: <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html>

#### Previewing files in S3

The Quilt catalog allows users to preview files in S3 by downloading and processing/converting them inside the Python webserver running on local machine. Neither your AWS credentials nor data requested goes through any third-party cloud services aside of S3.

## `config`

```
usage: quilt3 config [-h] [--set KEY=VALUE [KEY=VALUE ...]] [catalog_url]

Configure Quilt

positional arguments:
  catalog_url           URL of catalog to config with, or empty string to
                        reset the config

options:
  -h, --help            show this help message and exit
  --set KEY=VALUE [KEY=VALUE ...]
                        Set a number of key-value pairs for config_values(do
                        not put spaces before or after the = sign). If a value
                        contains spaces, you should define it with double
                        quotes: foo="this is a sentence". Note that values are
                        always treated as strings.
```

## `config-default-remote-registry`

```
usage: quilt3 config-default-remote-registry [-h] default_remote_registry

Configure default remote registry for Quilt

positional arguments:
  default_remote_registry
                        The default remote registry to use, e.g. s3://quilt-ml

options:
  -h, --help            show this help message and exit
```

## `disable-telemetry`

```
usage: quilt3 disable-telemetry [-h]

Disable anonymous usage metrics

options:
  -h, --help  show this help message and exit
```

## `install`

```
usage: quilt3 install [-h] [--registry REGISTRY] [--top-hash TOP_HASH]
                      [--dest DEST] [--dest-registry DEST_REGISTRY]
                      [--path PATH]
                      name

Install a package

positional arguments:
  name                  Name of package, in the USER/PKG format

options:
  -h, --help            show this help message and exit
  --registry REGISTRY   Registry where package is located, usually s3://MY-
                        BUCKET. Defaults to the default remote registry.
  --top-hash TOP_HASH   Hash of package to install. Defaults to latest.
  --dest DEST           Local path to download files to.
  --dest-registry DEST_REGISTRY
                        Registry to install package to. Defaults to local
                        registry.
  --path PATH           If specified, downloads only PATH or its children.
```

## `list-packages`

```
usage: quilt3 list-packages [-h] registry

List all packages in a registry

positional arguments:
  registry    Registry for packages, e.g. s3://quilt-example

options:
  -h, --help  show this help message and exit
```

## `login`

```
usage: quilt3 login [-h]

Log in to configured Quilt server

options:
  -h, --help  show this help message and exit
```

## `logout`

```
usage: quilt3 logout [-h]

Log out of current Quilt server

options:
  -h, --help  show this help message and exit
```

## `push`

```
usage: quilt3 push --dir DIR [-h] [--registry REGISTRY] [--dest DEST]
                   [--message MESSAGE] [--meta META] [--workflow WORKFLOW]
                   [--force] [--dedupe] [--no-copy]
                   name

Pushes the new package to the remote registry

positional arguments:
  name                 Name of package, in the USER/PKG format

required arguments:
  --dir DIR            Directory to add to the new package

optional arguments:
  -h, --help           show this help message and exit
  --registry REGISTRY  Registry where to create the new package. Defaults to
                       the default remote registry.
  --dest DEST          Where to copy the objects in the package
  --message MESSAGE    The commit message for the new package
  --meta META          Sets package-level metadata. Format: A json string with
                       keys in double quotes '{"key": "value"}'
  --workflow WORKFLOW  Workflow ID or empty string to skip workflow
                       validation. If not specified, the default workflow will
                       be used.
  --force              Skip the parent top hash check and create a new
                       revision even if your local state is behind the remote
                       registry.
  --dedupe             Skip the push if the local package hash matches the
                       remote hash.
  --no-copy            Do not copy data. Package manifest entries will
                       reference the data at the original location.
```

## `verify`

```
usage: quilt3 verify [-h] --registry REGISTRY --top-hash TOP_HASH --dir DIR
                     [--extra-files-ok]
                     name

Verify that package contents matches a given directory

positional arguments:
  name                 Name of package, in the USER/PKG format

options:
  -h, --help           show this help message and exit
  --registry REGISTRY  Registry where package is located, usually s3://MY-
                       BUCKET
  --top-hash TOP_HASH  Hash of package to verify
  --dir DIR            Directory to verify
  --extra-files-ok     Whether extra files in the directory should cause a
                       failure
```

## Environment variables

### `QUILT_DISABLE_CACHE`

Turn off cache. Defaults to `False`.

```sh
export QUILT_DISABLE_CACHE=true
```

### `QUILT_DISABLE_USAGE_METRICS`

Disable anonymous usage collection. Defaults to `False`

```sh
export QUILT_DISABLE_USAGE_METRICS=true
```

### `QUILT_MANIFEST_MAX_RECORD_SIZE`

Maximum size of a record in package manifest. **Setting this variable is strongly discouraged.** Defaults to `1_000_000`.

### `QUILT_MINIMIZE_STDOUT`

Turn off TQDM progress bars for log files. Defaults to `False`

```sh
export QUILT_MINIMIZE_STDOUT=true
```

### `QUILT_TRANSFER_MAX_CONCURRENCY`

Number of threads for file transfers. Defaults to `10`.

This variable could be tried for improving file transfer rate. The optimal value depends on network bandwidth, CPU performance, file sizes, etc.

```sh
export QUILT_TRANSFER_MAX_CONCURRENCY=20
```

### `XDG_*`

`quilt3` uses platformdirs so you can set one or more of the [XDG environment variables](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) to customize where `quilt3` writes to disk.

For example, AWS Lambda requires the user to use `/tmp/*` as the scratch directory. You can override `quilt3.util.CACHE_PATH`, so that `quilt3 install` will succeed, by setting the `XDG_CACHE_HOME` environment variable in code or in your [AWS Lambda environment variables](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html).

> You must **set XDG environment variables before** `import quilt3` in order for them to take effect.

## Constants (see [util.py](https://github.com/quiltdata/quilt/blob/master/api/python/quilt3/util.py) for more)

* `APP_AUTHOR`
* `APP_NAME`
* `BASE_DIR` - Base directory of the application
* `BASE_PATH` - Base pathlib path for the application directory
* `CACHE_PATH` - Pathlib path for the user cache directory
* `CONFIG_PATH` - Base pathlib path for the application configuration file
* `OPEN_DATA_URL` - Application data url
* `PACKAGE_NAME_FORMAT` - Regex for legal package names
* `TEMPFILE_DIR_PATH` - Base pathlib path for the application `tempfiles`


# Known Limitations

By design, Quilt is backed by Amazon S3 and scales to billions of objects and petabytes of data. The underlying limitations of S3 apply.

The Quilt catalog can browse packages and S3 buckets of any size.

## Catalog push

To ensure usability and quick package pushes, the Quilt web catalog imposes the following limits on pushes (which vary depending on whether the chunked checksums are enabled on the stack). These limits do not apply to the `quilt3` Python API.

| Dimension                                                  | Max (classic / chunked checksums) |
| ---------------------------------------------------------- | --------------------------------- |
| Package manifest size (metadata)                           | 100 MiB                           |
| Package size (data; via promotion or from an S3 directory) | 100 GiB / 5 TiB                   |
| Total size of uploaded files (soft limit)                  | 20 GB                             |
| Total size of files from S3 (soft limit)                   | 50 GB / 5 TB                      |
| Maximum file size                                          | 10 GiB / 5 TiB                    |
| Maximum number of files per push (soft limit)              | 1,000                             |
| Maximum number of files per push (hard limit)              | 5,000                             |

### Important note about S3 prefix enumeration

When creating a package from an S3 prefix (folder) using the Catalog GUI, the interface will display a warning if the prefix contains more than 1,000 files (the soft limit): "Total number of files exceeds recommended maximum of 1000." You can proceed past this warning, but the number of files the Catalog enumerates from S3 may vary, and browser performance may degrade with large file counts. The hard limit is 5,000 files per push (see table above). Additionally, S3 does not guarantee a stable ordering of listed objects, so the exact set of files included may differ between attempts.

**Workarounds for large directories:**

* Use the `quilt3` Python API's `Package.set_dir()` method, which has no such limitation
* Use the [Packaging Engine](/quilt-platform-catalog-user/packaging) to automate package creation from S3 prefixes
* Split your data into multiple packages, each under the 1,000 file limit

## API

As of this writing, with sufficient client-side memory, you can comfortable scale Quilt packages to at least one million objects per package, with no practical limit on object size (save S3's 5 TB per object limit). A fast network, or better yet an AWS compute instance in the same region as your Quilt S3 buckets, is recommended.

## Metadata

See [Metadata for teams](/metadata).


# Authentication Guide

## Overview

Quilt supports two authentication methods for accessing your data:

* **Interactive Login**: Web-based OAuth/SSO authentication (best for notebooks, local development)
* **API Keys**: Token-based authentication (best for automation, CI/CD, scripts)

## Interactive Authentication

Interactive authentication uses OAuth or SSO to authenticate through your web browser. This is the recommended method for personal use, Jupyter notebooks, and local development.

### Login

```python
import quilt3

# Opens your browser for authentication
quilt3.login()
```

This command will:

1. Open your default web browser
2. Redirect you to your Quilt catalog's login page
3. After successful authentication, save credentials locally
4. Return you to your Python session

### Check Authentication Status

```python
# Returns catalog URL if authenticated, None otherwise
catalog_url = quilt3.logged_in()

if catalog_url:
    print(f"Authenticated to: {catalog_url}")
else:
    print("Not authenticated")
```

### Logout

```python
# Clear all credentials (both interactive and API keys)
quilt3.logout()
```

## API Key Authentication

> NOTE: This feature requires Quilt Platform version 1.67 or higher

API keys provide programmatic access to Quilt without requiring browser-based authentication. Keys are created through the Python API and can be used for automated workflows.

### When to Use API Keys

#### Use API Keys For

* Automated scripts and data pipelines
* Server-side applications and microservices
* Containerized applications (Docker, Kubernetes)
* Cloud functions and serverless workloads
* CI/CD pipelines and automated workflows
* Scheduled jobs and batch processing

### Creating Your First API Key

#### Step 1: Authenticate Interactively

First, log in using the interactive method:

```python
import quilt3
quilt3.login()
```

#### Step 2: Create an API Key

```python
# Create a key that expires in 90 days (default)
key, secret = quilt3.api_keys.create("my-automation-key")

# Or specify a custom expiration (1-365 days)
key, secret = quilt3.api_keys.create("my-key", expires_in_days=180)

# IMPORTANT: Save the secret - it's only shown once!
print(f"Key ID: {key.id}")
print(f"Owner: {key.email}")
print(f"Secret: {secret}")
print(f"Fingerprint: {key.fingerprint}")
print(f"Expires: {key.expires_at}")

# Example output:
# Secret: qk_1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9s0t1u2v3w4x5y6z...
```

**⚠️ Security Warning**: The secret is only shown once during creation. Save it immediately in a secure location. If you lose it, you'll need to create a new key.

#### Step 3: Store the Secret Securely

Choose a secure storage method based on your environment:

**Local Development**

Create a `.env` file in your project directory:

```bash
# .env
QUILT_API_KEY=qk_your_secret_here
```

Load it in your Python code:

```python
from dotenv import load_dotenv
import os

load_dotenv()  # Load .env file
api_key = os.environ["QUILT_API_KEY"]
```

**IMPORTANT**: Add `.env` to your `.gitignore` to prevent committing secrets.

For production deployments, embed it in environment variables or AWS secrets manager.

#### Step 4: Use the API Key

```python
import os
import quilt3

# Load from environment
api_key = os.environ["QUILT_API_KEY"]
quilt3.login_with_api_key(api_key)

# Now use Quilt normally
pkg = quilt3.Package.browse("mypackage", "s3://mybucket")
```

NOTE: quilt3 will **not** automatically detect and use the QUILT\_API\_KEY. You must explicitly login with it.

### Managing Your API Keys

#### List All Your Keys

```python
import quilt3

# List all your keys
keys = quilt3.api_keys.list()
for key in keys:
    print(f"\n{key.name}")
    print(f"  Owner: {key.email}")
    print(f"  ID: {key.id}")
    print(f"  Fingerprint: {key.fingerprint}")
    print(f"  Status: {key.status}")
    print(f"  Created: {key.created_at}")
    print(f"  Expires: {key.expires_at}")
    print(f"  Last used: {key.last_used_at or 'Never'}")
```

#### Filter Keys

```python
# Find keys by name
production_keys = quilt3.api_keys.list(name="production-pipeline")

# Find keys by fingerprint
key = quilt3.api_keys.list(fingerprint="qk_abc...xyz")

# Find only active keys
active_keys = quilt3.api_keys.list(status="ACTIVE")

# Find expired keys
expired_keys = quilt3.api_keys.list(status="EXPIRED")
```

#### Get a Specific Key

```python
# Get details about a specific key
key = quilt3.api_keys.get("key-id-here")

if key:
    print(f"Name: {key.name}")
    print(f"Owner: {key.email}")
    print(f"Status: {key.status}")
    if key.status == "EXPIRED":
        print("This key has expired and needs to be rotated")
else:
    print("Key not found")
```

#### Revoke a Key

```python
# Revoke by ID (if you know it)
quilt3.api_keys.revoke(id="key-id-here")

# Or revoke by secret (useful for immediate emergency revocation)
quilt3.api_keys.revoke(secret="qk_your_secret")
```

## Best Practices

### Security Guidelines

* 🔐 **Never commit API keys to version control**
  * Add `.env` files to `.gitignore`
  * Use secret scanning tools (GitGuardian, GitHub Advanced Security)
* 🔐 **Use environment variables or secret managers**
  * Never hardcode keys in source code
  * Prefer managed secret services in production
* 🔐 **Rotate keys regularly**
  * Set up rotation before expiration (60-90 days)
  * Plan rotation during low-traffic periods
* 🔐 **Use descriptive names**
  * Include purpose, environment, and date: `ci-github-prod-2026q1`
  * Makes key management and auditing easier
* 🔐 **Revoke unused keys immediately**
  * Delete keys when pipelines are retired
  * Conduct regular key audits
* 🔐 **Use separate keys per environment**
  * Different keys for dev, staging, production
  * Limits blast radius if a key is compromised

## Administrator Guide

Administrators have additional capabilities to manage API keys across all users.

### Prerequisites

You must be an admin user to access these functions:

```python
import quilt3.admin
```

### List All API Keys

```python
# List all keys in the system
all_keys = quilt3.admin.api_keys.list()
print(f"Total keys: {len(all_keys)}")

# Filter by user email
user_keys = quilt3.admin.api_keys.list(email="user@example.com")
print(f"Keys for user@example.com: {len(user_keys)}")

# Filter by key name
pipeline_keys = quilt3.admin.api_keys.list(key_name="ci-pipeline")

# Filter by status
active_keys = quilt3.admin.api_keys.list(status="ACTIVE")
expired_keys = quilt3.admin.api_keys.list(status="EXPIRED")

print(f"Active keys: {len(active_keys)}")
print(f"Expired keys: {len(expired_keys)}")
```

### Get Key Details

```python
# Get details about a specific key
key = quilt3.admin.api_keys.get("key-id-here")

if key:
    print(f"Key: {key.name}")
    print(f"Owner: {key.user_email}")
    print(f"ID: {key.id}")
    print(f"Status: {key.status}")
    print(f"Created: {key.created_at}")
    print(f"Expires: {key.expires_at}")
    print(f"Last used: {key.last_used_at or 'Never'}")
```

### Revoke a User's Key

```python
# Revoke a specific key by ID
key_id = "key-id-here"
quilt3.admin.api_keys.revoke(id=key_id)
print(f"Revoked key: {key_id}")
```

### Audit Key Usage

```python
from datetime import datetime, timedelta

# Find keys that haven't been used in 90 days
threshold = datetime.now() - timedelta(days=90)
all_keys = quilt3.admin.api_keys.list(status="ACTIVE")

unused_keys = []
for key in all_keys:
    if key.last_used_at is None or key.last_used_at < threshold:
        unused_keys.append(key)
        print(f"⚠️  Unused key: {key.name}")
        print(f"   Owner: {key.user_email}")
        print(f"   ID: {key.id}")
        print(f"   Last used: {key.last_used_at or 'Never'}")

print(f"\nFound {len(unused_keys)} unused keys")
```

### Generate Usage Reports

```python
# Get summary statistics
all_keys = quilt3.admin.api_keys.list()

active = sum(1 for k in all_keys if k.status == "ACTIVE")
expired = sum(1 for k in all_keys if k.status == "EXPIRED")

print("API Key Usage Report")
print("=" * 60)
print(f"Total keys: {len(all_keys)}")
print(f"Active: {active}")
print(f"Expired: {expired}")

# Show recent keys
print("\nRecently created keys:")
sorted_keys = sorted(all_keys, key=lambda k: k.created_at, reverse=True)
for key in sorted_keys[:10]:
    print(f"  {key.name}")
    print(f"    Created: {key.created_at}")
    print(f"    Status: {key.status}")
    print(f"    Owner: {key.user_email}")
```

### Athena Query for Audit Trail

For detailed audit trail queries, see the [GxP documentation](/quilt-platform-administrator/best-practices/good-practice#querying-with-athena).

Example query to find all API key usage in the last 30 days:

```sql
SELECT
    eventtime,
    eventname,
    json_extract_scalar(useridentity, '$.email') as user_email,
    json_extract_scalar(useridentity, '$.sessionContext.auth.keyName') as key_name,
    json_extract_scalar(useridentity, '$.sessionContext.auth.keyFingerprint') as key_fingerprint,
    sourceipaddress
FROM audit_trail
WHERE date >= date_format(current_date - interval '30' day, '%Y/%m/%d')
    AND json_extract_scalar(useridentity, '$.sessionContext.auth.type') = 'api_key'
ORDER BY eventtime DESC
LIMIT 100;
```

## Troubleshooting

### Authentication Failed

**Error**: `Authentication failed. Check your credentials or API key.`

#### Solutions

**Verify the Key Format**

```python
api_key = "qk_..."  # Must start with qk_
if not api_key.startswith("qk_"):
    print("Invalid key format!")
```

**Check if Key is Expired**

```python
import quilt3
quilt3.login()  # Use interactive login first

keys = quilt3.api_keys.list()
for key in keys:
    print(f"{key.name}: {key.status}")
```

**Clear Old Sessions**

```python
import quilt3
quilt3.logout()  # Clear all credentials
quilt3.login_with_api_key(api_key)  # Try again
```

### API Key Prefix Error

**Error**: `API key must start with 'qk_' prefix`

#### Solutions

* Verify you copied the complete secret
* Check for whitespace: `api_key = api_key.strip()`
* Regenerate the key if needed

### Key Expired

**Error**: Key shows `status: EXPIRED`

#### Solution

Create a new key:

```python
import quilt3

# Check expiration
quilt3.login()
keys = quilt3.api_keys.list(name="my-key")
for key in keys:
    print(f"Status: {key.status}")
    print(f"Expires: {key.expires_at}")

# Create new key
new_key, secret = quilt3.api_keys.create("my-key-v2", expires_in_days=90)
print(f"New secret: {secret}")

# Revoke old key
quilt3.api_keys.revoke(id=keys[0].id)
```

### Environment Variable Not Set

**Error**: `QUILT_API_KEY environment variable not set`

#### Solutions

**Check if Variable is Set**

```python
import os
print(os.environ.get("QUILT_API_KEY"))  # Should show qk_...
```

**Set in Current Session**

```bash
export QUILT_API_KEY="qk_your_secret"
```

**Add to Shell Profile**

For \~/.bashrc or \~/.zshrc:

```bash
echo 'export QUILT_API_KEY="qk_..."' >> ~/.bashrc
source ~/.bashrc
```

**Use .env File (Recommended)**

```bash
echo 'QUILT_API_KEY="qk_..."' >> .env
```

Then load it in Python:

```python
from dotenv import load_dotenv
load_dotenv()
```

## Migration from Interactive to API Key

If you have existing scripts using `quilt3.login()`, here's how to migrate:

### Before (Interactive Login)

```python
import quilt3

# Requires browser - doesn't work in CI/CD
quilt3.login()

pkg = quilt3.Package.browse("data/latest", "s3://mybucket")
```

### After (API Key)

```python
import os
import quilt3

# One-time setup: Create API key
# (Run this once on your local machine)
# quilt3.login()
# key, secret = quilt3.api_keys.create("my-script")
# print(f"Secret: {secret}")

# In your script: Use the API key
api_key = os.environ.get("QUILT_API_KEY")
if not api_key:
    raise ValueError("QUILT_API_KEY environment variable required")

quilt3.login_with_api_key(api_key)

pkg = quilt3.Package.browse("data/latest", "s3://mybucket")
```

### Migration Checklist

* [ ] Create API key using interactive login
* [ ] Save secret to secure location
* [ ] Add environment variable to deployment
* [ ] Update code to use `login_with_api_key()`
* [ ] Test in development environment
* [ ] Deploy to production
* [ ] Verify API key works
* [ ] Document key location for team

## API Reference

For detailed API documentation, see:

* [quilt3.api\_keys](/quilt-python-sdk/api-reference/api#api-keys) - User API
* [quilt3.admin.api\_keys](/quilt-platform-administrator/admin-1#api-keys) - Admin API
* [Audit Trail Events](/quilt-platform-administrator/best-practices/good-practice#apikeys-namespace) - Security & Compliance

## Additional Resources

* [Installation Guide](/quilt-python-sdk/installation)
* [Quick Start](/quilt-python-sdk/quickstart)
* [GxP & Security Best Practices](/quilt-platform-administrator/best-practices/good-practice)
* [Troubleshooting](/quilt-python-sdk/more/troubleshooting)


# Custom SSL Certificates

When using the `quilt3` API or CLI with a client-to-site VPN you may need to trust a custom certificate in Python.

### Mac OS X

1. Direct your browser to an HTTPS website that uses a custom certificate.
2. Click the lock icon in the address bar.
3. Click View certificates and copy the certificate name to a safe place.
4. Open Keychain Access and select System Keychains > System Roots.
5. Click the Certificates tab.
6. Find the certificate that you noted above.
7. Click File > Export Items... to export the root certificate.
8. Convert the exported certificate in Terminal as follows:

   ```sh
   openssl x509 -inform der -in /path/to/your/certificate.cer -out /path/to/converted/certificate.crt
   ```
9. Export the following variable. You may wish to do this in a startup file for repeatability.

   ```ssh
   export REQUESTS_CA_BUNDLE=/path/to/converted/certificate.crt
   ```

### Linux

1. Export the following variable. You may wish to do this in a startup file for repeatability.

   ```sh
   export REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
   ```

> The single-file version of your CA certificate may be found in different locations depending upon your operating system.

### Windows

1. Direct your browser to an HTTPS website that uses a custom certificate.
2. Click the lock icon in the address bar.
3. Click View certificates and copy the certificate name to a safe place.
4. Open the Command Prompt («Win + R») and type `certmgr` to open your Windows Certificate Manager.
5. Find the certificate that you noted above.

   ![MITM certificate](/files/h2lzlohtC5aXhaAkZQwG)
6. Export the certificate in Base-64 encoded X.509 (.CER) to your file system (`\Path\To\mycert.cer`).
7. Convert the exported certificate in the Command Prompt as follows (assumes OpenSSL is installed):

   * Certificate in `der` encoding:

   ```sh
   openssl x509 -inform der -in \Path\To\mycert.cer -out \Path\To\Converted\mycert.crt
   ```

   * Certificate in `pem` encoding (no conversion necessary):

   ```sh
   openssl x509 -in \Path\To\mycert.cer -out \Path\To\Converted\mycert.crt
   ```
8. Export the following variable. You may wish to do this in a startup file for repeatability.

   ```sh
   set REQUESTS_CA_BUNDLE=Path\To\Converted\mycert.crt
   ```

### Verification

`quilt3` should no longer fail with SSL errors related to the custom certificate.

### References

* [SSL certificate verification with Python `Requests` library](https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification)


# Advanced


# Browsing Buckets

Quilt allows you to create, read, and write packages both on your local filesystem and on S3 buckets configured to work with Quilt3. For convenience, we provide a simple API for working with S3 buckets that serves as an alternative to [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html).

### Connecting to a bucket

To connect to an S3 `Bucket`:

```python
import quilt3
b = quilt3.Bucket("s3://quilt-example")
```

This requires that the bucket is configured to work with Quilt 3. Unless this bucket is public, you will also first need to log into the catalog that controls this bucket:

```python
# only need to run this once
# ie quilt3.config('https://your-catalog-homepage/')
quilt3.config('https://open.quiltdata.com/')

# follow the instructions to finish login
quilt3.login()
```

### Introspecting a bucket

To see the contents of a `Bucket`, use `keys`:

```python
# returns a list of objects in the bucket
b.keys()
```

### Reading from a bucket

To download a file or folder from a bucket use `fetch`:

```python
# b.fetch("path/to/directory", "path/to/local")
b.fetch("aleksey/hurdat/", "./aleksey/")
b.fetch("README.md", "./read.md")
```

```
100%|██████████| 4.07M/4.07M [00:13<00:00, 304kB/s]   
100%|██████████| 1.55k/1.55k [00:01<00:00, 972B/s]
```

### Writing to a bucket

You can write data to a bucket.

```python
# put a file to a bucket
b.put_file("read.md", "./read.md")

# or put everything in a directory at once
b.put_dir("stuff", "./aleksey")
```

Note that `set` operations on a `Package` are `put` operations on a `Bucket`.

### Deleting objects in a bucket

```python
# always be careful when deleting

# delete a fle
b.delete("read.md")

# delete a directory
b.delete_dir("stuff/")
```

### Searching in a bucket

You can search for individual objects using `search`.

Note that this feature is currently only supported for buckets backed by a Quilt catalog instance. Before performing a search you must first configure a connection to that instance using `quilt3.config`.

```python
# for example
quilt3.config(navigator_url="https://open.quiltdata.com")
```

```
<QuiltConfig at '/Users/gregezema/Library/Application Support/Quilt/config.yml' {
    "navigator_url": "https://open.quiltdata.com",
    "default_local_registry": "file:///Users/gregezema/Library/Application%20Support/Quilt/packages",
    "default_remote_registry": null,
    "default_install_location": null,
    "registryUrl": "https://open-registry.quiltdata.com",
    "telemetry_disabled": false,
    "s3Proxy": "https://open-s3-proxy.quiltdata.com",
    "apiGatewayEndpoint": "https://sttuv8u2u4.execute-api.us-east-1.amazonaws.com/prod",
    "binaryApiGatewayEndpoint": "https://ap8tbn363c.execute-api.us-east-1.amazonaws.com/prod",
    "default_registry_version": 1
}>
```

Quilt supports unstructured search:

```python
# returns all files containing the word "thor"
b.search("thor")
```

```
{'took': 16,
 'timed_out': False,
 '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},
 'hits': {'total': 10,
  'max_score': 5.5741544,
  'hits': [{'_index': 'quilt-example-reindex-v8bc2377',
    '_type': '_doc',
    '_id': 'dima/node_modules2/highlight.js/README.md:KOGAC2bPIY9o7vQ3d3ryrD04VpGPmaH2',
    '_score': 5.5741544,
    '_source': {'size': 19316,
     'comment': '',
     'version_id': 'KOGAC2bPIY9o7vQ3d3ryrD04VpGPmaH2',
     'last_modified': '2019-12-12T01:33:15+00:00',
     'updated': '2019-12-12T01:33:15.209387',
     'key': 'dima/node_modules2/highlight.js/README.md'}},
   {'_index': 'quilt-example-reindex-v8bc2377',
    '_type': '_doc',
    '_id': 'akarve/amazon-reviews/camera-reviews.parquet:yoLoCR6tdnqE141f5F4EvbFbn2J12AJt',
    '_score': 0.080087036,
    '_source': {'user_meta': {},
     'size': 100764599,
     'comment': '',
     'version_id': 'yoLoCR6tdnqE141f5F4EvbFbn2J12AJt',
     'last_modified': '2019-10-08T02:53:01+00:00',
     'updated': '2019-10-08T02:53:31.040985',
     'key': 'akarve/amazon-reviews/camera-reviews.parquet'}}]}}
```

As well as structured search on metadata (note that this feature is experimental):

```python
# returns all files annotated {'name': 'thor'}
b.search("user_meta.name:'thor'")
```

```
{'took': 0,
 'timed_out': False,
 '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},
 'hits': {'total': 0, 'max_score': None, 'hits': []}}
```


# Filtering a Package

`quilt3` provides a handful of functions for operating on the contents of a package in an entry-by-entry manner. These are helpful for performing more complicated parsing operations on the package:

```python
import quilt3

# create a package
p = (quilt3.Package()
        .set_dir("dict", "/usr/share/dict/")
        .set("words", "/usr/share/dict/words"))

# element-wise transform entries, outputting a list
# here "lk" is shorthand for "logical_key"
# and "entry" is the package entry
p.map(lambda lk, entry: entry)

# filter out entries not meeting certain criteria
p.filter(lambda lk, entry: 'cool' not in lk)
```

Notice that these functions operate over `(logical_key, entry)` tuples. Each `logical_key` is a string. Each `entry` is `PackageEntry` object, as would be returned if you slice to a leaf node of the package (e.g. `p['bar']`).


# .quiltignore

`.quiltignore` is a special file which, when included in a directory, can be used to filter out files that are included when you call `quilt3.Package.set_dir` from inside that directory:

```bash
$ ls
foo.txt         bar.txt         baz.txt
$ python -c "import quilt3; print(quilt3.Package().set_dir('/', './'))"
(local Package)
 └─foo.txt
 └─bar.txt
 └─baz.txt
$ echo foo.txt >> .quiltignore
$ ls -a
foo.txt         bar.txt         baz.txt
.quiltignore
$ python -c "import quilt3; print(quilt3.Package().set_dir('/', './'))"
(local Package)
 └─bar.txt
 └─baz.txt
 └─.quiltignore
```

This can be used to keep non-data files in the directory out of the data package. This is very useful when, for example, your data and your code live in the same directory.

The `.quiltignore` syntax is exactly the same as that of the familiar `.gitignore`. Refer to the [git documentation](https://git-scm.com/docs/gitignore) for instructions on how to use it.


# Manipulating Manifests

Every data package is backed by a **manifest**. A manifest is a self-contained reference sheet for a package, containing all of the files data and metadata necessary to work with a package.

Every time you save a data package to a registry you also save its manifest. You can inspect the manifest yourself using the `manifest` property:

```python
import quilt3

# create test file
with open("data.csv", "w") as f:
    f.write("id, value\na, 42")

p = quilt3.Package().set("data.csv", "data.csv")
list(p.manifest)
```

```
[{'version': 'v0'},
 {'logical_key': 'data.csv',
  'physical_keys': ['file:///.../data.csv'],
  'size': 15,
  'hash': None,
  'meta': {}}]
```

Manifests saved to disk are in the [jsonl](http://jsonlines.org/)/[ndjson](http://ndjson.org/) format, e.g. JSON strings separated by newlines (`\n`). They are represented as a `list` of `dict` fragments in-memory.

### Manifest specification

The first item in the manifest contains the manifest version number (`version`), the package metadata (`user_meta`), and a package commit message (`message`).

`version` is used to ensure backwards compatibility should the serialization format change. There is currently only one valid `version`: `v0`.

`user_meta` is used to store any user-defined package metadata. In this case this package has no package metadata (yet) so `user_meta` is omitted.

`message` stores the package commit message. It will be `None` if the package is pushed without a commit message. The field is omitted if the package was never subject to a `push`.

Every item after that is a manifest **entry**. Entries may be files/objects or directories. All files in the package will have a corresponding entry, as will all directories with metadata. Directories without metadata are omitted.

The manifest fields are as follows:

* `logical_key` - The path to the entry within the package.
* `physical_keys` - A list of files. Currently this field will always have a single entry. This field is omitted if the entry is a directory.
* `size` - The size of the entry in raw bytes. This field is omitted if the entry is a directory.
* `hash` - [Materialized packages](/quilt-python-sdk/advanced/materialization) record a content hash for every entry in the package. This field is used to ensure package immutability (the tophash is partly a hash of these hashes).

  If the hash is present it will be a `dict` fragment of the form `{'type': 'SHA256', 'value': '...'}`. Un-materialized package entries have a `hash` of `None`, as in our example. Directory entries omit this field.
* `meta` - Package entry metadata. Package entries lacking metadata will have a `meta` of `{}` (empty `dict`).

### Saving and loading manifests

In almost all cases you should be using registries and `push` to handle sending manifests to and fro. However, there may be advanced use cases where you want to save or load a manifest directly. For that, you can use the low-level manifest API:

```python
import quilt3
p = quilt3.Package().set("data.csv", "data.csv")
p.set_meta({"key": "value"})

# Saving manifest to jsonl file
with open("example.jsonl", "w") as f:
    p.dump(f)

# Loading manifest
p.load(open("example.jsonl"))
```

```
Loading manifest: 100%|██████████| 2/2 [00:00<00:00, 2141.59entries/s]





(local Package)
 └─data.csv
```


# Materialization

### Materialization

`push` uploads the entries in a data package to a remote registry, performs the tophash calculations, and stores a package manifest referencing the remote files in the remote registry.

This is known as **materialization**, and it means that `push` creates **materialized packages**: packages which point solely to files located in an Amazon S3 bucket.

### Pushing unmaterialized packages

There are advanced use cases where automatically copying (potentially large) files is not the behavior you want.

To push an *unmaterialized* file to a remote registry, use `build` with a `registry`. For example:

```python
import quilt3
p = quilt3.Package().set("example.csv", "example.csv")
p.build("username/packagename", registry="s3://my-bucket")
```

Note that in this case it is up to you, the package author, to ensure that any local files in the package remain available and accessible to users.


# S3 Select

> NOTE: As of July 25, 2024, Amazon S3 Select is [no longer available](https://docs.aws.amazon.com/AmazonS3/latest/userguide/WhatsNew.html) to new customers. The following documentation only applies to accounts that were created before that date.

[S3 Select](https://aws.amazon.com/blogs/aws/s3-glacier-select/) is an S3 feature that allows you to operate on JSON, CSV, and Parquet files in a row-based manner using SQL syntax. QUILT features experimental support for S3 Select queries as part of the `Bucket` interface:

```python
import quilt3
b = quilt3.Bucket("s3://alpha-quilt-example")
b.select("foo/bar.csv", "SELECT * FROM S3Object LIMIT 5")
# <pandas DataFrame object at ...>
```


# More


# Changelog

## unreleased - YYYY-MM-DD

### Python API

* \[Removed] Drop support for Python 3.9 (end-of-life); `quilt3` now requires Python >= 3.10 ([#4941](https://github.com/quiltdata/quilt/pull/4941))
* \[Fixed] `quilt3.admin.buckets.list` no longer raises `TypeError` when its type hints are introspected on Python 3.14 ([#4940](https://github.com/quiltdata/quilt/pull/4940))

## 7.3.0 - 2026-04-07

### Python API

* \[Added] Admin role and policy management APIs: `quilt3.admin.roles` (list/get/get-default/create/update/patch/delete/set-default) and `quilt3.admin.policies` (list/get/create/update/patch/delete); all lookup and mutation methods accept name/title in addition to ID ([#4805](https://github.com/quiltdata/quilt/pull/4805))
* \[Added] `quilt3.admin.ManagedRole` now exposes `policies` and `permissions` fields ([#4805](https://github.com/quiltdata/quilt/pull/4805))
* \[Added] `quilt3.admin` now exports `Policy`, `PolicySummary`, `Permission` (with `.read()` / `.read_write()` shorthand constructors), `BucketPermissionLevel` types and `RoleAssignedError`, `RoleSsoConfigConflictError`, `RoleTypeMismatchError` exceptions ([#4805](https://github.com/quiltdata/quilt/pull/4805))

## 7.2.0 - 2026-01-28

### Python API

* \[Added] Verification of packages with crc64nvme hashes ([#4696](https://github.com/quiltdata/quilt/pull/4696))
* \[Added] API key support (`quilt3.api_keys`, `quilt3.admin.api_keys`, `login_with_api_key()`) ([#4690](https://github.com/quiltdata/quilt/pull/4690), [#4705](https://github.com/quiltdata/quilt/pull/4705))

## 7.1.0 - 2025-12-22

### Python API

* \[Added] `quilt3.admin.buckets` sub-module for managing bucket configuration ([#4667](https://github.com/quiltdata/quilt/pull/4667))

## 7.0.0 - 2025-08-25

### Python API

* \[Changed] **BREAKING:** Change the default `selector_fn` for `Package.push`. Now, files in the same bucket as the destination registry are not copied by default. This change was requested by users who were inadvertently creating extra copies of files in packages that were created from files in S3. ([#4464](https://github.com/quiltdata/quilt/pull/4464))

## 6.3.1 - 2025-03-10

### Python API

* \[Changed] **BREAKING:** Forbid using non-finite `float`s in package metadata. Previously that was allowed and resulted in package manifests that are not compliant to RFC 8259 ([#4340](https://github.com/quiltdata/quilt/pull/4340))

## 6.3.0 - 2025-01-24

### Python API

* \[Added] `quilt3.hooks`: `set_build_s3_client_hook()` function for customizing S3 client ([#4289](https://github.com/quiltdata/quilt/pull/4289))

## 6.2.0 - 2025-01-14

### CLI

* \[Fixed] Fix installation of `quilt3[catalog]` ([#4287](https://github.com/quiltdata/quilt/pull/4287))

### Python API

* \[Added] `quilt3.admin.tabulator`: `get_open_query()` and `set_open_query()` functions for managing Tabulator open query mode ([#4260](https://github.com/quiltdata/quilt/pull/4260))

## 6.1.1 - 2024-11-21

### Python API

* \[Changed] More informative error messages when package construction fails ([#4216](https://github.com/quiltdata/quilt/pull/4216))
* \[Fixed] Allow S3 paths starting with `/` in `Package.set_dir()` and `Package.set()` ([#4207](https://github.com/quiltdata/quilt/pull/4207))

## 6.1.0 - 2024-10-14

### Python API

* \[Added] `quilt3.admin.tabulator` sub-module for managing Tabulator configuration ([#4136](https://github.com/quiltdata/quilt/pull/4136))
* \[Added] `quilt3.get_boto3_session()` function for creation of Boto3 sessions with Quilt credentials ([#4169](https://github.com/quiltdata/quilt/pull/4169))

## 6.0.0 - 2024-08-19

### Python API

* \[Added] New `quilt3.admin.sso_config` sub-module for management of SSO configuration ([#4065](https://github.com/quiltdata/quilt/pull/4065), [#4098](https://github.com/quiltdata/quilt/pull/4098))

### Catalog, Lambdas

* \[Fixed] **SECURITY**: Remove `polyfill.io` references ([#4038](https://github.com/quiltdata/quilt/pull/4038))
* \[Fixed] Don't show negative number as search result count, in particular don't show `-1` when `secure_search: True` ([#4101](https://github.com/quiltdata/quilt/pull/4101))
* \[Changed] Renamed "Admin settings" to "Admin" ([#4045](https://github.com/quiltdata/quilt/pull/4045))
* \[Changed] Hide "Add bucket" button for non-admin users from main landing page ([#4106](https://github.com/quiltdata/quilt/pull/4106))
* \[Added] Admin: Support SSO permissions mapping (SSO config editor, disable role assignment for SSO-mapped users) ([#4070](https://github.com/quiltdata/quilt/pull/4070), [#4097](https://github.com/quiltdata/quilt/pull/4097), [#4099](https://github.com/quiltdata/quilt/pull/4099))

## 6.0.0a5 - 2024-06-25

### Python API

* \[Fixed] Fix PhysicalKey to URI conversion in Python 3.12 on Windows ([#4027](https://github.com/quiltdata/quilt/pull/4027))

### Catalog, Lambdas

* \[Removed] Drop MARKETING mode support, delete associated dead code ([#4009](https://github.com/quiltdata/quilt/pull/4009))
* \[Removed] Delete Google Tag Manager script integration ([#4039](https://github.com/quiltdata/quilt/pull/4039))
* \[Added] Support multiple roles per user ([#3982](https://github.com/quiltdata/quilt/pull/3982))
* \[Added] Add `ui.actions = False` and `ui.actions.writeFile` for configuring visibility of buttons ([#4001](https://github.com/quiltdata/quilt/pull/4001))
* \[Added] Support creating folders and rearranging entries with drag and drop in package creation dialog ([#3999](https://github.com/quiltdata/quilt/pull/3999))
* \[Added] Qurator AI Assistant for summarizing file contents using Bedrock API ([#3989](https://github.com/quiltdata/quilt/pull/3989))

## 6.0.0a4 - 2024-06-18

### Python API

* \[Added] New `quilt3.admin` API with more features (requires 1.53+ stack) ([#3990](https://github.com/quiltdata/quilt/pull/3990))
* \[Removed] `quilt3.admin` API ([#3990](https://github.com/quiltdata/quilt/pull/3990))
* \[Removed] Drop Python 3.8 support ([#3993](https://github.com/quiltdata/quilt/pull/3993))
* \[Fixed] If upload optimization during `push()` succeeds the checksum is calculated from local file instead of remote file ([#3968](https://github.com/quiltdata/quilt/pull/3968))
* \[Changed] Upload optimization check now tries to use S3 SHA-256 checksum and falls back to ETag ([#3968](https://github.com/quiltdata/quilt/pull/3968))

### Catalog, Lambdas

* \[Changed] Use promises for URLs in IGV to have fresh signing each time they used ([#3979](https://github.com/quiltdata/quilt/pull/3979))

## 6.0.0a3 - 2024-04-25

### Python API

* \[Added] `quilt3.search()` and `quilt3.Bucket.search()` now accepts custom Elasticsearch queries ([#3448](https://github.com/quiltdata/quilt/pull/3448))
* \[Fixed] `quilt3.search()` and `quilt3.Bucket.search()` now work with 2022+ Quilt stacks ([#3448](https://github.com/quiltdata/quilt/pull/3448))

### Catalog, Lambdas

* \[Added] Added "text" as a file type for quilt\_summarize.json ([#3946](https://github.com/quiltdata/quilt/pull/3946))
* \[Added] Sign URL in undocumented `compressedIndexURL` IGV property ([#3947](https://github.com/quiltdata/quilt/pull/3947))
* \[Fixed] Robust handling of PFS cookies ([#3962](https://github.com/quiltdata/quilt/pull/3962))
* \[Changed] Pre-select first catalog and database for Athena ([#3949](https://github.com/quiltdata/quilt/pull/3949))
* \[Changed] Move pagination to the bottom ([#3950](https://github.com/quiltdata/quilt/pull/3950))
* \[Changed] Search UI QoL improvements ([#3960](https://github.com/quiltdata/quilt/pull/3960), [#3967](https://github.com/quiltdata/quilt/pull/3967))

## 6.0.0a2 - 2024-04-15

### Python API

* \[Added] New 'unversioned' parameter to `Package.set_dir()` and `Package.set()` for use with S3 URIs, such as HealthOmics, that do not support `ListBucketVersions` and/or `GetObjectVersion` ([#3927](https://github.com/quiltdata/quilt/pull/3927))

## 6.0.0a1 - 2024-02-26

### Python API

* \[Removed] Drop Python 3.7 support ([#3841](https://github.com/quiltdata/quilt/pull/3841))
* \[Changed] Set S3 client `max_pool_connections` to `QUILT_TRANSFER_MAX_CONCURRENCY` ([#3867](https://github.com/quiltdata/quilt/pull/3867))
* \[Changed] **BREAKING:** Switch from a regular SHA256 checksum to a hash list (`sha2-256-chunked`) to match S3's built-in checksums ([#2782](https://github.com/quiltdata/quilt/pull/2782))
* \[Changed] **BREAKING:** Delay object hashing until package push to take advantage of S3's hashing; as a result, `dest` functions no longer receive a `top_hash` ([#2782](https://github.com/quiltdata/quilt/pull/2782))

### Catalog, Lambdas

* \[Added] Support chunked checksums ([#3403](https://github.com/quiltdata/quilt/pull/3403), [#3887](https://github.com/quiltdata/quilt/pull/3887))
* \[Added] Search: Help link to ElasticSearch docs ([#3861](https://github.com/quiltdata/quilt/pull/3861))
* \[Added] Support MP PAYGO: track subscription state, handle related errors ([#3906](https://github.com/quiltdata/quilt/pull/3906))
* \[Fixed] Faceted Search: show helpful message in case of search query syntax errors ([#3821](https://github.com/quiltdata/quilt/pull/3821))
* \[Fixed] JsonEditor: fix changing collections items, that have `.additionalProperties` or `.items` JSON Schema ([#3860](https://github.com/quiltdata/quilt/pull/3860))
* \[Fixed] Restore Catalog name / Database for Athena query execution ([#3902](https://github.com/quiltdata/quilt/pull/3902))
* \[Changed] Faceted Search: use non-linear scale for numeric range control ([#3805](https://github.com/quiltdata/quilt/pull/3805))
* \[Changed] Faceted Search: reliably find metadata facets ([#3809](https://github.com/quiltdata/quilt/pull/3809))
* \[Changed] Athena: add docs link for empty state, remove "Queries" tab for guests ([#3885](https://github.com/quiltdata/quilt/pull/3885))
* \[Changed] Updated supported node/npm version to v20 and v10 ([#3873](https://github.com/quiltdata/quilt/pull/3873))

## 5.4.0 - 2023-11-29

### Python API

* \[Added] `create_user()`, `delete_user()`, `set_role()` in `quilt3.admin` ([#3764](https://github.com/quiltdata/quilt/pull/3764))

### Catalog, Lambdas

* \[Fixed] Fixed file preview header layout ([#3454](https://github.com/quiltdata/quilt/pull/3454))
* \[Fixed] Fix getting custom styles and options for files listed in quilt\_summarize.json ([#3485](https://github.com/quiltdata/quilt/pull/3485))
* \[Fixed] Fix Header's orange flash on load ([#3487](https://github.com/quiltdata/quilt/pull/3487))
* \[Fixed] Fix code sample for package push ([#3499](https://github.com/quiltdata/quilt/pull/3499))
* \[Fixed] Make bookmarks optional (and fix Embed listings broken in #3697) ([#3705](https://github.com/quiltdata/quilt/pull/3705))
* \[Fixed] Disable opening file picker on metadata click, and fix dropping JSON as metadata ([#3707](https://github.com/quiltdata/quilt/pull/3707))
* \[Fixed] Faceted Search: crash due to infinite recursion on duplicate facets ([#3799](https://github.com/quiltdata/quilt/pull/3799))
* \[Fixed] Hide filters in a sidebar drawer on mobile ([#3801](https://github.com/quiltdata/quilt/pull/3801))
* \[Fixed] Fix copying selected text in code samples ([#3803](https://github.com/quiltdata/quilt/pull/3803))
* \[Fixed] Add current bucket as a successor if it's missed from config ([#3811](https://github.com/quiltdata/quilt/pull/3811))
* \[Added] Add filter for users and buckets tables in Admin dashboards ([#3480](https://github.com/quiltdata/quilt/pull/3480))
* \[Added] Add links to documentation and re-use code samples ([#3496](https://github.com/quiltdata/quilt/pull/3496))
* \[Added] Show S3 Object tags ([#3515](https://github.com/quiltdata/quilt/pull/3515))
* \[Added] Indexer lambda now indexes S3 Object tags ([#3691](https://github.com/quiltdata/quilt/pull/3691))
* \[Added] Add filters to Roles and Permissions in Admin dashboards ([#3690](https://github.com/quiltdata/quilt/pull/3690))
* \[Added] Add download and bookmarks button to file listings ([#3697](https://github.com/quiltdata/quilt/pull/3697))
* \[Changed] Enable user selection in perspective grids ([#3453](https://github.com/quiltdata/quilt/pull/3453))
* \[Changed] Hide columns without values in files listings ([#3512](https://github.com/quiltdata/quilt/pull/3512))
* \[Changed] Enable `allow-same-origin` for iframes in browsable buckets ([#3516](https://github.com/quiltdata/quilt/pull/3516))
* \[Changed] Allow users select files and directories and keep selection whenever they navigate to multiple directories or use filter ([#3527](https://github.com/quiltdata/quilt/pull/3527))
* \[Changed] Unify per-bucket and global search ([#3613](https://github.com/quiltdata/quilt/pull/3613))
* \[Changed] Allow use of `<br />` in Markdown ([#3720](https://github.com/quiltdata/quilt/pull/3720))
* \[Changed] Faceted search ([#3712](https://github.com/quiltdata/quilt/pull/3712))
* \[Changed] Specify condition for rendering Quilt manifests, allowing to render other types of files in `.quilt/packages/` ([#3816](https://github.com/quiltdata/quilt/pull/3816))

## 5.3.1 - 2023-05-02

### Python API

* \[Fixed] `Package.verify()` now raises exception if unsupported hash type is encountered ([#3401](https://github.com/quiltdata/quilt/pull/3401))

### Catalog, Lambdas

* \[Fixed] Fix file URLs in embed ([#3419](https://github.com/quiltdata/quilt/pull/3419))
* \[Changed] Increased available file size to render Markdown preview to 3MiB ([#3427](https://github.com/quiltdata/quilt/pull/3427))

## 5.3.0 - 2023-04-11

### Python API

* \[Added] Support [AnnData](https://anndata.readthedocs.io/en/latest/) format ([#2974](https://github.com/quiltdata/quilt/pull/2974))
* \[Added] `--no-copy` parameter to `Package.push()` ([#3398](https://github.com/quiltdata/quilt/pull/3398))

### Catalog, Lambdas

* \[Fixed] Fix package push failing for unrelated reason ([#3390](https://github.com/quiltdata/quilt/pull/3390))
* \[Fixed] Fix package page flash when there is Jupyter Notebook ([#3408](https://github.com/quiltdata/quilt/pull/3408))
* \[Added] Add `gallery` field for configuring galleries visibility ([#3421](https://github.com/quiltdata/quilt/pull/3421))

## 5.2.1 - 2023-04-05

### Python API

* \[Fixed] Fixed CSV serialization with pandas 2 ([#3395](https://github.com/quiltdata/quilt/pull/3395))

## 5.2.0 - 2023-03-27

### Python API

* \[Added] Validation of package entries metadata ([#3286](https://github.com/quiltdata/quilt/pull/3286))

### Catalog, Lambdas

* \[Added] Add basic support for tasklist in Markdown ([#3339](https://github.com/quiltdata/quilt/pull/3339))
* \[Added] Object-level validation, frontend ([#3336](https://github.com/quiltdata/quilt/pull/3336))
* \[Added] Frontend for permissive HTML rendering ([#3198](https://github.com/quiltdata/quilt/pull/3198))
* \[Added] Confirmation to enable Package Files Server ([#3388](https://github.com/quiltdata/quilt/pull/3388))
* \[Fixed] Fixed mobile layout for collaborators badges ([#3307](https://github.com/quiltdata/quilt/pull/3307))
* \[Fixed] Fixed metadata handling for entries without hash or size in pkgpush lambda ([#3314](https://github.com/quiltdata/quilt/pull/3314))
* \[Fixed] Fixed adding metadata for S3 entries ([#3367](https://github.com/quiltdata/quilt/pull/3367))
* \[Fixed] Fixed crash of the iframe in Bucket tab ([3387](https://github.com/quiltdata/quilt/pull/3387))
* \[Changed] Edit .quilt/config files with text editor ([#3306](https://github.com/quiltdata/quilt/pull/3306))
* \[Changed] Refactoring of buttons adapted to page width ([#3300](https://github.com/quiltdata/quilt/pull/3300))
* \[Changed] Restrict editing `user_meta` field only for object-level metadata ([#3337](https://github.com/quiltdata/quilt/pull/3337))
* \[Changed] Tabular format defaults to .csv ([#3382](https://github.com/quiltdata/quilt/pull/3382))

## 5.1.1 - 2023-01-25

### Python API

* \[Fixed] Reduce backtracking during `pip install quilt3[catalog]` ([#3292](https://github.com/quiltdata/quilt/pull/3292))

### Catalog, Lambdas

* \[Added] Add 'ECharts' and 'Text' file type switcher, significantly refactor this switcher ([#3240](https://github.com/quiltdata/quilt/pull/3240))
* \[Added] Add link to file from Athena results ([#3242](https://github.com/quiltdata/quilt/pull/3242))
* \[Added] Add link to package revisions from package list ([#3256](https://github.com/quiltdata/quilt/pull/3256))
* \[Added] WebP support in thumbnail lambda ([#3275](https://github.com/quiltdata/quilt/pull/3275))
* \[Added] Set default search mode in Admin Settings ([#3270](https://github.com/quiltdata/quilt/pull/3270))
* \[Added] Toggle buttons for file previews ([#3290](https://github.com/quiltdata/quilt/pull/3290))
* \[Fixed] Fix performance issue (missing memoization) in search results ([#3257](https://github.com/quiltdata/quilt/pull/3257))
* \[Fixed] Fix fetching and writing settings in Admin/Settings section ([#3276](https://github.com/quiltdata/quilt/pull/3276))
* \[Fixed] Fix iframe preview width ([#3279](https://github.com/quiltdata/quilt/pull/3279))
* \[Changed] Make file preview wrapper consistently 100% width ([#3245](https://github.com/quiltdata/quilt/pull/3245))
* \[Changed] Show collapsed values in JSON viewer ([#3249](https://github.com/quiltdata/quilt/pull/3249))
* \[Changed] Rename "Metadata" to "User metadata" ([#3255](https://github.com/quiltdata/quilt/pull/3255))
* \[Changed] Show selective metadata for packages as JSON ([#3259](https://github.com/quiltdata/quilt/pull/3259))
* \[Changed] Show selective metadata on one line and optionally on multiple lines ([#3284](https://github.com/quiltdata/quilt/pull/3284))

## 5.1.0 - 2022-12-09

### Python API

* \[Added] `dedupe` parameter for `Package.push()` ([#3181](https://github.com/quiltdata/quilt/pull/3181))
* \[Fixed] Fix check to determine if a file is a tempfile in Windows with Python 3.8+ ([#2900](https://github.com/quiltdata/quilt/pull/2900))
* \[Fixed] Speed up import and get rid of undeclared runtime dependency on `setuptools` ([#2994](https://github.com/quiltdata/quilt/pull/2994))
* \[Changed] Disable upload optimization for objects with SSE-KMS ([#2790](https://github.com/quiltdata/quilt/pull/2790))
* \[Changed] Use `platformdirs` instead of unmaintained `appdirs` ([#3140](https://github.com/quiltdata/quilt/pull/3140))

### CLI

* \[Added] `--dedupe` flag for `quilt3 push` ([#3181](https://github.com/quiltdata/quilt/pull/3181))

### Catalog, Lambdas

* \[Added] Add IGV renderer ([#2965](https://github.com/quiltdata/quilt/pull/2965))
* \[Added] Use `quilt_summarize.json` to control Perspective menu ([#2744](https://github.com/quiltdata/quilt/pull/2744))
* \[Added] Show bucket collaborators ([#2756](https://github.com/quiltdata/quilt/pull/2756))
* \[Added] Add `config` field to Perspective scope of `quilt_summarize.json` ([#2781](https://github.com/quiltdata/quilt/pull/2781))
* \[Added] Add `blocks` field to `.quilt/catalog/config.yaml` to control blocks visibility ([#2791](https://github.com/quiltdata/quilt/pull/2791))
* \[Added] Add theming with custom square logo and background color ([#2793](https://github.com/quiltdata/quilt/pull/2793))
* \[Added] Admin: new roles and policies UI ([#2819](https://github.com/quiltdata/quilt/pull/2819))
* \[Added] Deep search indexing for .pptx ([#2881](https://github.com/quiltdata/quilt/pull/2881))
* \[Added] Stack Status Admin UI ([#2935](https://github.com/quiltdata/quilt/pull/2935))
* \[Added] Render package manifests using Perspective ([#2971](https://github.com/quiltdata/quilt/pull/2971))
* \[Added] Athena default workflow config setting ([#2985](https://github.com/quiltdata/quilt/pull/2985))
* \[Added] Add missing README to package ([#2960](https://github.com/quiltdata/quilt/pull/2960), [#2979](https://github.com/quiltdata/quilt/pull/2979))
* \[Added] View and copy full Athena query by expanding table row ([2993](https://github.com/quiltdata/quilt/pull/2993))
* \[Added] Create packages from Athena query results ([#3004](https://github.com/quiltdata/quilt/pull/3004))
* \[Added] Add "Create text file" menu ([#3017](https://github.com/quiltdata/quilt/pull/3017))
* \[Added] Redirect to last selected Athena workgroup ([#3067](https://github.com/quiltdata/quilt/pull/3067))
* \[Added] `status_reports` lambda ([#2989](https://github.com/quiltdata/quilt/pull/2989), [#3088](https://github.com/quiltdata/quilt/pull/3088))
* \[Added] Stack Status Admin UI: reports ([#3068](https://github.com/quiltdata/quilt/pull/3068))
* \[Added] Edit button for text files in packages ([#3070](https://github.com/quiltdata/quilt/pull/3070))
* \[Added] Add execution context for Athena query execution ([#3062](https://github.com/quiltdata/quilt/pull/3062))
* \[Added] Add confirmation if now every row is valid for creating package from Athena results ([#3073](https://github.com/quiltdata/quilt/pull/3073))
* \[Added] Create file menu item for package ([#3127](https://github.com/quiltdata/quilt/pull/3127))
* \[Added] Expose Quilt Catalog automation API as `window.QuiltCatalog` ([#3141](https://github.com/quiltdata/quilt/pull/3143))
* \[Added] Add form to create workflow from quilt config editor ([#3158](https://github.com/quiltdata/quilt/pull/3158))
* \[Added] Add extended value for `ui.blocks.meta` in `.quilt/catalog/config.yaml` ([#3175](https://github.com/quiltdata/quilt/pull/3175))
* \[Fixed] Fix package creation in S3 buckets with SSE-KMS enabled ([#2754](https://github.com/quiltdata/quilt/pull/2754))
* \[Fixed] Fix creation of packages with large (4+ GiB) files ([#2933](https://github.com/quiltdata/quilt/pull/2933))
* \[Fixed] Fix pre-population of default dates when using "dateformat" + {"format": "date"} ([#3082](https://github.com/quiltdata/quilt/pull/3082))
* \[Fixed] Fix editing nested files in packages, fix editing files added from the different location to package ([#3117](https://github.com/quiltdata/quilt/pull/3117))
* \[Fixed] Fix enum detection in `anyOf`, `allOf`, `oneOf`, `not` fields and in arrays in JsonEditor ([#3169](https://github.com/quiltdata/quilt/pull/3169))
* \[Fixed] Fix adding new elements in JsonEditor ([#3169](https://github.com/quiltdata/quilt/pull/3169))
* \[Fixed] Fix enum free form fields ([#3185](https://github.com/quiltdata/quilt/pull/3185))
* \[Fixed] User role and admin status caching ([#3200](https://github.com/quiltdata/quilt/pull/3200))
* \[Changed] Clean up home page ([#2780](https://github.com/quiltdata/quilt/pull/2780)).
* \[Changed] Make `pkgpush` lambda directly invocable, adjust handling of parameters and errors ([#2776](https://github.com/quiltdata/quilt/pull/2776))
* \[Changed] Push packages via GraphQL ([#2768](https://github.com/quiltdata/quilt/pull/2768))
* \[Changed] Improve rendering performance for multi-slide .pptx ([#2878](https://github.com/quiltdata/quilt/pull/2878))
* \[Changed] Rework package indexing: now package indexes have documents only for current versions of package pointer objects, documents for 'latest' pointers have `package_hash`, `package_stats`, `comment`, `metadata` fields properly populated ([#2897](https://github.com/quiltdata/quilt/pull/2897))
* \[Changed] Remove ClientRequestToken (idempotency token) for making Athena queries ([#2992](https://github.com/quiltdata/quilt/pull/2992))
* \[Changed] Fixed config and docs mistyping: `ui.athena.defaultWorkflow` should be `ui.athena.defaultWorkgroup` ([#3067](https://github.com/quiltdata/quilt/pull/3067))
* \[Changed] Use dedicated columns field instead of first row, fix duplicated first row in table results ([#3101](https://github.com/quiltdata/quilt/pull/3101))
* \[Changed] Allow pushing empty packages, suggest creating a stub `README.md` file when trying to push an empty package ([#3114](https://github.com/quiltdata/quilt/pull/3114))
* \[Changed] Allow to save only latest revisions of files ([#3124](https://github.com/quiltdata/quilt/pull/3124))
* \[Changed] Render HTML files in LOCAL mode ([#3139](https://github.com/quiltdata/quilt/pull/3139))
* \[Changed] Support dots in bucket names while using S3 proxy ([#3147](https://github.com/quiltdata/quilt/pull/3147))
* \[Changed] Support `additionalProperties` and `items` in JsonEditor ([#3144](https://github.com/quiltdata/quilt/pull/3144))
* \[Changed] Initialize Catalog configuration synchronously from `QUILT_CATALOG_CONFIG` global var ([#3166](https://github.com/quiltdata/quilt/pull/3166))
* \[Changed] Handle rendering multiple molecules in one .sdf file ([#3179](https://github.com/quiltdata/quilt/pull/3179))
* \[Changed] Refactor Sentry set-up, add more integrations / instrumentation ([#3164](https://github.com/quiltdata/quilt/pull/3164))

## 5.0.0 - 2022-03-14

### CLI

* \[Added] `--force` flag for `quilt3 push` ([#2722](https://github.com/quiltdata/quilt/pull/2722))

### Catalog, Lambdas

* \[Fixed] Respect prefix filtering when packaging a folder ([#2706](https://github.com/quiltdata/quilt/pull/2706))
* \[Fixed] Fix package creation failing for some regions ([#2718](https://github.com/quiltdata/quilt/pull/2718))
* \[Added] NGL renderer for .pdb files ([#2711](https://github.com/quiltdata/quilt/pull/2711))
* \[Added] Admin: default role management ([#2721](https://github.com/quiltdata/quilt/pull/2721))
* \[Added] Preview CZI images ([#2727](https://github.com/quiltdata/quilt/pull/2727))

## 4.1.0 - 2022-02-22

### Python API

* \[Added] Automatically decompress gzip'ed package entries when deserializing ([#2677](https://github.com/quiltdata/quilt/pull/2677))
* \[Added] Semi-atomic push ([#2689](https://github.com/quiltdata/quilt/pull/2689))

## 4.0.0 - 2022-01-31

### Python API

* \[Added] Declared compatibility with `jsonschema==4.*`.
* \[Added] `--host` and `--port` parameters for `quilt3 catalog`.
* \[Added] `--no-browser` parameter for `quilt3 catalog`.
* \[Changed] `quilt3 catalog` now requires `quilt3` to be installed with `catalog` extra dependency (`pip install 'quilt3[catalog]'`).
* \[Changed] Dependencies on `flask` and `dnspython` are dropped.
* \[Removed] Deprecated passing subpackage as part of package name for `Package.install()`, use `path` parameter instead.
* \[Removed] Deprecated calling of `Package.resolve_hash()` without specifying `name` parameter.
* \[Removed] Deprecated `PackageEntry.physical_keys`, use `PackageEntry.physical_key` instead.

### CLI

* \[Removed] Deprecated passing subpackage as part of package name for `quilt3 install`, use `--path` parameter instead.

### Catalog, Lambdas

* \[Changed] pkgselect: make directly invocable, always use execution role ([#2560](https://github.com/quiltdata/quilt/pull/2560))
* \[Changed] Move package listing / querying to GraphQL ([#2552](https://github.com/quiltdata/quilt/pull/2552))
* \[Changed] Fix performance of `countPages` in thumbnail lambda ([#2616](https://github.com/quiltdata/quilt/pull/2616)).
* \[Changed] `pkgpush` lambda now uses dedicated lambda for hashing files to push larger packages faster from catalog.
* \[Changed] Local-mode-specific adjustments to landing page and catalog UI ([#2611](https://github.com/quiltdata/quilt/pull/2611))
* \[Changed] PDF preview: count pages again ([#2621](https://github.com/quiltdata/quilt/pull/2621))
* \[Changed] Make Vega download data files from S3 via proxy to avoid CORS issues ([#2631](https://github.com/quiltdata/quilt/pull/2631))
* \[Changed] Don't support packages in out-of-stack-buckets ([#2641](https://github.com/quiltdata/quilt/pull/2641))
* \[Changed] Use Perspective library for tabular data files (.csv, .tsv, .xls, .xlsx, .parquet, .jsonl) ([#2576](https://github.com/quiltdata/quilt/pull/2576), [#2691](https://github.com/quiltdata/quilt/pull/2691))
* \[Fixed] Improve upload performance and stability, fix some hashing-related errors ([#2532](https://github.com/quiltdata/quilt/pull/2532))
* \[Added] Echarts renderer ([#2382](https://github.com/quiltdata/quilt/pull/2382))
* \[Added] Set height for `quilt_summarize.json` files ([#2474](https://github.com/quiltdata/quilt/pull/2474))
* \[Added] Add a "transcode" lambda for previewing video files ([#2366](https://github.com/quiltdata/quilt/pull/2366/))
* \[Added] Add object-level metadata editor and move package metadata editor to popup ([#2510](https://github.com/quiltdata/quilt/pull/2510/))
* \[Added] Video previews ([#2540](https://github.com/quiltdata/quilt/pull/2540))
* \[Added] Audio previews ([#2547](https://github.com/quiltdata/quilt/pull/2547))
* \[Added] Powerpoint (`.pptx`) preview ([#2598](https://github.com/quiltdata/quilt/pull/2598), [#2626](https://github.com/quiltdata/quilt/pull/2626))

## 3.6.0 - 2021-10-15

### Python API

* \[Added] Validation of package names with `handle_pattern` in [workflows](/workflows) config.
* \[Added] Validation of package entries with `entries_schema` in [workflows](/workflows) config.

## 3.5.0 - 2021-09-07

### Python API

* \[Added] Size of each manifest record is now limited by 1 MB. This constraint is added to ensure that S3 select, Athena and downstream services work correctly. This limit can be overridden with `QUILT_MANIFEST_MAX_RECORD_SIZE` environment variable. ([#2114](https://github.com/quiltdata/quilt/pull/2114))
* \[Changed] Decrease size of `Package` in-memory representation ([#1943](https://github.com/quiltdata/quilt/pull/1943))

### CLI

* \[Added] `--workflow` argument to `push` command ([#2279](https://github.com/quiltdata/quilt/pull/2279))

### Catalog, Lambdas

* \[Added] Pre-populate today date for metadata ([#2121](https://github.com/quiltdata/quilt/pull/2121))
* \[Added] Limit and offset parameters in pkgselect lambda ([#2124](https://github.com/quiltdata/quilt/pull/2124))
* \[Added] File listing: "load more" button to fetch more entries from S3 ([#2150](https://github.com/quiltdata/quilt/pull/2150))
* \[Added] Voila Notebooks format support ([#2163](https://github.com/quiltdata/quilt/pull/2163))
* \[Added] Ability to add files from S3 while revising a package ([#2171](https://github.com/quiltdata/quilt/pull/2171))
* \[Added] Lambdas for pushing an existing package/creation of package ([#2147](https://github.com/quiltdata/quilt/pull/2147), [#2180](https://github.com/quiltdata/quilt/pull/2180))
* \[Added] Custom navbar link configurable via admin UI ([#2192](https://github.com/quiltdata/quilt/pull/2192))
* \[Added] Adding S3 objects to packages from configurable source buckets while revising ([#2193](https://github.com/quiltdata/quilt/pull/2193))
* \[Added] Add Athena SQL queries initial viewer and runner ([#2197](https://github.com/quiltdata/quilt/pull/2197))
* \[Added] Managing per-bucket permissions via admin UI ([#2228](https://github.com/quiltdata/quilt/pull/2228))
* \[Added] Deep search indexing for Excel ([#2247](https://github.com/quiltdata/quilt/pull/2247))
* \[Added] Deep search indexing for PDF ([#2256](https://github.com/quiltdata/quilt/pull/2256))
* \[Added] Add Vega/JSON view switcher for Vega files ([#2236](https://github.com/quiltdata/quilt/pull/2236))
* \[Added] Subpackage download ([#2258](https://github.com/quiltdata/quilt/pull/2258))
* \[Added] Per-bucket deep indexing settings ([#2290](https://github.com/quiltdata/quilt/pull/2290))
* \[Added] Embed: IPC, more debug features, docs ([#2314](https://github.com/quiltdata/quilt/pull/2314))
* \[Added] Download button on Summary page ([#2367](https://github.com/quiltdata/quilt/pull/2367))
* \[Added] Validation of package names and entries, pattern for default package name ([2364](https://github.com/quiltdata/quilt/pull/2364))
* \[Changed] New DataGrid-based file listing UI with arbitrary sorting and filtering ([#2097](https://github.com/quiltdata/quilt/pull/2097))
* \[Changed] Item selection in folder-to-package dialog ([#2122](https://github.com/quiltdata/quilt/pull/2122))
* \[Changed] Don't preview .tif (but keep .tiff), preview .results as plain text ([#2128](https://github.com/quiltdata/quilt/pull/2128))
* \[Changed] Sort packages by modification time by default ([#2126](https://github.com/quiltdata/quilt/pull/2126))
* \[Changed] Resolve logical keys in summaries and vega inside packages ([#2140](https://github.com/quiltdata/quilt/pull/2140))
* \[Changed] Embed: load polyfills, bring back prefix filtering, load more ([#2153](https://github.com/quiltdata/quilt/pull/2153))
* \[Changed] Scan more bytes (first 128 KiB) when trying to detect if a JSON file is a Vega visualization ([#2229](https://github.com/quiltdata/quilt/pull/2229))
* \[Changed] Use GraphQL for fetching and editing buckets ([#2240](https://github.com/quiltdata/quilt/pull/2240))
* \[Changed] Use registry for search requests ([#2242](https://github.com/quiltdata/quilt/pull/2242))
* \[Changed] Enhance `quilt_summarize.json` format, support title, description and multi-column layout ([#2245](https://github.com/quiltdata/quilt/pull/2245))
* \[Changed] PDF preview: don't count pages ([#2276](https://github.com/quiltdata/quilt/pull/2276))
* \[Changed] Default bucket icon changed from Quilt logo to more neutral ([#2287](https://github.com/quiltdata/quilt/pull/2287))
* \[Changed] Cachebust revision list request ([#2298](https://github.com/quiltdata/quilt/pull/2298))
* \[Changed] Wrap wide Vega charts with horizontal scroll ([#2304](https://github.com/quiltdata/quilt/pull/2304))
* \[Changed] Unify package creation and update dialogs (support adding S3 files in both) ([#2302](https://github.com/quiltdata/quilt/pull/2302))
* \[Changed] Warmer chart colors ([#2329](https://github.com/quiltdata/quilt/pull/2329), [#2338](https://github.com/quiltdata/quilt/pull/2338))
* \[Changed] Remove custom button for adding Readme, and re-use dialog for creating files. Fix creating README in package ([#3173](https://github.com/quiltdata/quilt/pull/3173))
* \[Fixed] `UnicodeDecodeError` in indexer and pkgselect lambdas ([#2123](https://github.com/quiltdata/quilt/pull/2123))
* \[Fixed] Catch and display package-related errors properly ("no such package" and "bad revision") ([#2160](https://github.com/quiltdata/quilt/pull/2160))
* \[Fixed] Crashing `pkgselect` lambda's folder view on an empty manifest ([#2147](https://github.com/quiltdata/quilt/pull/2147))
* \[Fixed] Infinite spinner on logout ([#2232](https://github.com/quiltdata/quilt/pull/2232))
* \[Fixed] Dismiss error page when navigating from it ([#2291](https://github.com/quiltdata/quilt/pull/2291))
* \[Fixed] Avoid crash on non-existent logical keys in pkgselect detail view ([#2307](https://github.com/quiltdata/quilt/pull/2307)
* \[Fixed] Error while rendering a preview inside a package ([#2328](https://github.com/quiltdata/quilt/pull/2328))
* \[FIxed] Bring back missing username at Admin/Users table ([#2339](https://github.com/quiltdata/quilt/pull/2339))

## 3.4.0 - 2021-03-15

### Python API

* \[Added] `QUILT_TRANSFER_MAX_CONCURRENCY` environment variable ([#2092](https://github.com/quiltdata/quilt/pull/2092))
* \[Added] `QUILT_DISABLE_CACHE` environment variable ([#2091](https://github.com/quiltdata/quilt/pull/2091))
* \[Added] Support for callable `dest` parameter in `Package.push()` ([#2095](https://github.com/quiltdata/quilt/pull/2095))
* \[Changed] Removed unused dependency on `packaging` ([#2090](https://github.com/quiltdata/quilt/pull/2090))
* \[Fixed] Possible downloading of truncated manifests ([#1977](https://github.com/quiltdata/quilt/pull/1977))
* \[Fixed] `TypeError` on import when running with `PYTHONOPTIMIZE=2` ([#2102](https://github.com/quiltdata/quilt/pull/2102))

### Catalog, Lambdas

* \[Added] Support for EventBridge S3 events to es/indexer ([#1987](https://github.com/quiltdata/quilt/pull/1987))
* \[Added] Generate and resolve Quilt package URIs ([#1935](https://github.com/quiltdata/quilt/pull/1935))
* \[Added] Buttons for copying canonical package URIs ([#1990](https://github.com/quiltdata/quilt/pull/1990))
* \[Added] Additional validation for package name ([#1998](https://github.com/quiltdata/quilt/pull/1998))
* \[Added] Populate package name with username prefix ([#2016](https://github.com/quiltdata/quilt/pull/2016))
* \[Added] Link from bucket overview page to bucket settings ([#2022](https://github.com/quiltdata/quilt/pull/2022))
* \[Added] Folder to package dialog ([#2040](https://github.com/quiltdata/quilt/pull/2040))
* \[Added] Search lambda: `freeform` action API and UI ([#2087](https://github.com/quiltdata/quilt/pull/2087), [#2088](https://github.com/quiltdata/quilt/pull/2088))
* \[Added] Spreadsheets Drag'n'Drop to Metadata ([#2094](https://github.com/quiltdata/quilt/pull/2094))
* \[Changed] Tree view for files in package update dialog ([#1989](https://github.com/quiltdata/quilt/pull/1989))
* \[Changed] Lambda indexing retry logic to not fail content extraction ([#2007](https://github.com/quiltdata/quilt/pull/2007))
* \[Changed] Number of retries per ES and S3 failure in indexing Lambda ([#1987](https://github.com/quiltdata/quilt/pull/1987))
* \[Changed] Handle delete markers in ES ([#1997](https://github.com/quiltdata/quilt/pull/1997), [#2000](https://github.com/quiltdata/quilt/pull/2000), [#2003](https://github.com/quiltdata/quilt/pull/2003), [#2017](https://github.com/quiltdata/quilt/pull/2017), [#2023](https://github.com/quiltdata/quilt/pull/2023))
* \[Changed] Two-column layout for package dialogs ([#2001](https://github.com/quiltdata/quilt/pull/2001))
* \[Changed] Show Schema validation errors in text mode ([#2010](https://github.com/quiltdata/quilt/pull/2010))
* \[Changed] Toolchain: use webpack\@5 ([#2036](https://github.com/quiltdata/quilt/pull/2036)) and TypeScript ([#2043](https://github.com/quiltdata/quilt/pull/2043)), ditch babel
* \[Changed] Use polyfill.io instead of bundled polyfills ([#2043](https://github.com/quiltdata/quilt/pull/2043))
* \[Changed] Render JSON with custom tree-like viewer ([#2037](https://github.com/quiltdata/quilt/pull/2037))
* \[Changed] Set JSON Schema defaults ([#2053](https://github.com/quiltdata/quilt/pull/2053))
* \[Changed] Don't upload unmodified files while revising a package ([#2080](https://github.com/quiltdata/quilt/pull/2080))
* \[Fixed] Bug that caused search to miss delete object and delete package events ([#1987](https://github.com/quiltdata/quilt/pull/1987))
* \[Fixed] lambda previews for time series `AICSImage` data (potential `IndexError` if odd number of time points) ([#1945](https://github.com/quiltdata/quilt/pull/1945))
* \[Fixed] Handle folders in search results ([#1992](https://github.com/quiltdata/quilt/pull/1992), [#1994](https://github.com/quiltdata/quilt/pull/1994))
* \[Fixed] Use EncodingType=url for S3 list requests to handle special chars in keys / prefixes ([#2026](https://github.com/quiltdata/quilt/pull/2026))
* \[Fixed] Empty response from `pkgselect` folder view for packages with all non-string logical\_keys or physical\_keys ([#1947](https://github.com/quiltdata/quilt/pull/1947))
* \[Fixed] "Download Directory" bug that caused 502 or failed downloads when any files contained the "+" character ([#2067](https://github.com/quiltdata/quilt/pull/2067/))
* \[Added] Ability to log in with Microsoft Azure Active Directory via OIDC ([#2089](https://github.com/quiltdata/quilt/pull/2089))

## 3.3.0 - 2020-12-08

### Python API

* \[Added] Metadata quality API ([#1855](https://github.com/quiltdata/quilt/pull/1874)). For details see this [section](/workflows).
* \[Changed] Improved formatting of package load progress bar ([#1897](https://github.com/quiltdata/quilt/pull/1897))
* \[Fixed] Crash during load of package manifest with unicode symbols with non-unicode locale set ([#1931](https://github.com/quiltdata/quilt/pull/1931))

### Catalog, Lambdas

* \[Added] Ad hoc package updates ([#1856](https://github.com/quiltdata/quilt/pull/1856))
* \[Added] Copy packages from one bucket to another ([#1932](https://github.com/quiltdata/quilt/pull/1932))
* \[Added] Enhanced JSON editor and schema validation for package metadata ([#1867](https://github.com/quiltdata/quilt/pull/1867))
* \[Added] Preview .pdbqt files as plain text ([#1855](https://github.com/quiltdata/quilt/pull/1855))
* \[Added] Retry logic for failed queries, minimize load on ES for sample, images overviews ([#1864](https://github.com/quiltdata/quilt/pull/1864/))
* \[Added] Buttons to download packages and directories as .zip archives ([#1868](https://github.com/quiltdata/quilt/pull/1868/))
* \[Added] Search help dropdown for the index landing page ([#1838](https://github.com/quiltdata/quilt/pull/1838))
* \[Changed] Get package revisions from ElasticSearch, not S3 ([#1851](https://github.com/quiltdata/quilt/pull/1851))
* \[Changed] Render vega specs smaller than 20 MiB right away, render larger ones after pressing a button ([#1873](https://github.com/quiltdata/quilt/pull/1873))
* \[Changed] Prefix filtering for directory view similar to AWS Console ([#1876](https://github.com/quiltdata/quilt/pull/1876), [#1880](https://github.com/quiltdata/quilt/pull/1880))
* \[Changed] Preview `*notes` files as plain text ([#1896](https://github.com/quiltdata/quilt/pull/1896))
* \[Changed] Default search operator to "AND" (was "OR") for more precise searches ([#1924](https://github.com/quiltdata/quilt/pull/1924))
* \[Changed] `top_hash`-based package routes (timestamp routes are still supported in the same way) ([#1938](https://github.com/quiltdata/quilt/pull/1938))
* \[Fixed] Incomplete package stats for empty packages in es/indexer Lambda ([#1869](https://github.com/quiltdata/quilt/pull/1869))
* \[Fixed] Slow parquet preview rendering (and probably other occurrences of JsonDisplay) ([#1878](https://github.com/quiltdata/quilt/pull/1878))

## 3.2.1 - 2020-10-14

### Python API

* \[Performance] 2X to 5X faster multi-threaded hashing of S3 objects ([#1816](https://github.com/quiltdata/quilt/issues/1816), [#1788](https://github.com/quiltdata/quilt/issues/1788))
* \[Fixed] Bump minimum required version of tqdm. Fixes a crash (`UnseekableStreamError`) during upload retry. ([#1853](https://github.com/quiltdata/quilt/issues/1853))

### CLI

* \[Added] `--meta` argument to `push` ([#1793](https://github.com/quiltdata/quilt/issues/1793))
* \[Fixed] Crash in `list-packages` ([#1852](https://github.com/quiltdata/quilt/issues/1852))

### Catalog, Lambdas

* \[Added] Ability to preview larger Jupyter notebooks; warning when cells are elided ([#1823](https://github.com/quiltdata/quilt/issues/1823), [#1822](https://github.com/quiltdata/quilt/issues/1822))
* \[Added] Object size to package browsing experience in catalog ([#1744](https://github.com/quiltdata/quilt/issues/1744))
* \[Added] Total number of packages to catalog Overview tab ([#1808](https://github.com/quiltdata/quilt/issues/1808))
* \[Added] PDF and other file formats, including .gz ones, to catalog Overview tab
* \[Added] Drag-n-drop package creation ([#1786](https://github.com/quiltdata/quilt/pull/1786))
* \[Added] Glacier support ([#1794](https://github.com/quiltdata/quilt/pull/1794), [#1796](https://github.com/quiltdata/quilt/pull/1796))
* \[Added] Show package metadata ([#1806](https://github.com/quiltdata/quilt/pull/1806))
* \[Added] Search facets help ([#1828](https://github.com/quiltdata/quilt/pull/1828/))
* \[Added] Admin/buckets: re-index and repair ([#1824](https://github.com/quiltdata/quilt/pull/1824))
* \[Changed] Case-insensitive package filtering ([#1807](https://github.com/quiltdata/quilt/pull/1807))
* \[Changed] Show PDFs in bucket overviews ([#1811](https://github.com/quiltdata/quilt/pull/1811))
* \[Changed] Admin/buckets: adjust SNS ARN input, make it possible to not subscribe to a topic ([#1824](https://github.com/quiltdata/quilt/pull/1824))
* \[Changed] Show "push package" button when there's no packages in a bucket ([#1843](https://github.com/quiltdata/quilt/pull/1843))
* \[Fixed] Bugs involving bad or missing package stats during S3 Select calls ([#1829](https://github.com/quiltdata/quilt/issues/1829))
* \[Fixed] Overly aggressive 40X retry logic in es/indexer ([#1804](https://github.com/quiltdata/quilt/issues/1804))
* \[Fixed] Semantic bugs in ElasticSearch timeouts (queries now time out properly) ([#1803](https://github.com/quiltdata/quilt/issues/1803))
* \[Fixed] Missing Helvetica issues for PDFs ([#1792](https://github.com/quiltdata/quilt/issues/1792))
* \[Fixed] Bulletproof file downloads via HTTP header override ([#1787](https://github.com/quiltdata/quilt/pull/1787))
* \[Fixed] Previews not rendering in global search ([#1787](https://github.com/quiltdata/quilt/pull/1787))

## 3.2.0 - 2020-09-08 - Package Registry Refactor

### Python

* Refactors local and s3 storage-layer code around a new PackageRegistry base class (to support improved file layouts in future releases)
* Multi-threaded download for large files, 2X to 5X performance gains when installing packages with large files, especially on larger EC2 instances
* Package name added to Package.resolve\_hash
* Bugfix: remove package revision by shorthash
* Performance improvements for build and push

### Catalog & Lambdas

* PDF previews
* Browse full package contents (no longer limited to 1000 files)
* Indexing and search package-level metadata
* Fixed issue with download button for certain text files
* FCS files: content indexing and preview
* Catalog sign-in with email (or username)
* Catalog support for sign-in with Okta

## 3.1.14 - 2020-06-13 - Python API features, fixes, catalog capabilities, backend optimizations

### Catalog

* .cef preview
* allow hiding download button
* only show stats for 2-level extensions for .gz files

### Python

* `quilt3.logged_in()`
* fix retries during hashing
* improve progress bars
* fix `quilt3 catalog`
* expanded documentation
* reduce `pyyaml` requirements to prevent version conflicts

### Backend

* improve unit test coverage for indexing lambdas
* fix real-time delete handling (incl. for unversioned objects)
* handle all s3:ObjectCreated: and ObjectRemoved: events (fixes ES search state and bucket Overview)

## 3.1.13 - 2020-04-15 - Windows Support

### Python API

* Official support for Windows
* Add support for Python 3.7, 3.8
* Fix Package import in Python
* Updated libraries for stability and security
* Quiet TQDM for log files ($ export QUILT\_MINIMIZE\_STDOUT=true )
* CLI setting of config parameters

### Catalog

* new feature to filter large S3 directories with regex
* more reliable bucket region inference
* Support preview of larger Jupyter notebooks in S3 (via transparent GZIP)
* JS (catalog) dependencies for stability and security
* extended Parquet file support (for files without a .parquet extension)
* Improvements to catalog signing logic for external and in-stack buckets

Special thanks to @NathanDeMaria (CLI and Windows support) and @JacksonMaxfield for contributing code to this release.

## 3.1.12 - 2020-03-11 - Command line push

Python

* Add `push` to CLI

## 3.1.11 - 2020-03-10 - New command line features, bug fixes

Catalog

* Updated JS dependencies
* Display package truncation warning in Packages

Python

* `quilt3 install foo/bar/subdirectory`
* Bug fixes for CopyObject and other exceptions

## 3.1.10 - 2020-01-29

### Python Client

* Fix bug introduced in 3.1.9 where uploads fail due to incorrect error checking after a HEAD request to see if an object already exists (#1512)

## 3.1.9 - 2020-01-29

### Python Client

* `quilt3 install` now displays the tophash of the installed package (#1461)
* Added `quilt3 --version` (#1495)
* Added `quilt3 disable-telemetry` CLI command (#1496)
* CLI command to launch catalog directly to file viewer - `quilt3 catalog $S3_URL` (#1470, #1487)
* No longer run local container for `quilt3 catalog` (#1504). See (#1468, #1483, #1482) for various bugs leading to this decision.
* Add PhysicalKey class to abstract away local files vs unversioned s3 object vs versioned s3 object (#1456, #1473, #1478)
* Changed cache directory location (#1466)
* More informative progress bars (#1506)
* Improve support for downloading from public buckets (#1503)
* Always disable telemetry during tests (#1494)
* Bug fix: prevent misleading CLI argument abbreviations (#1481) such as `--to` referring to `--tophash`
* Bug fix: background upload/download threads are now killed if the main thread is interrupted (#1486)
* Performance improvements: load JSONL manifest faster (#1480)
* Performance improvement: If there is an error when copying files, fail quickly (#1488)

### Catalog

* Better package listing UX (#1462)
* Improve bucket stats visualization when there are many categories (#1469)

## 3.1.8 - 2019-12-20 - Catalog Command Fixes and Performance Improvements

### Python API

* Bug-fixes for `quilt3.config` and `quilt3.catalog`
* Performance improvements for Packages

### Catalog

* Updated landing page

## 3.1.7 - 2019-12-13 - Package Cache

### Catalog

* New `LOCAL` mode for running the catalog on localhost

### Python API

* `quilt3 catalog` command to run the Quilt catalog on your local machine
* `quilt3 verify` compares the state of a directory to the contents of a package version
* Added a local file cache for installed packages
* Performance improvements for upload and download
* Support for short hashes to identify package versions
* Adding telemetry for API calls

## 3.1.6 - 2019-12-03 - Package rollback

### API Improvements

* Implement Package.rollback
* Drop support for object metadata (outside of packages)
* Change the number of threads used when installing and pushing from 4 to 10 (S3 default)
* Misc bug fixes

## 3.1.5 - 2019-11-20 - Catalog and API improvements

### Catalog

* Fix package listing for packages with more 100 revisions
* Add stacked area charts for downloads
* 2-level file-extensions for bucket summary

### Python

* Fix uploads of very large files
* Remove unnecessary copying during push

## 3.1.4 - 2019-10-17

* [`delete_package`](https://docs.quilt.bio/api-reference/api#delete_package) for a specific version via `top_hash=`

## 3.1.3 - 2019-10-11

* Bug fix: when adding python objects to a package a temporary file would be created and then deleted when the object was pushed, leading to a crash if you tried to push that package again (PR #1264)

## 3.1.2 - 2019-10-11

* Added support for adding an in-memory object (such as a `pandas.DataFrame`) to a package via `package.set()`
* Fix to work with pyarrow 0.15.0
* Performance improvements for list\_packages and delete\_package
* Added `list_package_versions` function

## 3.0.0 - 2019-05-24 - Quilt 3 (formerly Quilt T4) Initial Release

This is the initial release of the new and improved Quilt 3 ([formerly Quilt T4](https://github.com/quiltdata/t4)). For more information [refer to the documentation](https://docs.quilt.bio/).

## 2.9.15 - 2019-01-09 - Teams Config

### Compiler

Adds a feature to allow `quilt config` to set a registry URL for a private Teams registry.

## 2.9.14 - 2018-12-20 - Push Package by Hash

### Compiler

* Adding a hash argument to `quilt.push` to allow pushing any package version to a registry.

### Registry

* Make object sizes required.
* Update urllib3 version for security patch

### Docs

* Improved instructions for running registries.

## 2.9.13 - 2018-11-12 - Fix ascii decoding bug

* Fix an ascii decoding issue related to ellipses …

## 2.9.12 - 2018-10-11 - Pyarrow 0.11 compatibility

### Make Quilt work with pyarrow 0.11

* Update Parquet reading code to match the API change in pyarrow 0.11.
* Fix downloading of zero-byte files

## 2.9.11 - 2018-09-11 - Save Objects to Existing Packages

### Compiler

* New helper function `quilt.save` adds an object (e.g., a Pandas DataFrame) to an existing package by performing a sub-package build and push in a single step
* BugFix: `quilt.load` now correctly returns sub-packages (fixes issue #741)

### Registry

* Send a welcome email to new users after activation

## 2.9.10 - 2018-08-08 - Minor updates and improved documentation

### Compiler

* fixes an issue with packages created on older versions of pyarrow
* improves readability for `quilt inspect`
* allow adding a node with metadata using sub-package build/push

### Registry

* adds documentation for running a private registry in AWS

## 2.9.9 - 2018-07-31 - Bug fixes

* Suppress numpy warnings under Python 2.7
* Fix subpackage build and push

## 2.9.8 - 2018-07-30 - Flask-internal Authentication

### Compiler

* Added support for sub-package build and push to allow updates to allow adding nodes to large packages without materializing the whole package
* First-class support for `ndarray`

### Registry

* Replaced dependence on external OAuth2 provider with a built-in authentication and session management
* Registry support for sub-package push

### Catalog

* Updated to support new registry authentication

## 2.9.7 - 2018-07-11 - Asa extensions

### Compiler

* added Bracket accessor for GroupNodes
* asa.plot to show images in packages
* asa.torch to convert packages to PyTorch Datasets
* Enforce fragment store as read-only

### Catalog

* Added source maps and CI for catalog testing

## 2.9.6 - 2018-06-13 - Documentation and Bugfixes

### Documentation

Expands and improves documentation for working with Quilt packages.

### Bug fixes and small improvements

* Load packages by hash
* Choose a custom loader for DataNodes with asa=

### Registry

* Specify Ubuntu version in Dockerfiles

## 2.9.5 - 2018-05-23 - Package Filtering

### Catalog

* display package traffic stats in catalog

### Compiler

* filter packages based on per-node metadata
* get/set metadata for package nodes
* support custom loaders in the \_data method

### Registry

* package commenting

## 2.9.4 - 2018-04-20 - Metadata only package install

### Compiler

* Metadata-only package install
* Build DataFrames from existing Parquet files
* Remove HDF5 dependencies
* Code cleanup and refactoring

### Registry

* Option for metadata-only package installs
* New endpoint for fetching missing fragments (e.g., from partially installed packages)
* Improved full-text search

## 2.9.3 - 2018-03-20 - Package Composition

### Compiler

* Allow building packages out of other packages and elements from other packages. A new build-file keyword, `package` inserts a package (or sub-package) as an element in the package being built.

### Catalog

* Upgrade router and other dependencies
* Display packages by author

## 2.9.2 - 2018-03-01 - Quilt Teams

### Catalog Changes to support private registries

* Amin UI for controlling users and access
* Auditing views

### Globbing for package builds

* Allow specifying sets of input files in build.yml

### Command-line support for private registries

* Specify teams packages
* Admin commands to create and activate/deactivate users

## 2.9.1 - 2018-02-06 - Better Progress Bar

Version 2.9.1 introduces a better progress bar for installing (downloading) Quilt packages. Quilt push now sends objects' uncompressed size to the registry. The progress bar is now based on the total bytes downloaded instead of the number of files.

## 2.9.0 - 2018-02-02 - Shared Local Package Storage

### Shared Local Package Storage

Import packages from shared local directories to save storage overhead and network traffic when sharing packages on the same local network.

### Registry Install Stats

Log package installs in the registry to display stats on package use.

### New Python API commands

* generate
* rm
* search

### Drop support for Python 3.4

### (BETA) Team Registries

Updates to commands and local storage to allow users to connect to different registries to support teams running private registries for internal sharing.

## 2.8.4 - 2018-01-24 - Fix download retry

Fixes a bug in download that prevented retrying failed downloads.

## 2.8.3 - 2018-01-19 - Remove Unneeded Pandas dependency

\#186 introduced an undeclared dependency on Pandas >= 0.21.0 (by catching ParserError during CSV parsing). This release removes that dependency and resolves #291.

## 2.8.2 - 2018-01-17 - Hotfix for several quilt commands

PR <https://github.com/quiltdata/quilt/pull/290>

## 2.8.1 - 2018-01-10 - Add Quilt Catalog

### Quilt Catalog

Source for the Quilt data catalog is now included in this repository.

### MySQL->Postgres

Ported the Quilt registry from MySQL to Postgres

### Docker Compose

Improvements to the docker configuration that allows running the registry, catalog, database and authentication service from Docker compose.

### Parallel Download

Data fragments can now be downloaded in parallel leading to much faster package installs for large packages.

## 2.8.0 - 2017-12-07 - Centralized Local Package Store

## Release Highlights

### Quilt packages live in a centralized location on your machine

Quilt data packages are now available wherever you run Python. We recommend that users **quilt push all local packages to the registry before upgrading**. Further details on migration are [here](https://docs.quilt.bio/troubleshooting.html).

### Faster builds with build cache

Quilt now caches build intermediates. So if you wish to update the README of a multi-gigabyte package, you can rebuild the entire package in one second.

### Group-level build parameters

You can now specify build parameters (like transform) for all children of a group in one shot. The updated syntax and docs are [here](https://docs.quilt.bio/buildyml.html).

### quilt.yml is like requirements.txt but for data

You can now express dependencies on multiple packages in a single file. Docs [here](https://docs.quilt.bio/cli.html#installing-via-requirements-file).

### Experimental: build a package from a GitHub repo

Quilt build now accepts GitHub URLs. If you use data stored on GitHub you can turn it into a Quilt package with quilt build.

## 2.7.1 - 2017-11-09 - Checks: unit tests for data packages

Version 2.7.1 includes several minor bug fixes and one new feature, checks. Checks allow a user to specify data integrity checks that are enforced during quilt build.

## 2.7.0 - 2017-08-18 - Subpackages and more efficient uploads/downloads

* Support installing subpackages as `quilt install usr/pkg/path`
* Upload fragments in parallel
* Use http sessions when accessing S3

## 2.6.3 - 2017-07-22 - Clear session to prevent quilt.login() bugs in Jupyter

## 2.6.1 - 2017-07-20 - Package Delete

This release adds a new command to delete a package including all versions and history from the registry.

## 2.6.0 - 2017-07-14 - Fast Builds for Large Packages

Building a package from a directory of input files now skips generating a build file. That speeds up the build process and makes it easier to change the package contents and rebuild.

## 2.5.1 - 2017-07-06 - Push Public Packages

This release includes support for paid plans on quiltdata.com and is recommended for all individual and business-plan users. It adds a shortcut to push packages and make them public in a single command and improves documentation.


# Contributing

Quilt is an open source project, and we welcome contributions from the community.

Contributors must adhere to the [Code of Conduct](https://github.com/quiltdata/quilt/blob/master/docs/CODE_OF_CONDUCT.md).

If you are looking for a quick map of where major components live, start with the repository [`README.md`](https://github.com/quiltdata/quilt/blob/master/README.md).

### Reporting issues

Unsure about something? To get support, check out our [Slack community](https://slack.quilt.bio).

Found a bug? File it in our [GitHub issues](https://github.com/quiltdata/quilt/issues).

### Cloning

To work on `quilt` you will first need to clone the repository.

```bash
git clone https://github.com/quiltdata/quilt
```

You can then set up your own branch version of the code, and work on your changes for a pull request from there.

```bash
cd quilt
git checkout -B new-branch-name
```

### Local package development

Choose your area first:

* Python SDK and CLI work: `api/python`
* Web catalog work: `catalog`
* AWS services and background processing: `lambdas`
* Platform and user docs: `docs`

#### Python Environment

We use [`uv`](https://github.com/astral-sh/uv) for dependency management. First, install `uv`:

```bash
# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or with pip
pip install uv
```

`uv` ensures the environment is properly set up before executing a command, so you don't have to do anything else.

Run `uv run poe` to see all configured tasks (or refer to `pyproject.toml`).

#### Python Testing

All new code contributions are expected to have complete unit test coverage, and to pass all preexisting tests.

Use `pytest` to test your changes during normal development:

```bash
cd api/python
# Run all tests
uv run poe test

# Run tests verbosely
uv run poe test-verbose

# Run with coverage
uv run poe test-cov

# Run specific test file directly
uv run poe test tests/test_util.py
```

### Local catalog development

Note that, at the current time, it is only possible to run a local catalog if you already have a catalog deployed to AWS, because the catalog relies on certain services (namely, AWS Lambda and the AWS Elasticsearch Service) which cannot be run locally.

#### Catalog Environment

Use `npm` to install the catalog dependencies locally:

```bash
cd catalog
npm install
```

#### Build

To build a static code bundle, as would be necessary in order to serve the catalog:

```bash
npm run build
```

To run the catalog locally, create a development config:

```bash
cp config.js.example static-dev/config.js
# edit static-dev/config.js for your environment
```

To run the catalog in developer mode:

```bash
npm start
```

This uses `webpack` under the hood to compile code changes on the fly and provide live reloading, useful when developing.

Make sure that any images you check into the repository are [optimized](https://kinsta.com/blog/optimize-images-for-web/) at check-in time.

#### Catalog Testing

To run the catalog unit tests:

```bash
npm run test
```

### Creating a release

1. Once you are ready to cut a new release, update the version in `api/python/pyproject.toml` ([`uv version`](https://docs.astral.sh/uv/guides/package/#updating-your-version) can help with this) and in `docs/CHANGELOG.md`.
2. Create PR with these changes.
3. Once PR is merged, create a tag from commit with merge: `git tag $VERSION $COMMIT_HASH`.
4. Once you push the tag to GitHub with `git push origin $VERSION` a new CI build that makes PyPI release is triggered.

### Updating documentation

Documentation is served via GitBook, and is based on the `docs/` folder in the `master` branch of the `quilt` repository.

Documentation changes go live at pull request merge time. There is currently no way to preview documentation updates except locally.

#### Updating the API Reference

The API Reference section of the documentation is served by processing the docstrings in the codebase using a script. We use [our own fork](https://github.com/quiltdata/pydoc-markdown/tree/quilt) of the `pydoc-markdown` package to do the necessary work.

To modify the API Reference, modify the docstring associated with a method of interest.

Then, run `uv run poe gendocs` from the `api/python` directory.

The resulting files will land in `docs/` and will be ready to be checked in.

#### Updating everything else

All other pages in the documentation are served from corresponding Markdown pages in the `docs` directory. To edit the page, edit the Markdown file. Then check that file in.

### License

Quilt is open source under the [Apache License, Version 2.0](https://github.com/quiltdata/quilt/blob/master/LICENSE/README.md).


# Frequently Asked Questions

### How do I sync my notebook and all of its data and models to S3 as a package?

```python
p = quilt3.Package()
p.set_dir(".", ".")
p.push("USR/PKG", message="MSG", registry="s3://BUCKET")
```

> Use a [.quiltignore file](https://docs.quilt.bio/advanced-usage/.quiltignore) for more control over which files `set_dir()` includes.

### How does Quilt versioning relate to S3 object versioning?

Quilt packages are one level of abstraction above S3 object versions. Object versions track mutations to a single file, whereas a quilt package references a *collection* files and assigns this collection a unique version.

It is strongly recommended that you enable object versioning on the S3 buckets that you push Quilt packages to. Object versioning ensures that mutations to every object are tracked, and provides some protection against deletion.

### Where are the Quilt 2 packages?

Visit [legacy.quiltdata.com](https://legacy.quiltdata.com/) and use [`quilt`](https://pypi.org/project/quilt/) on PyPI.

### Does `quilt3` collect anonymous usage statistics?

Yes, to find bugs and prioritize features.

You can disable anonymous usage collection with an environment variable:

```bash
export QUILT_DISABLE_USAGE_METRICS=true
```

Or call `quilt3.disable_telemetry()` to persistently disable anonymous usage statistics.

### Can I turn off TQDM progress bars for log files?

Yes:

```bash
export QUILT_MINIMIZE_STDOUT=true
```

### Which version of Quilt are you on?

#### Python client

```bash
quilt3 --version
```

#### CloudFormation application

1. Go to CloudFormation > Stacks > YourQuiltStack > Outputs
2. Copy the row labeled TemplateBuildMetadata
3. "git\_revision" is your template version

This information is also available in the footer of the main page of the Catalog.

### Hashing during `push` takes a long time. Can I speed it up?

Yes. Follow these steps:

1. Run your compute in the same region as your S3 bucket (as opposed to a local machine or foreign region)—I/O is much faster.
2. Use a larger instance with more vCPUs.
3. Increase [`QUILT_TRANSFER_MAX_CONCURRENCY`](/quilt-python-sdk/api-reference/cli#quilt_transfer_max_concurrency) above its default to match your available vCPUs.
4. If you are using Quilt Catalog 1.51 (released Feb 2024), you can enable the `ChunkedChecksums` CloudFormation parameter so it will calculate the checksums in parallel, or reuse them if already existing in S3. Parallel checksums are also available by default in `quilt3` v6 or later (pre-released Feb 2024).

### Does Quilt work with R?

In the scientific computing community, the [R Project](https://www.r-project.org/) is commonly used as an alternative, or companion, to Python. It is a language and environment for statistical computing and graphics, and is available as Free Software under the [GNU General Public License](https://www.r-project.org/COPYING).

Currently there are no plans to release a Quilt package for distribution through the [CRAN package repository](https://cloud.r-project.org/). However, you can still use Quilt with R, using either:

1. The Command Line Interface (CLI) API
2. [Reticulate](https://rstudio.github.io/reticulate/)

#### Using the Quilt CLI API with R

You can script the Quilt CLI directly from your shell environment and chain it with your R scripts to create a unified workflow:

```bash
quilt3 install my-package # download Quilt data package
[Run R commands or scripts] # modify the data in Quilt data package using R
quilt3 push --dir path/to/remote-registry my-package
# upload Quilt data package to the remote registry
```

#### Using Quilt with Reticulate

The [Reticulate](https://rstudio.github.io/reticulate/) package provides a set of tools for interoperability between Python and R by embedding a Python session within your R session.

### How do I delete a data package and all of the objects in the data package?

You may have a test data package that you wish to delete at some point to ensure your data repository is clean and organized. *Please do this very carefully!* In favor of immutability, Quilt makes deletion a bit tricky. First, note that `quilt3.Package.delete` only deletes the *package manifest*, not the *underlying objects*. If you wish to delete the entire package *and* its objects, *delete the objects first*.

*Warning: the objects you delete will be lost forever. Ditto for the package revision.*

To delete, first browse the package then walk it, deleting its entry objects as follows:

```python
import boto3
import quilt3 as q3

s3 = boto3.client("s3")

reg = "s3://quilt-bio-staging"
pname = "akarve/delete-object"
p = q3.Package.browse(pname, registry=reg)

for (k, e) in p.walk():
    pk = e.physical_key
    s3.delete_object(Bucket=pk.bucket, Key=pk.path, VersionId=pk.version_id)
```

You can then follow the above with `q3.delete_package(pname, registry=reg, top_hash=p.top_hash)`.

### Do I have to login via quilt3 to use the Quilt APIs?

### How do I push to Quilt from a headless environment like a Docker container?

Configure [AWS CLI credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) and `quilt3` will use the same for its API calls.

> Be sure to run `quilt3 logout` if you've previously logged in.

Select among multiple profiles in your shell as follows:

```bash
export AWS_PROFILE=your_profile
```

The S3 permissions needed by `quilt3` are similar to [this bucket policy](https://docs.quilt.bio/advanced/crossaccount#bucket-policies) but `quilt3` does not need either `s3:GetBucketNotification` or `s3:PutBucketNotification`.

### How complex can my Athena queries be?

Amazon Athena supports a subset of Data Definition Language (DDL) and Data Manipulation Language (DML) statements, functions, operators, and data types, based on [Presto](https://prestodb.io/) and [Trino](https://trino.io/).

This allows for extremely granular querying of your data package name, metadata, and contents and includes logical operators, comparison functions, conditional expressions, mathematical functions, bitwise functions, date and time functions and operators, regular expression functions, and aggregate functions. Please review the references linked below to learn more.

#### Helpful examples

`regexp_extract_all(string, pattern)`

Return the substring(s) matched by the regular expression `pattern` in `string`

```sql
SELECT regexp_extract_all('1a 2b 14m', '\d+');
```

#### Considerations and limitations

There are [many considerations and limitations](https://docs.aws.amazon.com/athena/latest/ug/other-notable-limitations.html) when writing Amazon Athena queries.

#### References

* [SQL reference for Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/ddl-sql-reference.html)
* [Functions in Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/presto-functions.html)

### Are there any limitations on characters in Quilt filenames?

Yes. Quilt is built on top of Amazon S3, and has the same character limitations. Although any UTF-8 character is supported in an object key name (filename), using certain characters can result in problems with some applications and protocols. The following guideline will help you maximize compliance. For a comprehensive list of safe characters, characters that might require special handling, and characters to avoid, please review the official Amazon S3 documentation linked below.

#### List of safe characters

* Alphanumeric characters:
  * 0-9
  * a-z
  * A-Z
* Special characters:
  * Exclamation point (`!`)
  * Hyphen (`-`)
  * Underscore (`_`)
  * Period (`.`)
  * Asterisk (`*`)
  * Single quote (`'`)
  * Open parenthesis (`(`)
  * Close parenthesis (`)`)

For more details, see [Creating object key names](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) in the Amazon S3 documentation.

### How many IPs does a standard Quilt stack require?

Currently, a full size, multi-Availability Zone deployment (without [Voila](https://docs.quilt.bio/catalog/visualizationdashboards#voila)) requires at least 256 IPs. This means a minimum CIDR block of `/24`.

Optional additional features (such as automated data packaging) require additional IPs.

### The "Last Modified" column in the Quilt catalog is empty

Amazon S3 is a key-value store with prefixes but no true "folders". In the Quilt Catalog Bucket view, as in AWS Console, only objects have a "Last modified" value, whereas package entries and prefixes do not.


# Troubleshooting

For Catalog-specific issues, see [Catalog Troubleshooting](/quilt-platform-administrator/advanced/troubleshooting).

## Missing metadata when working with Quilt packages via the API

> `Package.set_dir()` on the package root (".") overrides package-level metadata. If you do not provide `set_dir(".", foo, meta=baz)` with a value for `meta=`, `set_dir` will set package-level metadata to `None`.

A common pattern is to `Package.browse()` to get the most recent version of a package, and then `Package.push()` updates. You can preserve package-level metadata when calling `set_dir(".", ...)` as follows:

```python
import quilt3

p = quilt3.Package.browse(
    "user-packages/geodata", 
    registry="s3://bucket_1"
)

p.set_dir(
    ".",
    "s3://bucket_2/path/to/new/geofiles",
    meta=p.meta
)

# Push changes to the S3 registry
p.push(
    "user-packages/geodata",
    registry="s3://bucket_1",
    message="Updating package geodata source data"
)
```

* [Reference](https://docs.quilt.bio/api-reference/package#package.set_dir).

## User creation and log in

Users can either be invited directly or are *just-in-time provisioned (JIP)* when they sign in via SSO and receive the "default role."

### Important conditions and pre-requisites

* If an admin (or any user) is created by JIP, or created through CloudFormation with an SSO Provider set to anything other than Disabled, then setting the password for that user has no effect and *password login will never succeed* for that user. Said another way, users created through SSO can only log in through SSO.
* You *must disable SSO* and enable `PasswordAuth` if you wish to log in as an admin using a password (as opposed to SSO).

### Unable to log in

The following are common causes of failed logins. In most cases we recommend that you check the browser's network panel for details.

1. SSO connector misconfigured. See [Single sign-on (SSO)](/quilt-platform-administrator/technical-reference#single-sign-on-sso) for details.
2. SSL errors are often caused by misspelled names, or incomplete Subject Alternate Names. The ACM certificate for `CertificateArnELB` must cover all three Quilt [hostnames](/quilt-platform-administrator/installation#dns-records) (plus the Connect host if Connect is enabled and `CertificateArnConnect` is empty) either via a suitable `*` or explicit Subject Alternate Names.

### Changing the admin email or password

Changing the admin password is only possible with `PasswordAuth=Enabled` in CloudFormation and is subject to the following limitations for security reasons:

* Has no effect if SSO is in use, or was in use when the admin was first created.
* Has no effect on pre-existing admin username/password pairs.

You can click "reset password" on the login page.

To change the admin email (e.g. you have accidentally broken your admin user) try the following:

1. Change the value of the `AdminEmail` CloudFormation parameter *to a net new email*.
2. Apply the change as a stack *Update*.
3. Once the update is successful, the new admin can log in, set roles, and nominate other admins as needed.

## General stack update failure steps

On rare occasions, Quilt stack deployment updates might fail. This can happen for several reasons. To expedite resolution of stack deployment issues, it's helpful to have the following data and output from the following [AWS CLI](https://aws.amazon.com/cli/) commands when contacting <support@quilt.bio>.

1. Quilt stack outputs:

   ```sh
   STACK_NAME="YOUR_QUILT_STACK"
   aws cloudformation describe-stacks \
       --stack-name "$STACK_NAME" \
       --query 'Stacks[].Outputs'
   ```
2. Initiate drift detection:

   ```sh
   aws cloudformation detect-stack-drift \
       --stack-name "$STACK_NAME"
   ```
3. *After drift detection is complete*:

   ```sh
   aws cloudformation describe-stack-resource-drifts \
       --stack-name "$STACK_NAME"
   ```
4. Quilt stack events:

   ```sh
   aws cloudformation describe-stack-events \
       --stack-name "$STACK_NAME"
   ```

## Collect logs to diagnose

To expedite the resolution of any errors encountered while using Quilt, please capture the following logs and share them with Quilt support:

### Elastic Container Service (ECS)

1. Find the name of your Quilt stack:

   ```sh
   aws cloudformation list-stacks
   ```
2. Capture Quilt log events for the last 30 minutes as follows:

   ```sh
   STACK_NAME="YOUR_QUILT_STACK"
   aws logs filter-log-events \
       --log-group-name "$STACK_NAME" \
       --start-time "$(( ($(date +%s) - 1800) * 1000 ))" \
       --end-time "$(( $(date +%s) * 1000 ))" > log-quilt-ecs-events.json
   ```

### IAM permissions

Determine which principal you're using as follows:

```sh
aws sts get-caller-identity
```

### S3 objects

Inspect problematic objects with the following commands:

```sh
BUCKET="YOUR_BUCKET"
PREFIX="YOUR_PREFIX"
aws s3api list-object-versions --bucket "$BUCKET" --prefix "$PREFIX"
aws s3api get-object-tagging --bucket "$BUCKET" --key "$PREFIX"
```

### Specific logical resources

Sometimes you may wish to find an ID or other information from a logical resource in a Quilt stack. The following example is for security groups. Modify the commands as needed for other resource types.

```sh
STACK_NAME="YOUR_QUILT_STACK"
RESOURCE_ID="YOUR_LOGICAL_ID"
SG_ID=$(
  aws cloudformation describe-stack-resource \
    --stack-name "$STACK_NAME" \
    --logical-resource-id "$RESOURCE_ID" \
    --query 'StackResourceDetail.PhysicalResourceId' \
    --output text
)
aws ec2 describe-security-groups --group-ids "${SG_ID}"
```

### Event source mapping

The event source mapping is a Lambda resource that reads from SQS.

```sh
STACK_NAME="YOUR_QUILT_STACK"
aws lambda get-event-source-mapping --uuid \
    $(aws cloudformation describe-stack-resource \
        --stack-name "$STACK_NAME" \
        --logical-resource-id LambdaFunctionEventSourceMapping \
        --query StackResourceDetail.PhysicalResourceId --output text)
```

### Remediation

If for some reason the event source mapping is disabled, it can be enabled as follows.

```sh
STACK_NAME="YOUR_QUILT_STACK"
aws lambda update-event-source-mapping --uuid \
    $(aws cloudformation describe-stack-resource \
        --stack-name "$STACK_NAME" \
        --logical-resource-id LambdaFunctionEventSourceMapping \
        --query StackResourceDetail.PhysicalResourceId \
        --output text) \
    --enabled
```


# Benchling Packager

The Benchling Webhook creates a seamless connection between [Benchling](https://www.benchling.com)'s Electronic Lab Notebook (ELN) and [Quilt](https://www.quilt.bio)'s Scientific Data Management System (SDMS) for Amazon S3. It not only allows you to view Benchling metadata and attachments inside Quilt packages, but also enables users to browse Quilt package descriptions from inside Benchling notebooks.

The webhook works through a [Benchling App](https://docs.benchling.com/docs/getting-started-benchling-apps) that must be installed in your Organization by a Benchling Administrator and configured to call your stack's unique webhook (see Installation, below).

## Availability

It is available in the Quilt Platform (1.65 or later; Referenced Entities and bucketless deployments require 1.71.0 or later) or as a standalone CDK stack via the `@quiltdata/benchling-webhook` [npm package](https://www.npmjs.com/package/@quiltdata/benchling-webhook).

## Functionality

### Auto-Packaging

![Packaged Notebook](/files/DjDTIl9WxrxsPBifvzDB)

When scientists create notebook entries in Benchling, this webhook automatically:

* **Creates a dedicated Quilt package** for each notebook entry
* **Synchronizes metadata** from Benchling into that package, e.g.:
  * `authors` (list)
  * `created_at` (ISO timestamp)
  * `creator` (e.g. `"First Last <user_id@ent_XXXX>"`)
  * `display_id` (e.g. `"EXP0000XXXXXX"`)
  * `entry_id` (e.g. `"etr_XXXXXX"`)
  * `files` (list)
  * `links` (list of referenced Benchling objects — see [Referenced Entities](#referenced-entities))
  * `modified_at` (ISO timestamp)
  * `web_url` (URL string)
* **Copies attachments** from that notebook into Amazon S3 as part of the package.
* **Enables organizational data discovery** by making contents available in ElasticSearch, and metadata available in Amazon Athena.

### Bucketless Deployments

[Auto-Packaging](#auto-packaging) writes each entry's package to a configured **package bucket**. That bucket is now **optional** — you can run the webhook without one. This suits organizations that link Benchling entries to Quilt packages spread across many buckets rather than a single dedicated one.

When no package bucket is configured:

* **Setup omits the bucket.** The configuration wizard and secret creation no longer require a package bucket, so you can stand up the integration without dedicating one.
* **No default package is auto-created.** Entry and canvas events skip per-entry package creation and instead surface the Quilt packages that already reference the entry.
* **Discovery spans every bucket.** Linked-package search runs across all Quilt package-view buckets in the stack (via Amazon Athena) instead of a single bucket, and the source bucket is preserved when you browse a linked package's files or metadata from a Benchling canvas.

> **Note:** Requires Quilt Platform 1.71.0 or later (or standalone benchling-webhook v0.19.0 or later).

### Referenced Entities

When packaging a notebook entry, the webhook also discovers the Benchling objects that entry references — custom entities, sequences, results tables, and so on — and makes them **searchable by their human-readable name**. This answers questions like *"show me every experiment that referenced QB-2743.1."*

The package metadata gains a `links` array, one object per referenced entity, each with four fields:

```json
[
  {
    "type": "custom_entity",
    "id": "bfi_xCUXNVyG",
    "name": "QB-2743.1",
    "slug": "qb-2743-1"
  }
]
```

* **`type`** and **`id`** identify the referenced object.
* **`name`** is the authoritative Benchling display name, resolved via the Benchling API. It is the field you search on. It is `null` when the app lacks registry access for that object or the object type is unsupported.
* **`slug`** is a lossy token parsed from the object's URL, shown for reference only — it is **never** treated as a name or matched by name search.

To find packages that reference a given entity, search the Quilt Catalog for its name (e.g. `QB-2743.1`); matches are scoped to `links.name`.

The raw discovery is also written to a `links.json` file in each package for auditing and reprocessing.

> **Note:** This requires Quilt Platform 1.71.0 or later (or standalone benchling-webhook v0.18.0 or later). It is distinct from the manual [Package Linking](#package-linking) below, which tags packages by `experiment_id`.

### Package Linking

![experiment\_id](/files/r4dwfcjNdskQMrFN5psB)

In addition, Quilt users can 'tag' additional packages by setting the `experiment_id` (or a custom metadata key) to the display ID of a Benchling notebook, e.g., `EXP00001234`.

From inside the Quilt Catalog:

1. Navigate to the package of interest
2. Click 'Revise Package'
3. Go the metadata editor in the bottom left
4. In the bottom row, enter `experiment_id` as key and the display ID as the value.
5. Set the commit message and click 'Save'

### Benchling App Canvas

![App Canvas - Home](/files/9Pc5vhODt5sR3dwhvnpD)

The webhook includes a Benchling App Canvas, which allows Benchling users to view, browse, and sync the associated Quilt packages.

* Clicking the package name opens it in the Quilt Catalog
* The `sync` button will open the package or file in [QuiltSync](https://www.quilt.bio/quiltsync), if you have it installed.
* The `Update` button manually refreshes the package. The canvas stays browsable while Quilt re-exports in the background, showing a "pending" badge that flips to "complete" when the new revision is ready.
* Package updates also happen automatically: any metadata change to the linked entry triggers a refresh, as does a Benchling `reviewRecord` event (e.g. when a notebook entry is submitted for review).

![App Canvas - Updated](/files/ZbJjX2ALOreMHBDUKdSR)

The canvas also allows you to browse package contents:

![App Canvas - Browse](/files/3qfhiWkSh27GTbWT8KT5)

and view package metadata:

![App Canvas - Metadata](/files/1eqIiZCQLWtc77TRCgoV)

#### Inserting a Canvas

If the App Canvas is not already part of your standard notebook template, Benchling users can add it themselves:

1. Create a notebook entry
2. Select "Insert" → "Canvas"
3. Choose "Quilt Package"
4. After it is inserted, click the "Create" button

![App Canvas - Insert](/files/bJcs2F10Mbf6rgHCJygo)

## Installation

### 1. Installing the Benchling App

This requires a Benchling admin to use `npx` from [NodeJS](https://nodejs.org) version 18 or later.

#### 1.1 Generate a manifest

```bash
npx @quiltdata/benchling-webhook@latest manifest
```

This will generate an `app-manifest.yaml` file in your local folder

#### 1.2 Upload the manifest to Benchling

Follow Benchling's [create](https://docs.benchling.com/docs/getting-started-benchling-apps#creating-an-app-from-a-manifest) and [install](https://docs.benchling.com/docs/getting-started-benchling-apps#installing-your-app) instructions. Save the **App Definition ID**, **Client ID**, and **Client Secret** for the next step.

### 2. Configuring the Benchling App

Your command-line environment must have AWS credentials for the account containing your Quilt stack. All you need to do is use `npx` to run the package:

```bash
npx @quiltdata/benchling-webhook@latest
```

The wizard will guide you through:

1. **Catalog discovery** - Detect your Quilt catalog configuration
2. **Stack validation** - Extract settings from your CloudFormation stack
3. **Credential collection** - Enter Benchling app credentials
4. **Deployment mode selection**:
   * **Integrated**: Uses your Quilt stack's built-in webhook, if any
   * **Standalone**: Deploys a separate webhook stack for testing

**Note**: Configuration is stored in `~/.config/benchling-webhook/` using the [XDG Base Directory](https://wiki.archlinux.org/title/XDG_Base_Directory) standard, supporting multiple profiles.

### 3. Configure Webhook URL

Add the webhook URL (displayed after setup) to your [Benchling app settings](https://docs.benchling.com/docs/getting-started-benchling-apps#installing-your-app).

In the Benchling **Webhook Setup** dialog, set **Webhook Routing Setting** to **Suffixed**. Benchling then appends a path suffix based on the kind of event (`/lifecycle`, `/event`, or `/canvas`) to your configured webhook URL, which is what the Quilt webhook expects.

![Webhook Routing Setting](/files/rtBr66WZ3aHcGm0YJxft)

> **Important:** Do **not** select **Stable**. It posts every event to the bare webhook URL with no suffix, which the Quilt webhook does not handle — requests return `404 Endpoint not found`.

### 4. Test Integration

In Benchling:

1. Create a notebook entry
2. Insert Canvas → Select "Quilt Package"
3. Click "Create"

A Quilt package will be automatically created and linked to your notebook entry. If you run into problems, contact [Quilt Support](mailto:support@quilt.bio)


# Event-Driven Packaging

> EDP is in private preview. Ask your Quilt account manager for details.

## Overview

Data tend to be created in logical batches by machines, people, and pipelines. Detecting these logical events from Amazon S3 events alone is complex and requires extensive logic.

Quilt's *Event-Driven Packaging* (EDP) service intelligently groups one or more Amazon S3 object events into a single batch-level event. You can easily (and if desired, **automatically**) trigger logical events like data package creation that depend on batches rather than on individual files.

> Any AWS service or action that generates S3 object events may trigger the EDP service.

## Requirements

1. A pre-existing VPC that either includes a [NAT Gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) or the following [VPC endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html#concepts-vpc-endpoints):
   * Amazon S3 ([gateway endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/gateway-endpoints.html) or [interface endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html)).
   * EventBridge ([interface endpoint](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-related-service-vpc.html)).
2. Enable [EventBridge S3 Events](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications-eventbridge.html) for all S3 buckets to be monitored by EDP.

## Deployment

EDP deploys Lambda and RDS resources to monitor S3 and generate EventBridge events under user-configurable conditions.

### Networking

* Lambda and RDS resources are placed in the `VPC` and `Subnets` that you provide.
* `Subnets` are normally private and must be able to reach Amazon services such as EventBridge via port 443 (e.g. by means of a NAT gateway, or VPC endpoint).
* `SecurityGroup` should allow outbound access to AWS services on port 443. Does not need inbound access.

### Parameters

EDP is deployed by a standalone CloudFormation template with the following parameters:

| Parameter                   | Description                                                                                                                                             |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `VPC`                       | For EDP resources and Subnets.                                                                                                                          |
| `Subnets`                   | For EDP Lambda, RDS (see above for configuration).                                                                                                      |
| `SecurityGroup`             | For EDP Lambdas (see above for configuration).                                                                                                          |
| `BucketName`                | Name of the Amazon S3 bucket to monitor.                                                                                                                |
| `BucketIgnorePrefixes`      | Text string of comma separated bucket path segments to ignore, for example `raw/*, scratch/*`. Default value is an empty string (i.e. nothing ignored). |
| `BucketPrefixDepth`         | The number of `/`-separated *common* path segments at the beginning of an S3 object key. Default value is `2`.                                          |
| `BucketThresholdDuration`   | Trigger a notification when this number of seconds has elapsed since the last object event in the S3 bucket occurred. Default value is `300` seconds.   |
| `BucketThresholdEventCount` | Trigger a notification when this number of files have been created (since the prior trigger). Default value is `20`.                                    |
| `DBUser`                    | Username for EDP RDS instance.                                                                                                                          |
| `DBPassword`                | Password for EDP RDS instance.                                                                                                                          |
| `EventBusName`              | Name of custom EventBridge event bus that receives events.                                                                                              |

## How EDP works

1. EDP monitors S3 object events for *s3://bucket-name*
2. After a fixed number of object events (`BucketThresholdEventCount`) or a maximum duration within a common prefix (`BucketThresholdDuration`), EDP creates a `package-objects-ready` event that signals there is sufficient information to make Quilt data packages from a batch of files:

   * S3 bucket name
   * Common prefix
   * Number of files
   * Timestamp of event

   The event payload is JSON:

   ```json
   {
       "version":"0",
       "id":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
       "detail-type":"package-objects-ready",
       "source":"com.quiltdata.edp",
       "account":"XXXXXXXXXXXX",
       "time":"2022-12-08T20:01:34Z",
       "region":"us-east-1",
       "resources":[
           "arn:aws:s3:::bucket-name"
       ],
       "detail":{
           "version":"0.1",
           "bucket":"bucket-name",
           "prefix":"prefix-path-1/prefix-path-2/"
       }
   }
   ```
3. EDP publishes the event to an AWS EventBridge bus. From there the event can be forwarded to any [services that can be targeted from AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html) for additional manual or automatic processing.

EDP, upon completion and if configured to do so, may warm its contents to a File Gateway where it has read permissions to ensure that new EDP-created Quilt packages are available to Gateway clients like Windows Workspaces.

> Users can optionally subscribe directly to the EDP SNS topic. This is useful for both debugging and viewing how events are structured.

## Example: Lambda function to automatically create data packages

1. An instrument automatically uploads a folder containing files from a single experiment into *s3://instrument-bucket/instrument-name/experiment-id/*.
2. EDP listens for events in *s3://instrument-bucket/instrument-name/experiment-id/\**. After the specified duration or event count, a `package-objects-ready` event is generated and sent to EventBridge.
3. A custom SNS topic is created for monitoring data package creation that Lab and Computational scientists subscribe to (`SNS_TOPIC_ARN`).
4. A custom lambda function triggered by the `package-objects-ready` event processes the experiment files and generates a data package. Additional processing includes (but is not limited to):

   * Enhance the package with documentation, charts, and metadata, such as the following:
     * `README.md`: Noting that the package was created by EDP, a custom lambda function, and validated with a [Quilt workflow](/workflows).
     * [`quilt_summarize.json`](https://docs.quilt.bio/quilt-ecosystem-integrations/pages/TI1BMRExTH2qFUEjKhOJ#quilt_summarize.json)
     * [`.quiltignore`](/quilt-python-sdk/advanced/.quiltignore)
   * Package metadata creation and validation: Send an SNS notification on [metadata validation](/workflows) failure.

   ```python
   import datetime
   import functools
   import os
   import pathlib
   import tempfile
   import boto3
   import quilt3 as quilt3
   from aws_lambda_powertools import Logger

   logger = Logger()
   s3 = boto3.client("s3")
   sns = boto3.client("sns")

   # Configuration environment variables defined for Lambda function
   WORKFLOW_NAME = os.environ.get("WORKFLOW_NAME") or ...
   QUARANTINE_BUCKET_NAME = os.environ["QUARANTINE_BUCKET_NAME"]
   SNS_TOPIC_ARN = os.environ["SNS_TOPIC_ARN"]
   QUILT_URL = os.environ["QUILT_URL"]

   # README.md default Markdown
   QUILT_README_STR = f"""#Quilt package auto-generated by EDP\n\n
   Created on {datetime.date.today()} by an
   automated Lambda agent for the {WORKFLOW_NAME} workflow."""

   # File system files for Quilt to ignore
   QUILT_IGNORE_STR = """.DS_*
   Icon
   ._*
   .TemporaryItems
   .Trashes
   .VolumeIcon.icns
   """

   # Define helpful additional data package files
   beautify_files = {
       "README.md": QUILT_README_STR,
       ".quiltignore": QUILT_IGNORE_STR,
   }

   @logger.inject_lambda_context
   def lambda_handler(event, context):

       # EDP event data
       bucket = event["detail"]["bucket"]
       prefix = event["detail"]["prefix"]

       # Add every file in the prefix folder to the new data package
       pkg = quilt3.Package().set_dir(".", f"s3://{bucket}/{prefix}")

       # Decorate the data package with example required metadata (as defined by WORKFLOW_NAME)
       meta = {
           "Author": "EDP",
           "ComputerName": "Genome Lab - 1234",
           "Date": datetime.date.today().strftime("%Y-%m-%d"),
           "ProjectID": "YYD",
           "StudyID": "ABC-23-023394"
       }

       with tempfile.TemporaryDirectory() as tmpdir:
           tmpdir_path = pathlib.Path(tmpdir)
           for name, body in beautify_files.items():
               if name in pkg:
                   logger.debug(f"File {name} already exists. Ignoring.")
                   continue
               logger.debug(f"File {name} does not exist at {prefix}. Creating.")
               file_path = tmpdir_path / name
               file_path.write_text(body)
               pkg.set(name, file_path)

           # Add metadata to package
           pkg.set_meta(meta)
           # Remove leading & trailing characters
           pkg_name = prefix.strip("/")

           # Define callable Quilt push()
           push = functools.partial(
               pkg.push,
               pkg_name,
               registry=f"s3://{bucket}",
               force=True,
               message="Created by EDP",
               workflow=WORKFLOW_NAME
           )

           # Validate against the Quilt workflow schema
           try:
               push(dedupe=True)
           except quilt3.workflows.WorkflowValidationError as e:
               logger.warning("Workflow check failed")

               # Write out error to README.md file in quarantine bucket
               file_path = tmpdir_path / "README.md"
               file_path.write_text(str(e))
               pkg.set("README.md", file_path)

               # Push package to quarantine bucket
               push(registry=f"s3://{QUARANTINE_BUCKET_NAME}", workflow=...)

               # Error SNS notification content
               subject = f"Failed to create package"
               message = (
                   f"Validation failed for workflow {WORKFLOW_NAME} while pushing "
                   f"package with name {pkg_name} to {bucket}. It was pushed to "
                   f"{QUARANTINE_BUCKET_NAME} instead.\n"
                   f"{QUILT_URL}/b/{QUARANTINE_BUCKET_NAME}/packages/{pkg_name}\n\n"
                   f"Error message is:\n{e}\n"
               )
               # Publish notification to SNS topic
               sns.publish(
                   TopicArn=SNS_TOPIC_ARN,
                   Message=message,
                   Subject=subject,
               )
   ```
5. If a metadata validation error occurs, an SNS event is sent to `SNS_TOPIC_ARN` noting that the package was created in the quarantine bucket. The SNS notification is routed to subscribers.
6. Computational scientist opens the new data package for additional analysis, modeling, and versioning.

## Debugging

EDP includes a [CloudWatch](https://aws.amazon.com/cloudwatch/) dashboard which exposes some metrics useful for debugging:

* **EDP event bus topic**: Displays the number of events emitted by EDP. If EDP is working correctly there should be one or more events received (depending on the time range selected).
* **Per-bucket metrics**:
  * **S3 EventBridge rule**: The number of events published to EventBridge from the specified Amazon S3 bucket. If there is no data, there are several possibilities:
    * **Invocations**: If this value is zero, the S3 bucket isn't correctly configured (`Send notifications to Amazon EventBridge for all events in this bucket` is not turned `On`).
    * **TriggeredRules**: If this value is zero, there was a problem with the automated EventBridge rule creation process during deployment. In general, you want the number of invocations to approximately equal the number of triggered rules.
    * **Failed Invocations**: This value should be zero. If greater than zero, there is an EDP configuration issue.
  * **Store in DB lambda**: If EDP is configured correctly, there should be zero errors and a 100% success rate.
  * **Emit event lambda**: If EDP is configured correctly, there should be zero errors and a 100% success rate.

![](/files/ZJuZse4pXsdrzbdntGQY)

## Limitations

* Each EDP stack monitors one S3 bucket.


# Nextflow Plugin

* [Nextflow](https://github.com/nextflow-io/nextflow) is a bioinformatics workflow manager that enables the development of portable and reproducible workflows. The software is used by scientists and engineers to write, deploy and share data-intensive, highly scalable, workflows on any infrastructure.
* `nf-quilt` is a [Nextflow plugin](https://www.nextflow.io/docs/latest/plugins.html) developed by Quilt Data that enables you read and write directly to [Quilt data packages](https://docs.quilt.bio) instead of just S3 locations.

## Quick Start

All you need to do is add the `nf-quilt` plugin to a Nextflow pipeline that writes to Amazon S3. The plugin will automatically create a Quilt package with metadata from each run. You can do this in one of three ways.

1. Add it to the command-line:

   ```sh
   nextflow run nf-core/rnaseq -plugins nf-quilt --outdir "s3://quilt-example-bucket/test/nf_quilt_rnaseq"
   ```
2. Include it in the nextflow config file (e.g., `main.nf`):

   ```groovy
   plugins {
       id 'nf-quilt'
   }
   ```
3. Specify it in the Advanced Options for a Seqera Platform job:

![Advanced Options > Nextflow config file](https://raw.githubusercontent.com/quiltdata/nf-quilt/master/README-Tower.png)

### Using Earlier Versions

To use older versions of `nf-quilt`, you can specify the version number of the plugin using the '@' sign:

```sh
nextflow run main.nf -plugins nf-quilt@0.7.16
```

### Using Prerelease Versions

To use unreleased versions of the `nf-quilt` plugin, you must also set the location using environment variable. For example, to use version 0.8.6, set `NXF_PLUGINS_TEST_REPOSITORY` from the command-line or the "Pre-run script" of the Seqera Platform:

```sh
# export NXF_VER=23.04.3
export LOG4J_DEBUG=true  # for verbose logging
export NXF_PLUGINS_TEST_REPOSITORY=https://github.com/quiltdata/nf-quilt/releases/download/0.8.6/nf-quilt-0.8.6-meta.json
```

## Output and Input URIs

The canonical reference to a package is defined by a `quilt+` URI. For example, the `s3://quilt-example-bucket/test/nf_quilt_rnaseq` S3 URI will create a package with the Quilt URI:

```string
quilt+s3://quilt-example-bucket#package=test/nf_quilt_rnaseq
```

You can then use that URI as input to future jobs, and similar URIs for the output, e.g.,

```bash
nextflow run my/analysis \
 --indir quilt+s3://quilt-example-bucket#package=test/nf_quilt_rnaseq.csv \
 --outdir quilt+s3://prod-bucket#package=experiment/analysis
```

### Additional Features

The `nf-quilt` plugin supports a wide range of additional options for configuring input, output, and metadata. For more details, or to participate in the development, please visit the [quiltdata/nf-quilt](https://github.com/quiltdata/nf-quilt) GitHub repository.


# QuiltSync Desktop Application

QuiltSync is a desktop application for syncing versioned Quilt data packages to your local machine. It provides local access to Quilt packages stored in S3, with support for Windows 10+, macOS 10.14+ (Intel & Apple Silicon), and Linux.

## Features

* Browse and sync packages via graphical interface
* Selective file sync to manage disk space
* Version control for data packages
* Browser-based OAuth 2.1 login (with legacy code-based fallback)
* Auto-generated commit messages
* Automatic detection of local and remote changes, with context-aware actions (e.g., **Commit and Push** when local edits exist, **Pull** when the remote is ahead, **Merge** when both sides diverged)
* One-click **Commit and Push** with per-user defaults (message template, workflow, metadata) configured in Settings
* Opt-in **Autosync** with independent Pull and Push toggles
* Tray icon with **Close to tray**, so Autosync keeps running with the main window closed
* Live filesystem watcher that refreshes local package status as files change
* Create local-only packages and set a remote later
* `.quiltignore` support with junk-file detection
* Unified Settings pane for general info, publish defaults, auth management, and diagnostics
* Companion `quilt` command-line interface published to [crates.io](https://crates.io/crates/quilt-cli)

## Getting Started

### Installation

Download and install QuiltSync from [quilt.bio/quiltsync](https://quilt.bio/quiltsync/).

### Opening Packages and Files

From the Quilt web catalog:

1. Navigate to a package or file
2. Click "Get Package" or "Get File"
3. Select "Open in QuiltSync"

![Open in QuiltSync](/files/XEKrXWeG8uhm9DRW0RWE)

### Authentication

On first use, QuiltSync authenticates you through your web browser. It prefers OAuth 2.1 Authorization Code flow with PKCE, and falls back to the legacy code-based flow for catalogs that do not yet support OAuth.

**OAuth (default):**

1. QuiltSync opens your browser to the Quilt Catalog login page
2. Sign in to your catalog
3. The browser returns to QuiltSync automatically via a `quilt://` deep link

**Legacy (fallback):**

1. QuiltSync opens your browser to the Quilt Catalog login page
2. Sign in to your catalog
3. Copy the access token from the browser back into QuiltSync

Either way the session is tied to your catalog login — no AWS credentials required.

![QuiltSync auth token](/files/e0JDwNjkIPHFzZSxwIEK)

### Selective Installation

When the package is opened, it shows a list of all files (pre-selected for download).

![QuiltSync download selected paths](/files/0SJkSkU0wMTHfmNd1DcD)

### Status-Aware Actions

QuiltSync continuously compares each package's local working copy against its remote revision and highlights only the actions that apply:

* **Commit and Push** activates when the package has a remote and something to ship — uncommitted changes, a pending local commit, or both. It uses the defaults configured in Settings → Commit and Push.
* **Pull** activates when the remote has new revisions not yet synced locally (disabled with a tooltip hint if the package has uncommitted local changes)
* **Merge** activates when local and remote have diverged
* **Set Remote** appears on local-only packages that have no remote yet

![QuiltSync package list with status-aware actions](/files/e4YAdxR4r17QZr4wedly)

### Background Autosync

> New in Quilt Platform version 1.70

QuiltSync includes an opt-in **Autosync** loop with independent **Pull** and **Push** toggles:

* **Auto-pull** refreshes `latest` for installed remote packages when the working tree is clean.
* **Auto-push** commits and publishes quiet local changes using your configured **Commit and Push** defaults. It pauses automatically when a package has pending changes you are still editing or when local and remote have diverged, so it never overwrites work in progress.

A live per-mapping **filesystem watcher** refreshes local package status when files change on disk, so status badges and entry lists update within about half a second without a manual reload. Both are configured in Settings:

![QuiltSync Autosync and Filesystem Watcher settings](/files/lRcsrwoeQyEmf4TzVN4D)

### Tray Icon and Close to Tray

> New in Quilt Platform version 1.70

A tray-resident shell keeps Autosync running even with the main window closed. Enable **Close to tray** in Settings to hide the window instead of quitting. The tray icon reflects current status — idle, syncing, paused, or error — and offers **Open Quilt** and **Quit** actions.

### Resolving Divergence (Merge)

> Updated in Quilt Platform version 1.70

When local and remote diverge, the **Merge** page labels each action by the direction data flows:

* **Promote my commit** pushes your local commit and then tags it as `latest` (the push now happens before tagging, so `latest` only ever points at a revision that has been uploaded).
* **Overwrite local with remote** resets your local state to the remote revision, discarding uncommitted local edits.

### Committing Changes

After modifying synced files locally, you can commit changes back to Quilt as a new package version:

1. Open the commit page in QuiltSync
2. Review the auto-generated commit message, which summarizes the changed files
3. Edit the message if needed
4. Click **Commit** to create a new revision
5. Click **Push** to upload that revision and set it as latest

![QuiltSync auto-generated commit](/files/b8VsSOAaCw2ogkyPv0RH)

![QuiltSync push](/files/xn94e0eR1gUNC95d5NcG)

### Ignoring Junk Files

QuiltSync honors `.quiltignore` files to keep build artifacts, OS metadata, and other noise out of your packages:

* Files matching a `.quiltignore` pattern are flagged with a "junk" badge in the entry list
* Per-entry popups let you **ignore** a file (adds a pattern to `.quiltignore`) or **un-ignore** one already covered by a pattern
* The package view also lets you toggle visibility of unmodified and ignored entries so only the files relevant to your next commit stay on screen

Use `.quiltignore` for transient outputs (e.g., `*.tmp`, `.DS_Store`, `node_modules/`) that shouldn't end up in committed revisions.

### Creating Local Packages

You can start a package entirely on your machine and wire it to a remote later:

1. Click **+ Create Local Package** in the Packages header
2. Enter a package name `pkg_prefix/pkg_prefix` (labeled **Namespace** in the dialog)
3. Optionally **Browse** to select a source directory to seed the package
4. Click **Create**

![QuiltSync create local package dialog](/files/Ah7DT1L8HJFxrJuHGgDg)

The new package appears in the list with a **Set Remote** action. Use it when you're ready to associate the package with an S3 bucket and push your first revision.

### Settings and Troubleshooting

Access settings via **SETTINGS** in the top-right header.

![QuiltSync Settings](/files/XLRL7bSMtqqz1AQYTOTx)

* **General**: Version (with release notes), home directory, and data directory
* **Commit and Push**: Defaults used by the one-click **Commit and Push** action — message template (with `{date}`, `{time}`, `{datetime}`, `{namespace}`, and `{changes}` placeholders), default workflow (bucket default or an override), and default metadata. **Edit** opens the defaults popup with a live message preview.
* **Autosync**: Independent **Pull** and **Push** toggles, the pull interval, and the quiet period to wait after the last edit before publishing (see [Background Autosync](#background-autosync))
* **Filesystem Watcher**: Toggle for the live on-disk status watcher
* **Auth**: List of authenticated catalogs with per-host **Re-Login** and **Logout** controls
* **Diagnostics**: Log level, logs directory, **Collect Logs**, then **Send to Sentry** or **Email Support** to share diagnostics (app version, OS, directory paths, authenticated host names, log files, and OAuth client IDs)

If QuiltSync fails to start after an upgrade, use **Re-Login** for the affected host or clear the data directory. Older cached manifests in Parquet format are automatically re-fetched from remote storage.

### Companion `quilt` CLI

> New in Quilt Platform version 1.70

The QuiltSync-based `quilt` command-line interface is published to [crates.io](https://crates.io) with prebuilt binaries for macOS and Linux. Install it with [`cargo binstall`](https://github.com/cargo-bins/cargo-binstall):

```bash
cargo binstall quilt-cli
```

The CLI shares QuiltSync's data directory, so you can manage the same set of local packages interchangeably from either the CLI or the GUI.

> This `quilt` CLI is distinct from the Python `quilt3` [CLI](/quilt-python-sdk/api-reference/cli), which ships with the `quilt3` package.

### Integration with Benchling

QuiltSync integrates with the [Benchling Webhook](/quilt-ecosystem-integrations/benchling) to provide seamless access to Quilt packages from Benchling notebooks.

![Benchling App Canvas](/files/9Pc5vhODt5sR3dwhvnpD)

When viewing a package in the Benchling App Canvas:

1. Click the "sync" button next to any package or file
2. QuiltSync automatically opens with the selected package
3. Select files to sync locally
4. Work offline with your data

This integration allows scientists to move from notebook entries to local datasets without leaving their Benchling workflow. For more details, see [Benchling App Canvas](/quilt-ecosystem-integrations/benchling#benchling-app-canvas).


# Introduction

[![docs on\_gitbook](https://img.shields.io/badge/docs-on_gitbook-blue.svg?style=flat-square)](https://docs.quiltdata.com/) [![chat on\_slack](https://img.shields.io/badge/chat-on_slack-blue.svg?style=flat-square)](https://slack.quiltdata.com/) [![codecov](https://codecov.io/gh/quiltdata/quilt/branch/master/graph/badge.svg)](https://codecov.io/gh/quiltdata/quilt) [![pypi](https://img.shields.io/pypi/v/quilt3.svg?style=flat-square)](https://pypi.org/project/quilt3/)

## Quilt is a self-organizing data hub

### Python Quick start, tutorials

If you have Python and an S3 bucket, you're ready to create versioned datasets with Quilt. Visit the [Quilt docs](https://docs.quiltdata.com/installation) for installation instructions, a quick start, and more.

### Quilt in action

* [open.quiltdata.com](https://open.quiltdata.com/) is a petabyte-scale open data portal that runs on Quilt
* [quiltdata.com](https://quiltdata.com) includes case studies, use cases, videos, and instructions on how to run a private Quilt instance
* [Versioning data and models for rapid experimentation in machine learning](https://medium.com/pytorch/how-to-iterate-faster-in-machine-learning-by-versioning-data-and-models-featuring-detectron2-4fd2f9338df5) shows how to use Quilt for real world projects

### Who is Quilt for?

Quilt is for data-driven teams and offers features for coders (data scientists, data engineers, developers) and business users alike.

### What does Quilt do?

Quilt manages data like code so that teams in machine learning, biotech, and analytics can experiment faster, build smarter models, and recover from errors.

### How does Quilt work?

Quilt consists of a Python client, web catalog, lambda functions—all of which are open source—plus a suite of backend services and Docker containers orchestrated by CloudFormation.

The backend services are available under a paid license on [quiltdata.com](https://quiltdata.com).

### Use cases

* **Share** data at scale. Quilt wraps AWS S3 to add simple URLs, web preview for large files, and sharing via email address (no need to create an IAM role).
* **Understand** data better through inline documentation (Jupyter notebooks, markdown) and visualizations (Vega, Vega Lite)
* **Discover** related data by indexing objects in ElasticSearch
* **Model** data by providing a home for large data and models that don't fit in git, and by providing immutable versions for objects and data sets (a.k.a. "Quilt Packages")
* **Decide** by broadening data access within the organization and supporting the documentation of decision processes through audit-able versioning and inline documentation

### Roadmap

#### I - Performance and core services

* [x] Address performance issues with push (e.g. re-hash)
* [x] Provide Presto-DB-powered services for filtering package repos with SQL
* [ ] Transition S3 manifests to [Apache Iceberg](https://iceberg.apache.org/) tables

#### II - CI/CD for data

* [ ] Ability to fork/merge packages
* [ ] Data quality monitoring

#### III - Storage agnostic (support Azure, GCP buckets)

* [ ] Evaluate min.io and ceph.io as shims
* [ ] Evaluate feasibility of on-prem local storage as a repo

#### IV - Cloud agnostic

* [ ] Evaluate K8s and Terraform to replace CloudFormation
* [ ] Shim lambdas (consider serverless.com)
* [ ] Shim ElasticSearch (consider SOLR)
* [ ] Shim IAM via RBAC


# Installation

Quilt has two user-facing components:

* A Python client
* A web catalog

## Python client

Python 3.6 or higher is required.

```bash
$ pip install quilt3[pyarrow]
```

If you do not need to serialize and deserialize dataframes with Quilt, you can obtain a smaller install, useful in disk-constrained environments like AWS Lambda, with `pip install quilt3`.

If you plan to use [Quilt Catalog Local Development Mode](/version-5.0.x/catalog/localmode), add `catalog` extra while installing `quilt3`, e.g.:

```bash
$ pip install quilt3[catalog,pyarrow]
```

If you wish to use AWS resources, such as S3 buckets, you will need valid AWS credentials. If this is your first time using the AWS CLI, run the following:

```bash
$ pip install awscli
$ aws configure
```

If you are already using the AWS CLI, you may use your existing profile, or [create a new profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html).

### Developer

Install the current Quilt client from `master`:

```bash
$ pip install git+https://github.com/quiltdata/quilt.git#subdirectory=api/python
```

## Web catalog and backend services (on AWS)

See [Enterprise install](/version-5.0.x/advanced/technical-reference) for installation instructions.


# Quick start

## Learn `quilt3` in Python

* [This is the on-ramp for coders. Let's go.](https://open.quiltdata.com/b/quilt-example/packages/examples/quickstart/tree/latest/QuickStart.ipynb)

## Video tutorials

* [How to work with S3 data sets using Quilt](https://www.youtube.com/playlist?list=PLmXfD6KoA_vBtgGgt0X4ui4cRlEkdJKp9)

## Jupyter notebooks for getting started with Quilt

* [CORD-19 exploration](https://open.quiltdata.com/b/quilt-example/packages/akarve/cord19) - Explore a real world dataset with the Quilt API

## Tutorial: Quilt + PyTorch for image recognition and deep learning

* [Versioning data and models for rapid experimentation in machine learning](https://medium.com/pytorch/how-to-iterate-faster-in-machine-learning-by-versioning-data-and-models-featuring-detectron2-4fd2f9338df5)

## Open data

Visit [open.quiltdata.com](http://open.quiltdata.com/).


# Mental model

Quilt represents datasets as *packages*. A package is an immutable collection of related files with a handle of the form `AUTHOR/DESCRIPTION`, a cryptographic *top-hash* (or hash of hashes) that uniquely identifies package contents, and a backing *manifest*.

The manifest is serialized as file that contains *entries*. Manifest entries are tuples of the following form:

`(LOGICAL_KEY, PHYSICAL_KEYS, HASH, METADATA)`

*Logical keys* are user-facing friendly names, like `"README.md"`. *Physical keys* are fully qualified paths to bytes on disk, or bytes in S3. A *hash* is a digest of the physical key's contents, usually SHA-256. *Metadata* are a dictionary that may contain user-defined keys for metadata like bounding boxes, labels, or provenance information (e.g. {"algorithm\_version": "4.4.1"} to indicate how a given file was created).

Package manifests are stored in *registries*. Quilt supports both local disk and Amazon S3 buckets as registry. A registry may store manifests as well as the primary data. S3 was chosen for its widespread adoption, first-class versioning support, and cost/performance profile. The Quilt roadmap includes plans to support more storage formats in the future (e.g. GCP, Azure, NAS, etc.).

By way of illustration first entry of a package manifest for the COCO machine learning dataset are shown below.

```json
{
    "logical_key": "annotations/captions_train2017.json",
    "physical_keys":
    ["s3://quilt-ml-data/data/raw/annotations/captions_train2017.json?versionId=UtzkAN8FP4irtroeN9bfYP1yKzX7ko3G"],
    "size": 91865115,
    "hash": {
    "type": "SHA256",
    "value":
    "4b62086319480e0739ef390d04084515defb9c213ff13605a036061e33314317"},
    "meta": {}
}
```

## Buckets are branches

In Quilt, S3 buckets are analogous to branches in git. Each bucket is a self-contained registry for one or more packages. As package data and schemas are refined, you can promote a package to a new bucket to signify its increased data quality.

We generally recommend a minimum of three buckets for the data lifecycle:

* Raw
* Stage
* Production

![](/files/UOwbkCcnxxdlBsDXUUVd)

See [Quilt workflows](/version-5.0.x/advanced/workflows) for more on how you can control data quality with schemas.


# Architecture

Quilt data packages are reproducible, discoverable, and trusted data sets that live in blob storage.

A package manifest includes both *metadata* and physical keys to primary data.

All package metadata and data are stored in your S3 buckets. A slice of the package-level metadata, as well as S3 object contents, are sent to an ElasticSearch cluster managed by Quilt. All Quilt package manifests are accessible via SQL using AWS Athena.

![](/files/Erg1onAYoTB3X3mYRrZB)


# Editing a Package

Data in Quilt is organized in terms of **data packages**. A data package is a logical group of files, directories, and metadata.

### Initializing a package

To edit a new empty package, use the package constructor:

```python
import quilt3
p = quilt3.Package()
```

To edit a preexisting package, we need to first make sure to install the package:

```python
quilt3.Package.install(
    "examples/hurdat",
    "s3://quilt-example",
)
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 5902.48entries/s]

Successfully installed package 'examples/hurdat', tophash=f8d1478 from s3://quilt-example

```

Use `browse` to edit the package:

```python
p = quilt3.Package.browse('examples/hurdat')
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 9920.30entries/s]
```

For more information on accessing existing packages see the section "[Installing a Package](/version-5.0.x/walkthrough/installing-a-package)".

### Adding data to a package

Use the `set` and `set_dir` commands to add individual files and whole directories, respectively, to a `Package`:

```python
# add entries individually using `set`
# ie p.set("foo.csv", "/local/path/foo.csv"),
# p.set("bar.csv", "s3://bucket/path/bar.csv")

# create test data
with open("data.csv", "w") as f:
    f.write("id, value\na, 42")

p = quilt3.Package()
p.set("data.csv", "data.csv")
p.set("banner.png", "s3://quilt-example/imgs/banner.png")

# or grab everything in a directory at once using `set_dir`
# ie p.set_dir("stuff/", "/path/to/stuff/"),
# p.set_dir("things/", "s3://path/to/things/")

# create test directory
import os
os.mkdir("data")
p.set_dir("stuff/", "./data/")
p.set_dir("imgs/", "s3://quilt-example/imgs/")
```

```
(remote Package)
 └─banner.png
 └─data.csv
 └─imgs/
   └─banner.png
 └─stuff/
```

The first parameter to these functions is the *logical key*, which will determine where the file lives within the package. So after running the commands above our package will look like this:

```python
p
```

```
(remote Package)
 └─banner.png
 └─data.csv
 └─imgs/
   └─banner.png
 └─stuff/
```

The second parameter is the *physical key*, which states the file's actual location. The physical key may point to either a local file or a remote object (with an `s3://` path).

If the physical key and the logical key are the same, you may omit the second argument:

```python
# assuming data.csv is in the current directory
p = quilt3.Package()
p.set("data.csv")
```

```
(local Package)
 └─data.csv
```

Another useful trick. Use `"."` to set the contents of the package to that of the current directory:

```python
# switch to a test directory and create some test files
import os
%cd data/
os.mkdir("stuff")
with open("new_data.csv", "w") as f:
    f.write("id, value\na, 42")

# set the contents of the package to that of the current directory
p.set_dir(".", ".")
```

```
(local Package)
 └─data.csv
 └─new_data.csv
```

### Deleting data in a package

Use `delete` to remove entries from a package:

```python
p.delete("data.csv")
```

```
(local Package)
 └─new_data.csv
```

Note that this will only remove this piece of data from the package. It will not delete the actual data itself.

### Adding metadata to a package

Packages support metadata anywhere in the package. To set metadata on package entries or directories, use the `meta` argument:

```python
p = quilt3.Package()
p.set("data.csv", "new_data.csv", meta={"type": "csv"})
p.set_dir("stuff/", "stuff/", meta={"origin": "unknown"})
```

```
(local Package)
 └─data.csv
 └─stuff/
```

You can also set metadata on the package as a whole using `set_meta`.

```python
# set metadata on a package
p.set_meta({"package-type": "demo"})
```

```
(local Package)
 └─data.csv
 └─stuff/
```


# Uploading a Package

Once your package is ready it's time to save and distribute it.

### Saving a package manifest locally

To save a package to your local disk use `build`.

```python
import quilt3
p = quilt3.Package()

top_hash = p.build("aneesh/test_data")
```

Building a package requires providing it with a name. Packages names must follow the `"${namespace}/${packagename}"` format. For small teams, we recommend using the package author's name as the namespace.

### Authenticating to a remote registry

To share a package with others via a remote registry you will first need to authenticate against, if you haven't done so already:

```python
# only need to run this once
# ie quilt3.config('https://your-catalog-homepage/')
quilt3.config('https://open.quiltdata.com/')

# follow the instructions to finish login
quilt3.login()
```

### Pushing a package to a remote registry

To share a package with others via a remote registry, use `push`:

```python
p = quilt3.Package()
p.push(
    "aneesh/test_data",
    "s3://quilt-example",
    message="Updated version my package"
)
```

`s3://quilt-example` is the *registry*—the storage backend that the package is available from.

You can omit the registry argument if you configure a `default_remote_registry` (this setting persists between sessions):

```python
quilt3.config(default_remote_registry='s3://quilt-example')
p = quilt3.Package()
p.push("aneesh/test_data")
```

You can control where files land using `dest`:

```python
p = quilt3.Package()
p.push(
    "aneesh/test_data",
    dest="s3://quilt-example/foo/bar"
)
```

> For even more fine-grained control of object landing paths see [Materialization](/version-5.0.x/advanced/materialization).

### Saving a package on a remote registry

`push` will send both a package manifest and its data to a remote registry. This will involve copying your data to S3. To save just the package manifest to S3 without any data copying, use `build`:

```python
p = quilt3.Package()
p.build("aneesh/test_data", "s3://quilt-example")
```

This will create a new version of your package with all of its physical keys preserved.

### Delete a package from a registry

To delete a package from a registry:

```python
# delete a package in the local registry
quilt3.delete_package("aneesh/test_data")

# delete a package in a remote registry
quilt3.delete_package("aneesh/test_data", "s3://quilt-example")
```

Note that this will not delete any package data, only the package manifest.


# Installing a Package

### Searching for packages

As explained in ["Uploading a Package"](/version-5.0.x/walkthrough/uploading-a-package), packages are managed using *registries*. There is a one local registry on your machine, and potentially many remote registries elsewhere "in the world". Use `list_packages` to see the packages available on a registry:

```python
import quilt3

# list local packages
list(quilt3.list_packages())
```

```
['aneesh/cli-push',
 'examples/hurdat',
 'aleksey/hurdat']
```

```python
# list remote packages
list(quilt3.list_packages("s3://quilt-example"))
```

```
['aleksey/hurdat',
 'examples/hurdat',
 'quilt/altair',
 'quilt/hurdat',
 'quilt/open_fruit',
 'quilt/open_images']
```

### Installing a package

To make a remote package and all of its data available locally, `install` it.

The examples in this section use the `examples/hurdat` [demo package](https://open.quiltdata.com/b/quilt-example/tree/examples/hurdat/):

```python
quilt3.Package.install(
    "examples/hurdat",
    "s3://quilt-example",
)
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 7049.25entries/s]

Successfully installed package 'examples/hurdat', tophash=f8d1478 from s3://quilt-example

```

Note that unless this registry is public, you will need to be logged into a user who has read access to this registry in order to install from it:

```python
# only need to run this once
# ie quilt3.config('https://your-catalog-homepage/')
quilt3.config('https://open.quiltdata.com/')

# follow the instructions to finish login
quilt3.login()
```

Data files that you download are written to a folder in your local registry by default. You can specify an alternative destination using dest:

```python
quilt3.Package.install(
    "examples/hurdat", 
    "s3://quilt-example", 
    dest="./"
)
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 9027.77entries/s]
Copying objects: 100%|██████████| 3.62M/3.62M [00:00<00:00, 303MB/s]

Successfully installed package 'examples/hurdat', tophash=f8d1478 from s3://quilt-example

```

Finally, you can install a specific version of a package by specifying the corresponding top hash:

```python
quilt3.Package.install(
    "examples/hurdat", 
    "s3://quilt-example", 
    top_hash="058e62c"
)
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 11491.24entries/s]
Copying objects: 100%|██████████| 35.4k/35.4k [00:02<00:00, 14.3kB/s]

Successfully installed package 'examples/hurdat', tophash=058e62c from s3://quilt-example

```

### Browsing a package manifest

An alternative to `install` is `browse`. `browse` downloads a package manifest without also downloading the data in the package.

```python
# load a package manifest from a remote registry
p = quilt3.Package.browse("examples/hurdat", "s3://quilt-example")

# load a package manifest from the default remote registry
quilt3.config(default_remote_registry="s3://quilt-example")
p = quilt3.Package.browse("examples/hurdat")
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 7541.00entries/s]
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 10710.68entries/s]
```

`browse` is advantageous when you don't want to download everything in a package at once. For example if you just want to look at a package's metadata.

### Importing a package

You can import a local package from within Python:

```python
from quilt3.data.examples import hurdat
```

```
Loading manifest: 100%|██████████| 5/5 [00:00<00:00, 9637.65entries/s]
```

This allows you to manage your data and code dependencies all in one place in your Python scripts or Jupyter notebooks.


# Getting Data from a Package

The examples in this section use the `aleksey/hurdat` [demo package](https://open.quiltdata.com/b/quilt-example/tree/aleksey/hurdat/):

```python
import quilt3
p = quilt3.Package.browse('aleksey/hurdat', 's3://quilt-example')
p
```

```
Loading manifest: 100%|██████████| 7/7 [00:00<00:00, 8393.40entries/s]





(remote Package)
 └─.gitignore
 └─.quiltignore
 └─notebooks/
   └─QuickStart.ipynb
 └─quilt_summarize.json
 └─requirements.txt
 └─scripts/
   └─build.py
```

### Slicing through a package

Use `dict` key selection to slice into a package tree:

```python
# returns PackageEntry("requirements.txt")
p["requirements.txt"]
```

```
PackageEntry('s3://quilt-example/aleksey/hurdat/requirements.txt?versionId=bQtxuZlaylNVHi0GmxkSMofT5qXJvP95')
```

```python
# returns (remote Package)
p["notebooks"]
```

```
(remote Package)
 └─QuickStart.ipynb
```

Slicing into a `Package` directory returns another `Package` rooted at that subdirectory. Slicing into a package entry returns an individual `PackageEntry`.

### Downloading package data to disk

To download a subset of files from a package directory to a `dest`, use `fetch`:

```python
# download a subfolder
p["notebooks"].fetch()

# download a single file
p["notebooks"]["QuickStart.ipynb"].fetch()

# download everything
p.fetch()
```

```
Copying objects: 100%|██████████| 36.7k/36.7k [00:01<00:00, 22.7kB/s]
100%|██████████| 36.7k/36.7k [00:01<00:00, 24.1kB/s]
Copying objects: 100%|██████████| 39.9k/39.9k [00:02<00:00, 16.5kB/s]





(local Package)
 └─.gitignore
 └─.quiltignore
 └─notebooks/
   └─QuickStart.ipynb
 └─quilt_summarize.json
 └─requirements.txt
 └─scripts/
   └─build.py
```

`fetch` will default to downloading the files to the current directory, but you can also specify an alternative path:

```python
p["notebooks"]["QuickStart.ipynb"].fetch("./references/")
```

```
100%|██████████| 36.7k/36.7k [00:01<00:00, 22.5kB/s]





PackageEntry('file:///Users/gregezema/Documents/programs/quilt/docs/Walkthrough/references/')
```

### Downloading package data into memory

Alternatively, you can download data directly into memory:

```python
p["quilt_summarize.json"].deserialize()
```

```
['notebooks/QuickStart.ipynb']
```

To apply a custom deserializer to your data, pass the function as a parameter to the function. For example, to load a hypothetical `yaml` file using `yaml.safe_load`:

```python
import yaml
# returns a dict
p["quilt_summarize.json"].deserialize(yaml.safe_load)
```

```
['notebooks/QuickStart.ipynb']
```

The deserializer should accept a byte stream as input.

### Getting entry locations

You can get the path to a package entry or directory using `get`:

```python
# returns /path/to/pkg/root/notebooks/QuickStart.ipynb
p["notebooks"]["QuickStart.ipynb"].get()
```

```
's3://quilt-example/aleksey/hurdat/notebooks/QuickStart.ipynb?versionId=PH.9gsCH6LM9RQIqsy1U4X6H6s.VoQ_B'
```

### Getting metadata

Metadata is available using the `meta` property.

```python
# get entry metadata
p["notebooks"]["QuickStart.ipynb"].meta

# get directory metadata
p["notebooks"].meta

# get package metadata
p.meta
```


# Working with the Catalog

The Quilt Catalog is the second half of Quilt. It provides an interface on top of your S3 bucket that brings Quilt features like data packages and search to the web.

[**For a hands-on demo, check out the public demo catalog**](https://open.quiltdata.com/b/quilt-example)**.**

Note that you can use the Quilt Python API without using the catalog product, but they are designed to work together.

### Brief tour

The Quilt catalog provides a homepage for your catalog, based on a `README.md` file that you can optionally create at the top of your bucket.

#### Browse

![](/files/KLRDldeHwCqUuttvz3zn)

The catalog lets you navigate packages in the registry on the packages tab.

![](/files/zSaJfq3VLh40HJl6hfh0)

You can also browse the underlying S3 files using the files tab.

![](/files/eclP6WnvWd6Wl2TUjGTD)

#### Search

Catalogs also enable you to search the contents of your bucket. We support both unstructured (e.g. "`San Francisco`") and structured with [Query String Queries](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/query-dsl-query-string-query.html#query-string-syntax) (e.g. "`metadata_key: metadata_value`") search. Hits are previewed right in the search results.

![](/files/DxnniMdF2T7lD5kISpvj)

#### Upload package

You can upload a new package providing the name of the package, commit message, files, metadata, and [workflow](/version-5.0.x/advanced/workflows).

The name should have the format `namespace/package-name`.

The message needs to add notes on a new revision for this package.

Files are the content of your package.

The associated workflow contains the rules for validating your package.

The metadata can be added with JSON editor, represented as a key/value table with infinite nesting. If workflow contains JSON schema, you will have predefined key/value pairs according to the schema.

**JSON editor**

To add a new key/value field double click on an empty cell and type key name, then press "Enter" or "Tab", or click outside of the cell. To change value double click on that value.

Values can be strings, numbers, arrays, or objects. Every value that you type will be parsed as JSON.

We don't support references and compound types yet.

#### Push to bucket

You can push the existing package from one bucket to another. To use this feature consult [workflows](/version-5.0.x/advanced/workflows) page.

#### Summarize

Adding a `quilt_summarize.json` file to a data package (or S3 directory path) will enable content preview right on the landing page.

![](/files/RBSmWUyiYz18KWLX0oVP)

Colocating data with context in this way is a simple way of making your data projects approachable and accessible to collaborators.

`quilt_summarize.json` can be a list of paths to files in S3 that you want to include in your summary. For example: `["description.md", "../notebooks/exploration.ipynb"]`. Additionally, note that if a `README.md` file is present, it will always be rendered as well.

> There are currently some small limitations with preview:
>
> * Objects linked to in `quilt_summarize.json` are always previewed as of the latest version, even if you are browsing an old version of a package.
> * Object titles and image thumbnails link to the file view, even if you are in the package view.

### Admin UI

The Quilt catalog includes an admin panel that allows you to manage users and buckets in your stack and to customize your Quilt catalog. See [Admin UI docs](/version-5.0.x/catalog/admin) for details.

[**To learn more, check out the public demo catalog**](https://open.quiltdata.com/b/quilt-example).


# Working with a Bucket

Quilt allows you to create, read, and write packages both on your local filesystem and on S3 buckets configured to work with Quilt3. For convenience, we provide a simple API for working with S3 buckets that serves as an alternative to [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html).

### Connecting to a bucket

To connect to an S3 `Bucket`:

```python
import quilt3
b = quilt3.Bucket("s3://quilt-example")
```

This requires that the bucket is configured to work with Quilt 3. Unless this bucket is public, you will also first need to log into the catalog that controls this bucket:

```python
# only need to run this once
# ie quilt3.config('https://your-catalog-homepage/')
quilt3.config('https://open.quiltdata.com/')

# follow the instructions to finish login
quilt3.login()
```

### Introspecting a bucket

To see the contents of a `Bucket`, use `keys`:

```python
# returns a list of objects in the bucket
b.keys()
```

### Reading from a bucket

To download a file or folder from a bucket use `fetch`:

```python
# b.fetch("path/to/directory", "path/to/local")
b.fetch("aleksey/hurdat/", "./aleksey/")
b.fetch("README.md", "./read.md")
```

```
100%|██████████| 4.07M/4.07M [00:13<00:00, 304kB/s]   
100%|██████████| 1.55k/1.55k [00:01<00:00, 972B/s]
```

### Writing to a bucket

You can write data to a bucket.

```python
# put a file to a bucket
b.put_file("read.md", "./read.md")

# or put everything in a directory at once
b.put_dir("stuff", "./aleksey")
```

Note that `set` operations on a `Package` are `put` operations on a `Bucket`.

### Deleting objects in a bucket

```python
# always be careful when deleting

# delete a fle
b.delete("read.md")

# delete a directory
b.delete_dir("stuff/")
```

### Searching in a bucket

You can search for individual objects using `search`.

Note that this feature is currently only supported for buckets backed by a Quilt catalog instance. Before performing a search you must first configure a connection to that instance using `quilt3.config`.

```python
# for example
quilt3.config(navigator_url="https://open.quiltdata.com")
```

```
<QuiltConfig at '/Users/gregezema/Library/Application Support/Quilt/config.yml' {
    "navigator_url": "https://open.quiltdata.com",
    "default_local_registry": "file:///Users/gregezema/Library/Application%20Support/Quilt/packages",
    "default_remote_registry": null,
    "default_install_location": null,
    "registryUrl": "https://open-registry.quiltdata.com",
    "telemetry_disabled": false,
    "s3Proxy": "https://open-s3-proxy.quiltdata.com",
    "apiGatewayEndpoint": "https://sttuv8u2u4.execute-api.us-east-1.amazonaws.com/prod",
    "binaryApiGatewayEndpoint": "https://ap8tbn363c.execute-api.us-east-1.amazonaws.com/prod",
    "default_registry_version": 1
}>
```

Quilt supports unstructured search:

```python
# returns all files containing the word "thor"
b.search("thor")
```

```
{'took': 16,
 'timed_out': False,
 '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},
 'hits': {'total': 10,
  'max_score': 5.5741544,
  'hits': [{'_index': 'quilt-example-reindex-v8bc2377',
    '_type': '_doc',
    '_id': 'dima/node_modules2/highlight.js/README.md:KOGAC2bPIY9o7vQ3d3ryrD04VpGPmaH2',
    '_score': 5.5741544,
    '_source': {'size': 19316,
     'comment': '',
     'version_id': 'KOGAC2bPIY9o7vQ3d3ryrD04VpGPmaH2',
     'last_modified': '2019-12-12T01:33:15+00:00',
     'updated': '2019-12-12T01:33:15.209387',
     'key': 'dima/node_modules2/highlight.js/README.md'}},
   {'_index': 'quilt-example-reindex-v8bc2377',
    '_type': '_doc',
    '_id': 'akarve/amazon-reviews/camera-reviews.parquet:yoLoCR6tdnqE141f5F4EvbFbn2J12AJt',
    '_score': 0.080087036,
    '_source': {'user_meta': {},
     'size': 100764599,
     'comment': '',
     'version_id': 'yoLoCR6tdnqE141f5F4EvbFbn2J12AJt',
     'last_modified': '2019-10-08T02:53:01+00:00',
     'updated': '2019-10-08T02:53:31.040985',
     'key': 'akarve/amazon-reviews/camera-reviews.parquet'}}]}}
```

As well as structured search on metadata (note that this feature is experimental):

```python
# returns all files annotated {'name': 'thor'}
b.search("user_meta.name:'thor'")
```

```
{'took': 0,
 'timed_out': False,
 '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},
 'hits': {'total': 0, 'max_score': None, 'hits': []}}
```


# quilt3

Quilt API

## config(\*catalog\_url, \*\*config\_values) <a href="#config" id="config"></a>

Set or read the QUILT configuration.

To retrieve the current config, call directly, without arguments:

```python
    >>> import quilt3
    >>> quilt3.config()
```

To trigger autoconfiguration, call with just the navigator URL:

```python
    >>> quilt3.config('https://example.com')
```

To set config values, call with one or more key=value pairs:

```python
    >>> quilt3.config(navigator_url='http://example.com',
    ...               elastic_search_url='http://example.com/queries')
```

Default config values can be found in `quilt3.util.CONFIG_TEMPLATE`.

**Arguments**

* **catalog\_url**: A (single) URL indicating a location to configure from
* **\*\*config\_values**: `key=value` pairs to set in the config

**Returns**

`QuiltConfig`: (an ordered Mapping)

## delete\_package(name, registry=None, top\_hash=None) <a href="#delete_package" id="delete_package"></a>

Delete a package. Deletes only the manifest entries and not the underlying files.

**Arguments**

* **name (str)**: Name of the package
* **registry (str)**: The registry the package will be removed from
* **top\_hash (str)**: Optional. A package hash to delete, instead of the whole package.

## list\_package\_versions(name, registry=None) <a href="#list_package_versions" id="list_package_versions"></a>

Lists versions of a given package.

Returns an iterable of (version, hash) of a package in a registry. If the registry is None, default to the local registry.

**Arguments**

* **name (str)**: Name of the package
* **registry (str)**: location of registry to load package from.

**Returns**

An iterable of tuples containing the version and hash for the package.

## list\_packages(registry=None) <a href="#list_packages" id="list_packages"></a>

Lists Packages in the registry.

Returns an iterable of all named packages in a registry. If the registry is None, default to the local registry.

**Arguments**

* **registry (str)**: location of registry to load package from.

**Returns**

An iterable of strings containing the names of the packages

## logged\_in() <a href="#logged_in" id="logged_in"></a>

Return catalog URL if Quilt client is authenticated. Otherwise return `None`.

## login() <a href="#login" id="login"></a>

Authenticate to your Quilt stack and assume the role assigned to you by your stack administrator. Not required if you have existing AWS credentials.

Launches a web browser and asks the user for a token.

## logout() <a href="#logout" id="logout"></a>

Do not use Quilt credentials. Useful if you have existing AWS credentials.

## search(query, limit=10) <a href="#search" id="search"></a>

Execute a search against the configured search endpoint.

**Arguments**

* **query (str)**: query string to search
* **limit (number)**: maximum number of results to return. Defaults to 10

Query Syntax: [simple query string query](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-simple-query-string-query.html)

**Returns**

a list of objects with the following structure:

```
[{
`"_id"`: <document unique id>
`"_index"`: <source index>,
`"_score"`: <relevance score>
    "_source":
`"key"`: <key of the object>,
`"size"`: <size of object in bytes>,
`"user_meta"`: <user metadata from meta= via quilt3>,
`"last_modified"`: <timestamp from ElasticSearch>,
`"updated"`: <object timestamp from S3>,
`"version_id"`: <version_id of object version>
`"_type"`: <document type>
}, ...]
```


# quilt3.Package

## Package(self) <a href="#package" id="package"></a>

In-memory representation of a package

### manifest

Provides a generator of the dicts that make up the serialized package.

### top\_hash

Returns the top hash of the package.

Note that physical keys are not hashed because the package has the same semantics regardless of where the bytes come from.

**Returns**

A string that represents the top hash of the package

### Package.\_\_repr\_\_(self, max\_lines=20) <a href="#package.__repr__" id="package.__repr__"></a>

String representation of the Package.

### Package.install(name, registry=None, top\_hash=None, dest=None, dest\_registry=None, \*, path=None) <a href="#package.install" id="package.install"></a>

Installs a named package to the local registry and downloads its files.

**Arguments**

* **name(str)**: Name of package to install.
* **registry(str)**: Registry where package is located. Defaults to the default remote registry.
* **top\_hash(str)**: Hash of package to install. Defaults to latest.
* **dest(str)**: Local path to download files to.
* **dest\_registry(str)**: Registry to install package to. Defaults to local registry.
* **path(str)**: If specified, downloads only `path` or its children.

### Package.resolve\_hash(name, registry, hash\_prefix) <a href="#package.resolve_hash" id="package.resolve_hash"></a>

Find a hash that starts with a given prefix.

**Arguments**

* **name (str)**: name of package
* **registry (str)**: location of registry
* **hash\_prefix (str)**: hash prefix with length between 6 and 64 characters

### Package.browse(name, registry=None, top\_hash=None) <a href="#package.browse" id="package.browse"></a>

Load a package into memory from a registry without making a local copy of the manifest.

**Arguments**

* **name(string)**: name of package to load
* **registry(string)**: location of registry to load package from
* **top\_hash(string)**: top hash of package version to load

### Package.\_\_contains\_\_(self, logical\_key) <a href="#package.__contains__" id="package.__contains__"></a>

Checks whether the package contains a specified logical\_key.

**Returns**

True or False

### Package.\_\_getitem\_\_(self, logical\_key) <a href="#package.__getitem__" id="package.__getitem__"></a>

Filters the package based on prefix, and returns either a new Package or a PackageEntry.

**Arguments**

* **prefix(str)**: prefix to filter on

**Returns**

PackageEntry if prefix matches a logical\_key exactly otherwise Package

### Package.fetch(self, dest='./') <a href="#package.fetch" id="package.fetch"></a>

Copy all descendants to `dest`. Descendants are written under their logical names *relative* to self.

**Arguments**

* **dest**: where to put the files (locally)

**Returns**

A new Package object with entries from self, but with physical keys pointing to files in `dest`.

### Package.keys(self) <a href="#package.keys" id="package.keys"></a>

Returns logical keys in the package.

### Package.walk(self) <a href="#package.walk" id="package.walk"></a>

Generator that traverses all entries in the package tree and returns tuples of (key, entry), with keys in alphabetical order.

### Package.load(readable\_file) <a href="#package.load" id="package.load"></a>

Loads a package from a readable file-like object.

**Arguments**

* **readable\_file**: readable file-like object to deserialize package from

**Returns**

A new Package object

**Raises**

file not found json decode error invalid package exception

### Package.set\_dir(self, lkey, path=None, meta=None, update\_policy='incoming') <a href="#package.set_dir" id="package.set_dir"></a>

Adds all files from `path` to the package.

Recursively enumerates every file in `path`, and adds them to the package according to their relative location to `path`.

**Arguments**

* **lkey(string)**: prefix to add to every logical key, use '/' for the root of the package.
* **path(string)**: path to scan for files to add to package. If None, lkey will be substituted in as the path.
* **meta(dict)**: user level metadata dict to attach to lkey directory entry.
* **update\_policy(str)**: can be either 'incoming' (default) or 'existing'. If 'incoming', whenever logical keys match, always take the new entry from set\_dir. If 'existing', whenever logical keys match, retain existing entries and ignore new entries from set\_dir.

**Returns**

self

**Raises**

* `PackageException`: When `path` doesn't exist.
* `ValueError`: When `update_policy` is invalid.

### Package.get(self, logical\_key) <a href="#package.get" id="package.get"></a>

Gets object from logical\_key and returns its physical path. Equivalent to self\[logical\_key].get().

**Arguments**

* **logical\_key(string)**: logical key of the object to get

**Returns**

Physical path as a string.

**Raises**

* `KeyError`: when logical\_key is not present in the package
* `ValueError`: if the logical\_key points to a Package rather than PackageEntry.

### Package.readme(self) <a href="#package.readme" id="package.readme"></a>

Returns the README PackageEntry

The README is the entry with the logical key 'README.md' (case-sensitive). Will raise a QuiltException if no such entry exists.

### Package.set\_meta(self, meta) <a href="#package.set_meta" id="package.set_meta"></a>

Sets user metadata on this Package.

### Package.build(self, name, registry=None, message=None, \*, workflow=Ellipsis) <a href="#package.build" id="package.build"></a>

Serializes this package to a registry.

**Arguments**

* **name**: optional name for package
* **registry**: registry to build to defaults to local registry
* **message**: the commit message of the package
* **workflow**: workflow ID or `None` to skip workflow validation. If not specified, the default workflow will be used.
* **For details see**: <https://docs.quiltdata.com/advanced-usage/workflows>

**Returns**

The top hash as a string.

### Package.dump(self, writable\_file) <a href="#package.dump" id="package.dump"></a>

Serializes this package to a writable file-like object.

**Arguments**

* **writable\_file**: file-like object to write serialized package.

**Returns**

None

**Raises**

fail to create file fail to finish write

### Package.set(self, logical\_key, entry=None, meta=None, serialization\_location=None, serialization\_format\_opts=None) <a href="#package.set" id="package.set"></a>

Returns self with the object at logical\_key set to entry.

**Arguments**

* **logical\_key(string)**: logical key to update
* **entry(PackageEntry OR string OR object)**: new entry to place at logical\_key in the package. If entry is a string, it is treated as a URL, and an entry is created based on it. If entry is None, the logical key string will be substituted as the entry value. If entry is an object and quilt knows how to serialize it, it will immediately be serialized and written to disk, either to serialization\_location or to a location managed by quilt. List of types that Quilt can serialize is available by calling `quilt3.formats.FormatRegistry.all_supported_formats()`
* **meta(dict)**: user level metadata dict to attach to entry
* **serialization\_format\_opts(dict)**: Optional. If passed in, only used if entry is an object. Options to help Quilt understand how the object should be serialized. Useful for underspecified file formats like csv when content contains confusing characters. Will be passed as kwargs to the FormatHandler.serialize() function. See docstrings for individual FormatHandlers for full list of options -
* **https**: //github.com/quiltdata/quilt/blob/master/api/python/quilt3/formats.py
* **serialization\_location(string)**: Optional. If passed in, only used if entry is an object. Where the serialized object should be written, e.g. "./mydataframe.parquet"

**Returns**

self

### Package.delete(self, logical\_key) <a href="#package.delete" id="package.delete"></a>

Returns the package with logical\_key removed.

**Returns**

self

**Raises**

* `KeyError`: when logical\_key is not present to be deleted

### Package.push(self, name, registry=None, dest=None, message=None, selector\_fn=None, \*, workflow=Ellipsis, force=False) <a href="#package.push" id="package.push"></a>

Copies objects to path, then creates a new package that points to those objects. Copies each object in this package to path according to logical key structure, then adds to the registry a serialized version of this package with physical keys that point to the new copies.

Note that push is careful to not push data unnecessarily. To illustrate, imagine you have a PackageEntry: `pkg["entry_1"].physical_key = "/tmp/package_entry_1.json"`

If that entry would be pushed to `s3://bucket/prefix/entry_1.json`, but `s3://bucket/prefix/entry_1.json` already contains the exact same bytes as '/tmp/package\_entry\_1.json', `quilt3` will not push the bytes to s3, no matter what `selector_fn('entry_1', pkg["entry_1"])` returns.

However, selector\_fn will dictate whether the new package points to the local file or to s3:

If `selector_fn('entry_1', pkg["entry_1"]) == False`, `new_pkg["entry_1"] = ["/tmp/package_entry_1.json"]`

If `selector_fn('entry_1', pkg["entry_1"]) == True`, `new_pkg["entry_1"] = ["s3://bucket/prefix/entry_1.json"]`

By default, push will not overwrite an existing package if its top hash does not match the parent hash of the package being pushed. Use `force=True` to skip the check.

**Arguments**

* **name**: name for package in registry
* **dest**: where to copy the objects in the package Must be either an S3 URI prefix in the registry bucket, or a callable that takes
* **logical\_key, package\_entry, and top\_hash and returns S3 URI. S3 URIs format is s3**: //$bucket/$key.
* **registry**: registry where to create the new package
* **message**: the commit message for the new package
* **selector\_fn**: An optional function that determines which package entries should be copied to S3. The function takes in two arguments, logical\_key and package\_entry, and should return False if that PackageEntry should be skipped during push. If for example you have a package where the files are spread over multiple buckets and you add a single local file, you can use selector\_fn to only push the local file to s3 (instead of pushing all data to the destination bucket).
* **workflow**: workflow ID or `None` to skip workflow validation. If not specified, the default workflow will be used.
* **For details see**: <https://docs.quiltdata.com/advanced-usage/workflows>
* **force**: skip the top hash check and overwrite any existing package

**Returns**

A new package that points to the copied objects.

### Package.rollback(name, registry, top\_hash) <a href="#package.rollback" id="package.rollback"></a>

Set the "latest" version to the given hash.

**Arguments**

* **name(str)**: Name of package to rollback.
* **registry(str)**: Registry where package is located.
* **top\_hash(str)**: Hash to rollback to.

### Package.diff(self, other\_pkg) <a href="#package.diff" id="package.diff"></a>

Returns three lists -- added, modified, deleted.

Added: present in other\_pkg but not in self. Modified: present in both, but different. Deleted: present in self, but not other\_pkg.

**Arguments**

* **other\_pkg**: Package to diff

**Returns**

added, modified, deleted (all lists of logical keys)

### Package.map(self, f, include\_directories=False) <a href="#package.map" id="package.map"></a>

Performs a user-specified operation on each entry in the package.

**Arguments**

* **f(x, y)**: function The function to be applied to each package entry. It should take two inputs, a logical key and a PackageEntry.
* **include\_directories**: bool Whether or not to include directory entries in the map.

Returns: list The list of results generated by the map.

### Package.filter(self, f, include\_directories=False) <a href="#package.filter" id="package.filter"></a>

Applies a user-specified operation to each entry in the package, removing results that evaluate to False from the output.

**Arguments**

* **f(x, y)**: function The function to be applied to each package entry. It should take two inputs, a logical key and a PackageEntry. This function should return a boolean.
* **include\_directories**: bool Whether or not to include directory entries in the map.

**Returns**

A new package with entries that evaluated to False removed

### Package.verify(self, src, extra\_files\_ok=False) <a href="#package.verify" id="package.verify"></a>

Check if the contents of the given directory matches the package manifest.

**Arguments**

* **src(str)**: URL of the directory
* **extra\_files\_ok(bool)**: Whether extra files in the directory should cause a failure.

**Returns**

True if the package matches the directory; False otherwise.

## PackageEntry(self, physical\_key, size, hash\_obj, meta) <a href="#packageentry" id="packageentry"></a>

Represents an entry at a logical key inside a package.

**\_\_init\_\_**

Creates an entry.

**Arguments**

* **physical\_key**: a URI (either `s3://` or `file://`)
* **size(number)**: size of object in bytes
* **hash({'type'**: string, 'value': string}): hash object
* **for example**: {'type': 'SHA256', 'value': 'bb08a...'}
* **meta(dict)**: metadata dictionary

**Returns**

a PackageEntry

### **slots**

Built-in immutable sequence.

If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.

If the argument is a tuple, the return value is the same object.

### PackageEntry.as\_dict(self) <a href="#packageentry.as_dict" id="packageentry.as_dict"></a>

Returns dict representation of entry.

### PackageEntry.set\_meta(self, meta) <a href="#packageentry.set_meta" id="packageentry.set_meta"></a>

Sets the user\_meta for this PackageEntry.

### PackageEntry.set(self, path=None, meta=None) <a href="#packageentry.set" id="packageentry.set"></a>

Returns self with the physical key set to path.

**Arguments**

* **path(string)**: new path to place at logical\_key in the package Currently only supports a path on local disk
* **meta(dict)**: metadata dict to attach to entry. If meta is provided, set just updates the meta attached to logical\_key without changing anything else in the entry

**Returns**

self

### PackageEntry.get(self) <a href="#packageentry.get" id="packageentry.get"></a>

Returns the physical key of this PackageEntry.

### PackageEntry.get\_cached\_path(self) <a href="#packageentry.get_cached_path" id="packageentry.get_cached_path"></a>

Returns a locally cached physical key, if available.

### PackageEntry.get\_bytes(self, use\_cache\_if\_available=True) <a href="#packageentry.get_bytes" id="packageentry.get_bytes"></a>

Returns the bytes of the object this entry corresponds to. If 'use\_cache\_if\_available'=True, will first try to retrieve the bytes from cache.

### PackageEntry.get\_as\_json(self, use\_cache\_if\_available=True) <a href="#packageentry.get_as_json" id="packageentry.get_as_json"></a>

Returns a JSON file as a `dict`. Assumes that the file is encoded using utf-8.

If 'use\_cache\_if\_available'=True, will first try to retrieve the object from cache.

### PackageEntry.get\_as\_string(self, use\_cache\_if\_available=True) <a href="#packageentry.get_as_string" id="packageentry.get_as_string"></a>

Return the object as a string. Assumes that the file is encoded using utf-8.

If 'use\_cache\_if\_available'=True, will first try to retrieve the object from cache.

### PackageEntry.deserialize(self, func=None, \*\*format\_opts) <a href="#packageentry.deserialize" id="packageentry.deserialize"></a>

Returns the object this entry corresponds to.

**Arguments**

* **func**: Skip normal deserialization process, and call func(bytes), returning the result directly.
* **\*\*format\_opts**: Some data formats may take options. Though normally handled by metadata, these can be overridden here.

**Returns**

The deserialized object from the logical\_key

**Raises**

physical key failure hash verification fail when deserialization metadata is not present

### PackageEntry.fetch(self, dest=None) <a href="#packageentry.fetch" id="packageentry.fetch"></a>

Gets objects from entry and saves them to dest.

**Arguments**

* **dest**: where to put the files Defaults to the entry name

**Returns**

None

### PackageEntry.\_\_call\_\_(self, func=None, \*\*kwargs) <a href="#packageentry.__call__" id="packageentry.__call__"></a>

Shorthand for self.deserialize()


# quilt3.Bucket

Bucket interface for Quilt.

**\_\_init\_\_**

Creates a Bucket object.

**Arguments**

* **bucket\_uri(str)**: URI of bucket to target. Must start with 's3://'

**Returns**

A new Bucket

## Bucket.search(self, query, limit=10) <a href="#bucket.search" id="bucket.search"></a>

Execute a search against the configured search endpoint.

**Arguments**

* **query (str)**: query string to search
* **limit (number)**: maximum number of results to return. Defaults to 10

Query Syntax: By default, a normal plaintext search will be executed over the query string. You can use field-match syntax to filter on exact matches for fields in your metadata. The syntax for field match is `user_meta.$field_name:"exact_match"`.

**Returns**

a list of objects with the following structure:

```
[{
`"key"`: <key of the object>,
`"version_id"`: <version_id of object version>,
`"operation"`: <"Create" or "Delete">,
`"meta"`: <metadata attached to object>,
`"size"`: <size of object in bytes>,
`"text"`: <indexed text of object>,
`"source"`: <source document for object (what is actually stored in ElasticSeach)>,
`"time"`: <timestamp for operation>,
}...]
```

## Bucket.put\_file(self, key, path) <a href="#bucket.put_file" id="bucket.put_file"></a>

Stores file at path to key in bucket.

**Arguments**

* **key(str)**: key in bucket to store file at
* **path(str)**: string representing local path to file

**Returns**

None

**Raises**

* if no file exists at path
* if copy fails

## Bucket.put\_dir(self, key, directory) <a href="#bucket.put_dir" id="bucket.put_dir"></a>

Stores all files in the `directory` under the prefix `key`.

**Arguments**

* **key(str)**: prefix to store files under in bucket
* **directory(str)**: path to directory to grab files from

**Returns**

None

**Raises**

* if writing to bucket fails

## Bucket.keys(self) <a href="#bucket.keys" id="bucket.keys"></a>

Lists all keys in the bucket.

**Returns**

List of strings

## Bucket.delete(self, key) <a href="#bucket.delete" id="bucket.delete"></a>

Deletes a key from the bucket.

**Arguments**

* **key(str)**: key to delete

**Returns**

None

**Raises**

* if delete fails

## Bucket.delete\_dir(self, path) <a href="#bucket.delete_dir" id="bucket.delete_dir"></a>

Delete a directory and all of its contents from the bucket.

**Arguments**

* **path (str)**: path to the directory to delete

## Bucket.ls(self, path=None, recursive=False) <a href="#bucket.ls" id="bucket.ls"></a>

List data from the specified path.

**Arguments**

* **path (str)**: bucket path to list
* **recursive (bool)**: show subdirectories and their contents as well

**Returns**

`list`: Return value structure has not yet been permanently decided Currently, it's a `tuple` of `list` objects, containing the `following`: (directory info, file/object info, delete markers).

## Bucket.fetch(self, key, path) <a href="#bucket.fetch" id="bucket.fetch"></a>

Fetches file (or files) at `key` to `path`.

If `key` ends in '/', then all files with the prefix `key` will match and will be stored in a directory at `path`.

Otherwise, only one file will be fetched and it will be stored at `path`.

**Arguments**

* **key(str)**: key in bucket to fetch
* **path(str)**: path in local filesystem to store file or files fetched

**Returns**

None

**Raises**

* if path doesn't exist
* if download fails

## Bucket.select(self, key, query, raw=False) <a href="#bucket.select" id="bucket.select"></a>

Selects data from an S3 object.

**Arguments**

* **key(str)**: key to query in bucket
* **query(str)**: query to execute (SQL by default)
* **query\_type(str)**: other query type accepted by S3 service
* **raw(bool)**: return the raw (but parsed) response

**Returns**

`pandas.DataFrame`: results of query


# CLI, environment

## `catalog`

```
usage: quilt3 catalog [-h] [--detailed_help] [--host HOST] [--port PORT]
                      [--no-browser]
                      [navigation_target]

Run Quilt catalog locally

positional arguments:
  navigation_target  Which page in the local catalog to open. Leave blank to
                     go to the catalog landing page, pass in an s3 url (e.g.
                     's3://bucket/myfile.txt') to go to file viewer, or pass
                     in a package name in the form 'BUCKET:USER/PKG' to go to
                     the package viewer.

optional arguments:
  -h, --help         show this help message and exit
  --detailed_help    Display detailed information about this command and then
                     exit
  --host HOST        Bind socket to this host
  --port PORT        Bind to a socket with this port
  --no-browser       Don't open catalog in a browser after startup
```

Run the Quilt catalog on your machine. Running `quilt3 catalog` launches a Python webserver on your local machine that serves a catalog web app and provides required backend services using temporary AWS credentials. Temporary credentials are derived from your default AWS credentials (or active `AWS_PROFILE`) using `boto3.sts.get_session_token`. For more details about configuring and using AWS credentials in `boto3`, see the AWS documentation: <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html>

#### Previewing files in S3

The Quilt catalog allows users to preview files in S3 by downloading and processing/converting them inside the Python webserver running on local machine. Neither your AWS credentials nor data requested goes through any third-party cloud services aside of S3.

## `config`

```
usage: quilt3 config [-h] [--set KEY=VALUE [KEY=VALUE ...]] [catalog_url]

Configure Quilt

positional arguments:
  catalog_url           URL of catalog to config with, or empty string to
                        reset the config

optional arguments:
  -h, --help            show this help message and exit
  --set KEY=VALUE [KEY=VALUE ...]
                        Set a number of key-value pairs for config_values(do
                        not put spaces before or after the = sign). If a value
                        contains spaces, you should define it with double
                        quotes: foo="this is a sentence". Note that values are
                        always treated as strings.
```

## `config-default-remote-registry`

```
usage: quilt3 config-default-remote-registry [-h] default_remote_registry

Configure default remote registry for Quilt

positional arguments:
  default_remote_registry
                        The default remote registry to use, e.g. s3://quilt-ml

optional arguments:
  -h, --help            show this help message and exit
```

## `disable-telemetry`

```
usage: quilt3 disable-telemetry [-h]

Disable anonymous usage metrics

optional arguments:
  -h, --help  show this help message and exit
```

## `install`

```
usage: quilt3 install [-h] [--registry REGISTRY] [--top-hash TOP_HASH]
                      [--dest DEST] [--dest-registry DEST_REGISTRY]
                      [--path PATH]
                      name

Install a package

positional arguments:
  name                  Name of package, in the USER/PKG format

optional arguments:
  -h, --help            show this help message and exit
  --registry REGISTRY   Registry where package is located, usually s3://MY-
                        BUCKET. Defaults to the default remote registry.
  --top-hash TOP_HASH   Hash of package to install. Defaults to latest.
  --dest DEST           Local path to download files to.
  --dest-registry DEST_REGISTRY
                        Registry to install package to. Defaults to local
                        registry.
  --path PATH           If specified, downloads only PATH or its children.
```

## `list-packages`

```
usage: quilt3 list-packages [-h] registry

List all packages in a registry

positional arguments:
  registry    Registry for packages, e.g. s3://quilt-example

optional arguments:
  -h, --help  show this help message and exit
```

## `login`

```
usage: quilt3 login [-h]

Log in to configured Quilt server

optional arguments:
  -h, --help  show this help message and exit
```

## `logout`

```
usage: quilt3 logout [-h]

Log out of current Quilt server

optional arguments:
  -h, --help  show this help message and exit
```

## `push`

```
usage: quilt3 push --dir DIR [-h] [--registry REGISTRY] [--dest DEST]
                   [--message MESSAGE] [--meta META] [--workflow WORKFLOW]
                   [--force]
                   name

Pushes the new package to the remote registry

positional arguments:
  name                 Name of package, in the USER/PKG format

required arguments:
  --dir DIR            Directory to add to the new package

optional arguments:
  -h, --help           show this help message and exit
  --registry REGISTRY  Registry where to create the new package. Defaults to
                       the default remote registry.
  --dest DEST          Where to copy the objects in the package
  --message MESSAGE    The commit message for the new package
  --meta META          Sets package-level metadata. Format: A json string with
                       keys in double quotes '{"key": "value"}'
  --workflow WORKFLOW  Workflow ID or empty string to skip workflow
                       validation. If not specified, the default workflow will
                       be used.
  --force              Skip the parent top hash check and create a new
                       revision even if your local state is behind the remote
                       registry.
```

## `verify`

```
usage: quilt3 verify [-h] --registry REGISTRY --top-hash TOP_HASH --dir DIR
                     [--extra-files-ok]
                     name

Verify that package contents matches a given directory

positional arguments:
  name                 Name of package, in the USER/PKG format

optional arguments:
  -h, --help           show this help message and exit
  --registry REGISTRY  Registry where package is located, usually s3://MY-
                       BUCKET
  --top-hash TOP_HASH  Hash of package to verify
  --dir DIR            Directory to verify
  --extra-files-ok     Whether extra files in the directory should cause a
                       failure
```

## Environment variables

### `QUILT_DISABLE_CACHE`

Turn off cache. Defaults to `False`.

```
$ export QUILT_DISABLE_CACHE=true
```

### `QUILT_DISABLE_USAGE_METRICS`

Disable anonymous usage collection. Defaults to `False`

```
$ export QUILT_DISABLE_USAGE_METRICS=true
```

### `QUILT_MANIFEST_MAX_RECORD_SIZE`

Maximum size of a record in package manifest. **Setting this variable is strongly discouraged.** Defaults to `1_000_000`.

### `QUILT_MINIMIZE_STDOUT`

Turn off TQDM progress bars for log files. Defaults to `False`

```
$ export QUILT_MINIMIZE_STDOUT=true
```

### `QUILT_TRANSFER_MAX_CONCURRENCY`

Number of threads for file transfers. Defaults to `10`.

This variable could be tried for improving file transfer rate. The optimal value depends on network bandwidth, CPU performance, file sizes, etc.

```
$ export QUILT_TRANSFER_MAX_CONCURRENCY=20
```

### `XDG_*`

Quilt uses appdirs for Python to determine where to write data. You can therefore override the following path constants with environment variables using the XDG standard (see [appdirs docs](https://pypi.org/project/appdirs/)).

For instance, AWS Lambda requires the user to use `tmp/*` as the scratch directory. You can override `quilt3.util.CACHE_PATH`, so that `quilt3 install` will succeed in Lambda, by setting the `XDG_CACHE_HOME` environment variable.

## Constants (see [util.py](https://github.com/quiltdata/quilt/blob/master/api/python/quilt3/util.py) for more)

* `APP_AUTHOR`
* `APP_NAME`
* `BASE_DIR` - Base directory of the application
* `BASE_PATH` - Base pathlib path for the application directory
* `CACHE_PATH` - Pathlib path for the user cache directory
* `CONFIG_PATH` - Base pathlib path for the application configuration file
* `OPEN_DATA_URL` - Application data url
* `PACKAGE_NAME_FORMAT` - Regex for legal package names
* `TEMPFILE_DIR_PATH` - Base pathlib path for the application `tempfiles`


# Known limitations

By design, Quilt is backed by Amazon S3 and scales to billions of objects and petabytes of data. The underlying limitations of S3 apply.

The Quilt catalog can browse packages and S3 buckets of any size.

## Catalog push

To ensure usability and quick package pushes, the Quilt web catalog imposes the following limits on pushes. These limits do not apply to the `quilt3` Python API.

| Dimension                                                  | Max    |
| ---------------------------------------------------------- | ------ |
| Package manifest size (metadata)                           | 100 MB |
| Package size (data; via promotion or from an S3 directory) | 100 GB |
| Total size of uploaded files (soft limit)                  | 20 GB  |
| Total size of files from S3 (soft limit)                   | 50 GB  |
| Maximum file size                                          | 10 GB  |
| Maximum number of files per push (soft limit)              | 1,000  |
| Maximum number of files per push (hard limit)              | 5,000  |

## API

As of this writing, with sufficient client-side memory, you can comfortable scale Quilt packages to at least one million objects per package, with no practical limit on object size (save S3's 5 TB per object limit). A fast network, or better yet an AWS compute instance in the same region as your Quilt S3 buckets, is recommended.

## Metadata

See [Metadata](/version-5.0.x/catalog/metadata).


# Admin UI

The Quilt catalog includes an admin panel that allows you to manage users and buckets in Quilt, as well as customize the Quilt catalog. You can access the panel via a dropdown menu under username in the navbar.

![](/files/FX00ah5ftskiBSr0NJs0)

The admin page is only accessible to designated administrators. The first admin is set during the CloudFormation installation. Subsequent admins may be designated through the panel. Only admins may create other admins.

Quilt requires at least one admin account per stack.

### Users and roles

This section provides comprehensive access control management for the follwing functions:

* Create/Delete user
* De/activate user
* De/admin user
* Assign roles to users

![](/files/KGAZ2nHeKTp9fbiXywy0)

You may invite new users to your Quilt stack by clicking the + button, upper right.

You may create custom roles for different groups of users. With the exception of administrators, users of managed roles are only aware of the buckets that they are permitted to read. All other stack buckets are invisible to users of managed roles.

![](/files/PRBEqgHRJCk0FSUjPd6v)

You must select the default role for all new users, else they will not be able to sign in to the Quilt catalog. The default role is shown in bold.

![](/files/JAvn4j03bbA55znvBRCW)

### Buckets

Here you can add or remove buckets from Quilt and configure bucket indexing and display settings.

![](/files/MptUMakFlPN0v10BkHN1)

![](/files/uFSluuiBaEzSv7eV9hY8)

#### S3 events

By default, when you add a bucket to the Quilt stack one of two things will happen:

1. If there is no existing bucket notification, Quilt will attempt to add a new notification
2. If there is an existing bucket notification, Quilt will use the existing notification if and only if it supports the required events (object creation and deletion)

If either of the above conditions fails, Quilt will not add the bucket in question.

See [S3 Events, EventBridge](https://github.com/quiltdata/quilt/blob/version-5.0.x/docs/Catalog/EventBridge.md) for more.

### Settings

This section allows you to customize your Quilt catalog, including custom links in the navbar.

![](/files/lIcUWnqIhRROSD7cwCAI)

### Further settings

See [Prefences](/version-5.0.x/catalog/preferences) for further control over the catalog user interface.


# Configuration

You can use your Quilt catalog's configuration file to show or hide certain tabs and buttons in the Quilt catalog. This gives you finer-grained control over how users interact with the Quilt catalog. The catalog config file is per-bucket. The file's path is `s3://BUCKET/.quilt/catalog/config.yaml`.

If there is no config.yaml, or your config.yaml file does not override the `ui` key, then the following defaults are used:

```
ui:
  nav:
    files: True
    packages: True
    queries: True
  actions:
    copyPackage: True
    createPackage: True
    deleteRevision: False
    revisePackage: True
  sourceBuckets:
    s3://BUCKET_1: {}
    s3://BUCKET_2: {}
```

## Properties

* `ui.nav.files: False` - hide Files tab
* `ui.nav.packages: False` - hide Packages tab
* `ui.nav.queries: False` - hide Queries tab
* `ui.actions.copyPackage: False` - hide buttons to push packages across buckets
* `ui.actions.createPackage: False` - hide buttons to create packages via drag-and-drop or from folders in S3
* `ui.actions.deleteRevision: True` - show buttons to delete package revision
* `ui.actions.revisePackage: False` - hide the button to revise packages
* `ui.sourceBuckets` - a dictionary of S3 bucket names that map to an empty object reserved for future enhancements; buckets in this dictionary are the ones offered when the user clicks Revise Package > Add files from Bucket; if the dictionary is not set or is empty the feature "Add files from Bucket" is disabled
* `ui.defaultSourceBucket` - source bucket from `ui.sourceBuckets` that is selected by default; if it doesn't match any bucket then it's ignored


# Embed

The Quilt catalog has limited-feature S3 browser called Embed, an embeddable iframe. Embed is served off the main catalog under `/__embed` route.

![](/files/mb3gqyW9CkAfxs2hV3HM)

### Usage

Here's a sample code snippet in plain JavaScript + React that outlines the basic use cases:

```js
import * as React from 'react'

const EMBED_ORIGIN = 'https://my-quilt-catalog'
const PARENT_ORIGIN = window.location.origin
const EVENT_SOURCE = 'quilt-embed'

const mkNonce = () => `${Math.random()}`.slice(2)

function Embed() {
  const iframeRef = React.useRef(null)

  const [nonce, setNonce] = React.useState(mkNonce)

  // nonce is used to identify "our" Embed instance and make sure we're receiving messages from the same instance
  // origin must be sent as a query parameter to enable cross-origin message passing from Embed to the parent
  const src = `${EMBED_ORIGIN}/__embed?nonce=${nonce}&origin=${encodeURIComponent(PARENT_ORIGIN)}`

  const postMessage = React.useCallback(
    // function for sending messages to the Embed (it will only handle messages sent by the window that opened it aka parent)
    (msg) => {
      if (!iframeRef.current) return
      // origin must be sent as a second parameter to enable cross-origin message passing
      iframeRef.current.contentWindow.postMessage(msg, EMBED_ORIGIN)
    },
    [iframeRef],
  )

  const initialize = React.useCallback(() => {
    // see the `init` command reference in the API section below for details
    postMessage({
      type: 'init',
      bucket: 'your-bucket-here',
      path: 'path-to-object-or-prefix',
      // deep linking can be implemented by storing Embed's route and then initializing it with this stored route
      route: location.search.route || undefined,

      // e.g. { provider, token } for SSO or { password, username } (which doesn't seem like a right choice in most cases)
      // getting credentials is your app's responsibility
      credentials: { provider: 'okta', token: 'my token' },

      theme: {
        palette: {
          primary: {
            main: '#282b50',
          },
          secondary: {
            main: '#339933',
          },
        },
        typography: {
          fontFamily: '"Comic Sans MS", "Comic Sans", cursive',
        },
      },

      overrides: {
        s3ObjectLink: {
          href: 'https://my-app/s3-browser?route=<%= encodeURIComponent(url) %>',
          // notification shown after copying href to the clipboard (if `emit` is not set to "override")
          emit: 'override',
        },
      },

      // arbitrary CSS files can be injected to further customize look and feel and/or layout
      css: [
        'https://my-cdn.com/my-custom-styles-1.css',
        'https://my-other-host.com/my-custom-styles-2.css',
      ],
    })
  }, [postMessage])

  const navigate = React.useCallback((route) => {
    // command the Embed to navigate to an arbitrary route
    postMessage({ type: 'navigate', route })
  }, [postMessage])

  const reloadIframe = React.useCallback(() => {
    // reload the iframe by generating a new nonce (and therefore changing `src` computed value)
    setNonce(mkNonce)
  }, [setNonce])

  const handleMessage = React.useCallback(
    (e) => {
      if (
        // ignore messages from other windows
        e.source !== iframeRef.current.contentWindow ||
        // ensure origin is what we expect it to be (user has not navigated away)
        e.origin !== EMBED_ORIGIN ||
        // ensure the message has expected format (.source set to 'quilt-embed')
        e.data.source !== EVENT_SOURCE ||
        // ensure this is "our" instance by comparing nonce passed to the iframe
        // via query string to the nonce passed back by the iframe
        nonce !== e.data.nonce
      ) {
        return
      }
      // handle messages from the Embed
      switch (e.data.type) {
        case 'error':
          console.error(e.data.message, e.data)
          return
        case 'ready':
          initialize()
          return
        case 'navigate':
          // store Embed state (route), e.g. update our URL to store the embed route in a query paramter or smth
          console.log('embed navigating to', e.data.route)
          return
        case 's3ObjectLink':
          // construct a custom link using e.data and copy it to clipboard or perform any other relevant action
          console.log('s3 object link clicked', e.data)
          return
      }
    },
    [iframeRef, nonce, initialize],
  )

  React.useEffect(() => {
    // subscribe to messages from Embed
    window.addEventListener('message', handleMessage)
    return () => {
      window.removeEventListener('message', handleMessage)
    }
  }, [handleMessage])

  return (
    <iframe
      ref={iframeRef}
      src={src}
      width="900"
      height="600"
      // other things like styling and stuff
    />
  )
}
```

### API reference

#### URL and query parameters

The Embed is served off the main catalog server under `/__embed` route which takes two optional query parameters:

`nonce` (any unique string) is used to identify "our" Embed instance and make sure we're receiving messages from that same instance.

`origin` must be sent to enable cross-origin message passing from Embed to the parent.

#### Commands to Embed

Embed accepts commands from the parent via [`postMessage` API](https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage). Command message data format:

```ts
interface Command {
  type: string
  // ...parameters
}
```

Available commands are listed below.

**`init`**

Initialize the Embed. This command must be sent after the Embed is ready (see `ready` message reference for details). Supported SSO providers are listed in the [Technical Reference](/version-5.0.x/advanced/technical-reference#single-sign-on-sso).

```ts
interface InitCommand extends Command {
  type: 'init'

  // Bucket name, e.g. 'my-bucket'
  bucket?: string
  // Path to object or prefix in the given bucket, e.g. 'some-prefix/some-object.csv'
  path?: string

  // Initial route Embed will navigate to, e.g. '/b/my-bucket/tree/some/path',
  // takes precedence over bucket / path
  route?: string

  // Embed accepts any credentials supported by the Quilt authentication endpoint,
  // e.g. { provider, token } for [SSO](../technical-reference.md#single-sign-on-sso)
  // or { password, username } (which doesn't seem like a right choice in most cases tho).
  // Getting credentials is your app's responsibility.
  credentials: { provider: string; token: string } | { username: string; password: string }

  // Embed can be "scoped" to a prefix, meaning that prefix will be a virtual "root" for the object browser,
  // but only for display purposes (i.e. formatting paths / rendering breadcrumbs),
  // it won't prevent navigating to the paths outside the scope if navigated directly
  // via a command (navigate or init) or a link, so it's not to be considered a security measure.
  scope?: string

  // Look and feel of the Embed can be customzied by providing theme overrides,
  // see [MUI theming reference](https://material-ui.com/customization/theming/)
  // and [Quilt theme construction code](https://github.com/quiltdata/quilt/blob/master/catalog/app/constants/style.js#L145)
  // for details.
  theme?: MUI.ThemeOptions // https://github.com/mui-org/material-ui/blob/v4.12.3/packages/material-ui/src/styles/createTheme.d.ts#L15

  // Some aspects of the UI can be overriden:
  overrides?: {
    // This prop is responsible for customizing the display and behaviour
    // of the "link" button in the object revision list menu
    s3ObjectLink?: {
      // Title of the link element
      title?: string
      // Link [template](https://lodash.com/docs/4.17.15#template).
      // Template context:
      //   url: string -- url / route of the object version in the context of the Embed
      //   s3HttpsUri: string -- HTTPS URI of the object version, e.g. https://my-bucket.s3.amazonaws.com/${key}?versionId=${version} (with properly encoded key)
      //   bucket: string -- current bucket
      //   key: string -- key of the browsed object
      //   version: string -- object version id
      // Example: 'https://my-app/s3-browser?route=<%= encodeURIComponent(url) %>'
      href?: string
      // Notification shown after copying href to the clipboard (if `emit` is not set to "override")
      notification?: string
      // Set to "notify" or "override" to enable Embed sending "s3ObjectLink" messages
      // (otherwise those messages won't be sent).
      // Set to "override" to disable default action on click (copying href to clipboard).
      emit?: 'notify' | 'override' | null
    }
  }

  // List of CSS URLs to be injected (to further customize look and feel and/or layout)
  // Example:
  // [
  //   'https://my-cdn.com/my-custom-styles-1.css',
  //   'https://my-other-host.com/my-custom-styles-2.css',
  // ],
  css?: string[]
}
```

**`navigate`**

Navigate to the given route.

```ts
interface NavigateCommand extends Command {
  type: 'navigate'
  // Route to navigate to, e.g. '/b/my-bucket/tree/some/path'
  route: string
}
```

#### Messages from Embed

Embed sends messages to its parent via [`postMessage` API](https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage). Message data format:

```ts
interface Message {
  source: 'quilt-embed'
  nonce: string
  type: string
  // ...parameters
}
```

Available messages are listed below.

**`ready`**

Sent when the Embed is done loading and ready to receive commands (waiting for `init` command).

```ts
interface ReadyMessage extends Message {
  type: 'ready'
}
```

**`error`**

Sent when an error occurs.

```ts
interface ErrorMessage extends Message {
  type: 'error'
  message: string
  credentials?: object // for authentication error
  init?: object // for initialization error
  data?: object // for navigation error
}
```

**`navigate`**

Sent when the Embed navigates to a new route. Useful for syncing Embed state with the parent app state.

```ts
interface NavigateMessage extends Message {
  type: 'navigate'
  route: string
  action: 'PUSH' | 'POP' | 'REPLACE'
}
```

**`s3ObjectLink`**

Enabled only when `overrides.s3ObjectLink.emit` parameter is set during initialization. Sent when the link button in the object version menu is clicked.

![](/files/575LiReN6BSLPviRbqdW)

```ts
interface S3ObjectLinkMessage extends Message {
  type: 's3ObjectLink'
  // URL aka route of the object version in the context of the Embed
  url: string
  // HTTPS URI of the object version, e.g. https://my-bucket.s3.amazonaws.com/${key}?versionId=${version} (with properly encoded key)
  s3HttpsUri: string
  // Bucket the object resides in
  bucket: string
  // Object's key
  key: string
  // Object's version ID
  version: string
}
```

### Testing and debugging

The catalog's `/__embed-debug` route is a simple driver for testing Embed:

![](/files/IvXOSlIDd4mqxa6slpuY)

`__embed-debug` is useful for trying different parameters and inspecting messages passed to and from the Embed.

Its main components are:

1. Inputs for `init` parameters and button for sending the `init` command.
2. "Navigate to" button and route input for sending the `navigate` command.
3. Embed window.
4. Message log.


# Metadata for teams

### Why metadata matters

Data without labels and documentation quickly become meaningless. In Quilt, metadata are represented as dictionaries that can refer to specific objects or entire packages.

Metadata solves the following problems:

* *Collaboration* — Metadata are the clues that enable developers, non-developers, and code to create a shared understanding and vocabulary
* *Discoverability* — Quilt package-level metadata are searchable via ElasticSearch; package-level and object-level metadata are queryable via AWS Athena
* *Trust* — Quilt metadata are screened against JSON schemas that you define to ensure that annotations are complete and type-safe. (See [Workflows](/version-5.0.x/advanced/workflows))
* *Understandability* - Metadata are a love letter to the future; with metadata in hand, users can better understand what data mean, where they came from, and how they might be used in the future
* *Longitudinal analysis* — Need to cut across packages and isolate data sets based on varying dimensions? Metadata makes this possible.

#### Metadata in Quilt

Quilt packages contain one of two types of metadata:

* *Object-level metadata* for each object or entry in the package
* *Package-level metadata* for each revision of the package

#### The Goldilocks problem

If you require your users to input too much metadata, they'll avoid your system. Too little metadata and it's hard to understand or trust your collection. Our rule of thumbs are:

> Minimize human-entered metadata to less than a dozen fields

> Maximize machine-entered data to capture any facts or dimensions that might be useful in the future: date, author, etc. (by default, Quilt automatically captures metadata like file size and SHA-256 hash)

### Entering metadata

When you create or revise a package in Quilt, you can edit the package-level metadata.

![](/files/SZdi6db9AYEt7S2dirEq)

Metadata can be entered by hand, or you can drag and drop one of the following file types on the Metadata section:

* CSV
* XLS
* XLSX
* XLSM
* ODS
* FODS

The keys of your dataset may be represented as either column headers, or as the values of a single column. Quilt picks the orientation that best overlaps with the workflow schema, be that orientation row-major or column-major.

#### Spreadsheet example

Suppose your spreadsheet looks like this:

| Metal   | Color  | Price |
| ------- | ------ | ----- |
| gold    | yellow | 1700  |
| silver  | gray   | 25    |
| rhodium | gray   | 25000 |

When you drag and drop the spreadsheet, Quilt converts it:

```json
{
  "Metal": ["gold", "silver", "rhodium"],
  "Color": ["yellow", "gray", "gray"],
  "Price": [1700, 25, 25000]
}
```

> Note: empty cells have a value of `null`

#### Supported types

Metadata may contain strings, numbers, objects, booleans, and dates. Dates are converted to the YYYY-MM-DD format.

If your workflow Schema has `{ type: "array" }` for a cell, Quilt converts this string to an array by splitting the cell on comma. If your input file contains JSON fragments, such as `"{"a": 1, "b", 2}"`, Quilt will convert those strings to objects.

### Limitations

Quilt recommends, and the APIs will soon enforce, that users limit each instance of package-level and object-level metadata to 1MB or less so that your package works well with S3 Select (1MiB row limit) and AWS Athena (32MB row limit), which are by the Quilt backend.


# Preview

The Quilt catalog renders previews of the following file types. Whenever possible, Quilt streams the smallest possible subset of the data needed to generate the preview.

Previews are supported for uncompressed files as well as for gzip archives (.gz).

### Plain text previews

Quilt can display any plaintext file format, including the following.

* Most programming languages, with syntax highlighting (.cpp, .json, .js, .py, .sh, .sql, etc.)
* Biological file formats (.bed, .cef, .gff, .fasta, .fastq, .sam, .pdbqt, .vcf, etc.)
* Text files (.csv, .md, .readme, .tsv, .txt, etc.)

### Binary and special file format previews

* Excel (.xls, .xlsx)
* FCS Flow Cytometry files (.fcs)
* Images (.gif, .jpg, .png, .tif, .tiff, etc.)
* Media (.mp4, .webm, .flac, .m2t, .mp3, .mp4, .ogg, .ts, .tsa, .tsv, .wav)
* .ipynb (Jupyter and Voila dashboards)
* .parquet


# Search & query

Out of the box, Quilt provides support for queries in the ElasticSearch DSL, as well as SQL queries in Athena (details forthcoming).

### ElasticSearch

The objects in S3 buckets connected to Quilt are synchronized to an ElasticSearch cluster, which powers Quilt's search features. For custom queries, you can use the Queries tab in the Quilt catalog to directly query ElasticSearch cluster.

Quilt uses ElasticsSearch 6.7 ([docs](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/index.html)).

#### Indexing

Quilt maintains a near-realtime index of the objects in your S3 bucket in ElasticSearch. Each bucket corresponds to one or more ElasticSearch indexes. As objects are mutated in S3, Quilt uses an event-driven system (via SNS and SQS) to update ElasticSearch.

There are two types of indexing in Quilt:

* *shallow* indexing includes object metadata (such as the file name and size)
* *deep* indexing includes object contents. Quilt supports deep indexing for the following file extensions:
  * .fcs (FlowJo)
  * .ipynb (Jupyter notebooks)
  * .parquet
  * .pdf
  * .html, .txt, .tsv, .csv, .md (plus many other plain-text formats)
  * .xls, .xlsx

#### Queries

![](/files/mE0xsjTysfiqUMLkWjMa)

Quilt ElasticSearch queries support the following keys:

* `index` — comma-separated list of indexes to search ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/multi-index.html))
* `filter_path` — to reducing response nesting, ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/common-options.html#common-options-response-filtering))
* `_source` — boolean that adds or removes the `_source` field, or a list of fields to return ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-request-source-filtering.html))
* `size` — limits the number of hits ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-uri-request.html))
* `from` — starting offset for pagination ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-uri-request.html))
* `body` — the search query body as a JSON dictionary ([learn more](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-request-body.html))

**Saved queries**

You can provide pre-canned queries for your users by providing a configuration file at `s3://YOUR_BUCKET/.quilt/queries/config.yaml`:

```yaml
version: "1"
queries:
  query-1:
    name: My first query
    description: Optional description
    url: s3://BUCKET/.quilt/queries/query-1.json
  query-2:
    name: Second query
    url: s3://BUCKET/.quilt/queries/query-2.json
```

The Quilt catalog displays your saved queries in a drop-down for your users to select, edit, and execute.


# Visualization & dashboards

Quilt packages are not only reproducible units of data and metadata, but units of *reporting*. You can use the following features to include interactive visualizations and light applications inside of packages.

Importantly, relative references to data are resolved relative to the parent package. This means that all of your reports are backed by immutable, versioned data, providing a common frame of reference that is lacking in BI applications that read from fast-moving databases and file systems.

In addition to rendering a wide variety of images, binary files, and text files, the Quilt catalog supports the following libraries for visualization and exploration:

* [Vega](https://vega.github.io/vega/),
* [Vega-lite](https://vega.github.io/vega-lite/),
* [ECharts](https://echarts.apache.org/en/index.html),
* [Voila](https://github.com/voila-dashboards/voila) (Developer preview).
* [Perspective](https://perspective.finos.org),

The above systems provide you with hundreds of charts out of the box.

## `quilt_summarize.json`

`quilt_summarize.json` is a configuration file that renders one or more dashboard elements in both Bucket view and Packages view. The contents of `quilt_summarize.json` are a JSON array of files that you wish to preview in the catalog. Each file may be represented as a string or, if you wish to provide more configuration, as an object.

The simplest summary is a list of relative paths to files that you wish to preview:

```json
// quilt_summarize.json
[
  "file1.json",
  "file2.csv",
  "file3.ipynb"
]
```

By default, each list element renders in its own row.

![](/files/dGoK1FDvndD6KPMUJYos)

For multi-column layouts, you can provide an array instead of a string for a given row:

```json
// quilt_summarize.json
[
  "file1.json",
  [{
    "path": "file2.csv",
    "width": "200px"
  }, {
    "path": "file3.ipynb",
    "title": "Scientific notebook",
    "description": "[See docs](https://docs.com)"
  }]
]
```

![](/files/IFV36tTEjJEexrUJnNVT)

Each element of an array in `quilt_summarize.json` can either be a path string or an object with one or more of the following properties:

* `path` - file path relative to `quilt_summarize.json`
* `title` - title rendered instead of file path
* `description` - description in markdown format
* `expand` - Display the file (`true`) or display a preview in an expandable box (`false`, default)
* `width` - column width either in pixels or ratio (default is ratio `1`)
* `types` - a list of rendering types; currently only singleton list values are supported:
  * `["echarts"]` to render JSON as an EChart
  * `["voila"]` to render a Jupyter notebook as an interactive Voila dashboard

If you need to control the height of an element (useful for Voila dashboards), use the following extended syntax:

```json
// quilt_summarize.json
[
  {
    "path": "file1.json",
    "types": [
      {
        "name": "echarts",
        "style": { "height": "1000px" }
      }
    ]
  }
]
```

At present `height` is the only supported `style` element.

## Vega and Vega-lite

The Quilt catalog uses vega-embed, vega, and vega-lite. See [package.json](https://github.com/quiltdata/quilt/blob/master/catalog/package.json) for specific library versions and compatibility.

In order to display a Vega or Vega-Lite visualization, simply reference a JSON file with a library-compatible schema in a JSON file as follows in your `quilt_summarize.json` file:

```json
[
    "relative/path/to/my/vega.json",
    "optionally/some/other/file.csv"
]
```

For both Vega and Vega Lite you may specify relative paths to package files as data sources and the Quilt catalog will resolve the same.

### Altair

The easiest way to create Vega-lite visualizations for Quilt packages is with [Altair](https://altair-viz.github.io/).

Here's a simple example:

```python
import altair as alt

df = pd.read_csv("file.csv")

alt.Chart(df).mark_area(
    color="gray",
    opacity=.2
).encode(
    x="score:Q",
    y='density:Q',
    tooltip=['count(score):Q']
).save("vega.json")
```

## ECharts

To render an EChart, you provide a JSON file (a dictionary that specifies the ECharts [option parameter](https://echarts.apache.org/en/option.html)) and you set the `"types"` property to `[ "echarts" ]`.

```json
// quilt_summarize.json
[
  {
    "path": "echarts-option-file.json",
    "title": "Awesome line chart",
    "types": ["echarts"]
  }
]
```

### Example

The following example is a [simple line chart from the ECharts documentation](https://echarts.apache.org/examples/en/editor.html?c=line-simple).

```json
// echarts.json
{
  "dataset": {
    "source": [
      ["Mon", 150],
      ["Tue", 230],
      ["Wed", 224],
      ["Thu", 218],
      ["Fri", 135],
      ["Sat", 147],
      ["Sun", 250]
    ]
  },
  "xAxis": {
    "type": "category"
  },
  "yAxis": {
    "type": "value"
  },
  "series": [
    {
      "type": "line"
    }
  ]
}
```

![](/files/IlUTUj4eJrn0Tcwb9io9)

### External files

As with Vega, you can provide either a relative path or URL to the dataset file.

```json
// echarts.json
{
  "dataset": {
    "source": "./dataset.csv"
  },
  "xAxis": {
    "type": "category"
  },
  "yAxis": {
    "type": "value"
  },
  "series": [
    {
      "type": "line"
    }
  ]
}
```

Relative paths are resolved relative to your echarts.json file and relative to the parent package.

### Limitations

At present, ECharts in Quilt does not support custom JavaScript. You are therefore limited to JSON types (numbers, strings, objects, arrays, etc.). Functions like [`symbolSize`](https://echarts.apache.org/en/option.html#series-scatter.symbolSize) are not available.

## Voila

*This feature is a Developer preview, details are subject to change.*

Enterprise deployments of Quilt support interactive Jupyter notebooks with [Voilà](https://github.com/voila-dashboards/voila).

In brief, a Voila dashboard version of your notebook will display all of the output cells and none of the input cells from the underlying notebook. This enables you to create interactive, Jupyter-driven apps for your Quilt catalog users.

The Voila libraries execute a remote Jupyter Kernel and stream the results to the browser with tornado. Jupyter kernels run on a single EC2 instance (`t3.small` by default) in Linux containers that have network access but do not have access to persistent storage. The catalog users's AWS credentials are passed to Jupyter kernel as [environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list).

## Environment variables

When you have a Voila dashboard inside of a Quilt package, you may wish to reference files *in the current package revision*. The Quilt catalog sets the following environment variables and passes them to the Voila kernel:

* `QUILT_PKG_BUCKET`
* `QUILT_PKG_NAME`
* `QUILT_PKG_TOP_HASH`

You can access these variables in Python and browse the package:

```python
import io
import os

import pandas as pd
import quilt3 as q3

bucket = os.environ.get("QUILT_PKG_BUCKET")
handle = os.environ.get("QUILT_PKG_NAME")
top_hash = os.environ.get("QUILT_PKG_TOP_HASH")

pkg = q3.browse(handle, registry=f"s3://{bucket}", top_hash=top_hash)
# Read data.csv from the current package from Voila
df = pkg["data.csv"].deserialize()
```

### Included Python packages

By default, Quilt Voila containers provide the following modules:

```
altair
bqplot
ipykernel
ipyvolume
ipywidgets
pandas
perspective-python
PyYAML
quilt3
scipy
```

## Perspective

Quilt renders tabular data formats into a Perspective Datagrid, including the following file extensions: .csv, .xls, .xlsx, .jsonl, .parquet, and .tsv.

For speed, Quilt loads a small preview of the rows stored in S3. You can click Load More to fetch up to about 6MB of zipped data. Beyond this size, click Download to see the entire file contents.

In order to open the analysis and visualization capabilities of Perspecitve, click the vertical ellipsis, upper left. To open the controls by default, set the `config.settings` property in `quilt_summarize.json` as follows:

```json
// quilt_summarize.json
[
  {
    "path": "file1.csv",
    "types": [
      {
        "name": "perspective",
        "config": {
          "settings": true
        }
      }
    ]
  }
]
```

Besides `{ "settings": true }` you can provide any object previously saved from Perspective.

![](/files/xHhua9twwaam9e82fcqq)

All filters and columns will be restored:

```json
// quilt_summarize.json
[
  {
    "path": "file1.csv",
    "types": [
      {
        "name": "perspective",
        "config": {
          "columns": ["name", "value"],
          "group_by": ["value"],
          "settings": true
        }
      }
    ]
  }
]
```

Drag columns from the left to the top to sort, filter, and pivot. Use the menu upper left to try different visualizations. Use the controls along the bottom to download, copy, resize, and more.

![](/files/h4BFqlSLg1sAA9m5sKtJ)


# Local Development Mode

You can securely and privately run the Quilt catalog in "single-player mode" on your machine. `quilt3 catalog` launches a Python webserver and local services that communicate with S3 using temporary AWS credentials, derived from your default AWS credentials (or active `AWS_PROFILE`) with `boto3.sts.get_session_token`. Data and credentials remain local and private to your machine and AWS account.

For more details about configuring and using AWS credentials in `boto3`, see the [AWS documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

### Installation

```bash
$ pip install quilt3[catalog]
```

### Invocation

```bash
quilt3 catalog
```

See the [CLI API reference](/version-5.0.x/api-reference/cli#catalog) for details.


# Git-like operations for datasets and Jupyter notebooks

`quilt3` provides a simple command-line for versioning large datasets and storing them in Amazon S3. There are only two commands you need to know:

* `push` creates a new package revision in an S3 bucket that you designate
* `install` downloads data from a remote package to disk

## Why not use Git?

In short, neither Git nor Git LFS have the capacity or performance to function as a repository for data. S3, on the other hand, is widely used, fast, supports versioning, and currently stores some trillions of data objects.

Similar concerns apply when baking datasets into Docker containers: images bloat and slow container operations down.

## Pre-requisites

You will need either an AWS account, credentials, and an S3 bucket, OR a Quilt enterprise stack with at least one bucket. In order to read from and write to S3 with `quilt3`, you must first do one of the following:

* [Configure your AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
* OR, if and only if your company runs a Quilt enterprise stack, run the following:

  ```bash
  pip install quilt3
  quilt3 config https://yourquilt.yourcompany.com
  quilt3 login
  ```

## Install a package

A Quilt package contains any collection of data (usually as files), metadata, and documentation that you specify.

Let's get a data package from S3 and write it `quilt-hurdat/data` .

```bash
mkdir reef-check
cd reef-check
quilt3 install \
    "akarve/reef-check" \
    --registry s3://quilt-example \
    --dest .
```

Now you've got data in the current working directory.

```
ls
CA-06-california-counties.json	quilt_summarize.json  urchins-interactive.json
README.md			reef-check.ipynb      urchins2006-2019.parquet
```

## Creating your first package

Now let's imagine that we've modified this data locally. We save our Jupyter notebook and push the results back to Quilt:

```bash
# Be sure to substitute YOUR_NAME and YOUR_BUCKET with the desired strings
quilt3 push \
    YOUR_NAME/reef-check \
    --dir . \
    --registry s3://YOUR_BUCKET \
    --message "Initial commit of reef data"
```

Quilt will then print out something like the following:

```bash
Package YOUR_NAME/reef-check@ea334b7 pushed to s3://YOUR_BUCKET
Successfully pushed the new package to https://yourquilt.yourocmpany.com/b/YOUR_NAME/packages/akarve/reef-check
```

## List the packages in a bucket

```bash
quilt3 list-packages s3://YOUR_BUCKET
```

In the Quilt catalog, you will now see a new package revision, complete with a README, data grid preview, and an interactive visualization in Altair.

![](/files/s8PgIxiavt8qO9ara4AU) ![](/files/YmIF4lzaA5Ja9Mu7kggb)

You can see an example of this package live [here](https://open.quiltdata.com/b/quilt-example/packages/akarve/reef-check/tree/8a0d875d5f46355a770143d66c4a1cfea43c7ffa215db600b9ff88fafe687676/).

## Learn more

Those are the basics of reading and writing Quilt packages with the CLI. See the [CLI reference](/version-5.0.x/api-reference/cli) for more.


# Filtering a Package

`quilt3` provides a handful of functions for operating on the contents of a package in an entry-by-entry manner. These are helpful for performing more complicated parsing operations on the package:

```python
import quilt3

# create a package
p = (quilt3.Package()
        .set_dir("foo/", "foo/")
        .set("bar", "bar"))

# element-wise transform entries, outputting a list
# here "lk" is shorthand for "logical_key"
# and "entry" is the package entry
p.map(lambda lk, entry: entry)

# filter out entries not meeting certain criteria
p.filter(lambda lk, entry: 'cool' not in lk)
```

Notice that these functions operate over `(logical_key, entry)` tuples. Each `logical_key` is a string. Each `entry` is `PackageEntry` object, as would be returned if you slice to a leaf node of the package (e.g. `p['bar']`).


# .quiltignore

`.quiltignore` is a special file which, when included in a directory, can be used to filter out files that are included when you call `quilt3.Package.set_dir` from inside that directory:

```bash
$ ls
foo.txt         bar.txt         baz.txt
$ python -c "import quilt3; print(quilt3.Package().set_dir('/', './'))"
(local Package)
 └─foo.txt
 └─bar.txt
 └─baz.txt
$ echo foo.txt >> .quiltignore
$ ls -a
foo.txt         bar.txt         baz.txt
.quiltignore
$ python -c "import quilt3; print(quilt3.Package().set_dir('/', './'))"
(local Package)
 └─bar.txt
 └─baz.txt
 └─.quiltignore
```

This can be used to keep non-data files in the directory out of the data package. This is very useful when, for example, your data and your code live in the same directory.

The `.quiltignore` syntax is exactly the same as that of the familiar `.gitignore`. Refer to the [git documentation](https://git-scm.com/docs/gitignore) for instructions on how to use it.


# Materialization

### `build` versus `push`

`build` and `push` have the following behaviors:

* `build` calculates a tophash based on the package entry data, package entry metadata, and top-level package metadata, injects that into the package manifest, and stores the package manifest in the local registry.
* `push` uploads the entries in the package to a remote registry, performs the same tophash calculations, and stores a package manifest referencing the remote files in the remote registry.

Under the hood, there are just two differences between `push` and `build`:

* `push` targets a remote registry; `build` targets a local registry
* `push` copies package files; `build` leaves files where they are

### Materialization

The latter of these two differences is known as **materialization**, and it means that `push` creates **materialized packages**: packages which point solely to files located in an S3 bucket.

A materialized package is stronger than a unmaterialized (or "local") package because S3 guarantees that individual object versions are never lost or destroyed (assuming object versioning is enabled). Furthermore, only materialized packages may be browsed in the online Quilt catalog.

### Pushing unmaterialized packages

There are advanced use cases where automatically copying (potentially large) files is not the behavior you want.

To push an *umaterialized* file to a remote registry, provide `build` with a `registry`. For example:

```python
import quilt3
p = quilt3.Package().set("example.csv", "example.csv")
p.build("username/packagename", registry="s3://my-bucket")
```

Note that in this case it is up to you, the package author, to ensure that any local files in the package remain available and accessible to users.




---

[Next Page](/llms-full.txt/1)