LogoLogo
HomeGitHub RepoBook Demo
dev
dev
  • About Quilt
  • Architecture
  • Mental Model
  • Metadata Management
  • Metadata Workflows
  • Quilt Platform (Catalog) User
    • About the Catalog
    • Bucket Browsing
    • Document Previews
    • Embeddable iFrames
    • Packaging Engine
    • Query
    • Quilt+ URIs
    • Qurator Omni
    • Search
    • Visualization & Dashboards
    • Advanced
      • Athena
      • Elasticsearch
      • Removing Stacks
  • Quilt Platform Administrator
    • Admin Settings UI
    • Catalog Configuration
    • Cross-Account Access
    • Enterprise Installs
    • quilt3.admin Python API
    • Advanced
      • Package Events
      • Private Endpoints
      • Restrict Access by Bucket Prefix
      • S3 Events via EventBridge
      • SSO Permissions Mapping
      • Tabulator
      • Troubleshooting
        • SSO Redirect Loop
    • Best Practices
      • GxP for Security & Compliance
      • Organizing S3 Buckets
  • Quilt Python SDK
    • Installation
    • Quick Start
    • Editing a Package
    • Uploading a Package
    • Installing a Package
    • Getting Data from a Package
    • Example: Git-like Operations
    • API Reference
      • quilt3
      • quilt3.Package
      • quilt3.Bucket
      • quilt3.hooks
      • Local Catalog
      • CLI, Environment
      • Known Limitations
      • Custom SSL Certificates
    • Advanced
      • Browsing Buckets
      • Filtering a Package
      • .quiltignore
      • Manipulating Manifests
      • Materialization
      • S3 Select
    • More
      • Changelog
      • Contributing
      • Frequently Asked Questions
      • Troubleshooting
  • Quilt Ecosystem Integrations
    • Benchling Packager
    • Event-Driven Packaging
    • Nextflow Plugin
Powered by GitBook
On this page
  • Catalog push
  • API
  • Metadata

Was this helpful?

  1. Quilt Python SDK
  2. API Reference

Known Limitations

PreviousCLI, EnvironmentNextCustom SSL Certificates

Last updated 1 year ago

Was this helpful?

By design, Quilt is backed by Amazon S3 and scales to billions of objects and petabytes of data. The underlying limitations of S3 apply.

The Quilt catalog can browse packages and S3 buckets of any size.

Catalog push

To ensure usability and quick package pushes, the Quilt web catalog imposes the following limits on pushes (which vary depending on whether the chunked checksums are enabled on the stack). These limits do not apply to the quilt3 Python API.

Dimension
Max (classic / chunked checksums)

Package manifest size (metadata)

100 MiB

Package size (data; via promotion or from an S3 directory)

100 GiB / 5 TiB

Total size of uploaded files (soft limit)

20 GB

Total size of files from S3 (soft limit)

50 GB / 5 TB

Maximum file size

10 GiB / 5 TiB

Maximum number of files per push (soft limit)

1,000

Maximum number of files per push (hard limit)

5,000

API

As of this writing, with sufficient client-side memory, you can comfortable scale Quilt packages to at least one million objects per package, with no practical limit on object size (save S3's 5 TB per object limit). A fast network, or better yet an AWS compute instance in the same region as your Quilt S3 buckets, is recommended.

Metadata

See .

Metadata for teams