LogoLogo
HomeGitHub RepoBook Demo
version-5.0.x
version-5.0.x
  • Introduction
  • Installation
  • Quick start
  • Mental model
  • Architecture
  • Walkthrough
    • Editing a Package
    • Uploading a Package
    • Installing a Package
    • Getting Data from a Package
    • Working with the Catalog
    • Working with a Bucket
  • API Reference
    • quilt3
    • quilt3.Package
    • quilt3.Bucket
    • CLI, environment
    • Known limitations
  • Catalog
    • Admin UI
    • Configuration
    • Embed
    • Metadata for teams
    • Preview
    • Search & query
    • Visualization & dashboards
    • Local Development Mode
  • Examples
    • Git-like operations for datasets and Jupyter notebooks
  • Advanced
    • Filtering a Package
    • .quiltignore
    • Materialization
    • Working with Manifests
    • S3 Select
    • Workflows
    • Enterprise install
    • S3 Events, EventBridge
    • Cross-account access
  • More
    • Frequently Asked Questions
    • Troubleshooting
    • Contributing
    • Changelog
Powered by GitBook
On this page
  • Brief tour
  • Admin UI

Was this helpful?

  1. Walkthrough

Working with the Catalog

PreviousGetting Data from a PackageNextWorking with a Bucket

Last updated 3 years ago

Was this helpful?

The Quilt Catalog is the second half of Quilt. It provides an interface on top of your S3 bucket that brings Quilt features like data packages and search to the web.

.

Note that you can use the Quilt Python API without using the catalog product, but they are designed to work together.

Brief tour

The Quilt catalog provides a homepage for your catalog, based on a README.md file that you can optionally create at the top of your bucket.

Browse

The catalog lets you navigate packages in the registry on the packages tab.

You can also browse the underlying S3 files using the files tab.

Search

Upload package

The name should have the format namespace/package-name.

The message needs to add notes on a new revision for this package.

Files are the content of your package.

The associated workflow contains the rules for validating your package.

The metadata can be added with JSON editor, represented as a key/value table with infinite nesting. If workflow contains JSON schema, you will have predefined key/value pairs according to the schema.

JSON editor

To add a new key/value field double click on an empty cell and type key name, then press "Enter" or "Tab", or click outside of the cell. To change value double click on that value.

Values can be strings, numbers, arrays, or objects. Every value that you type will be parsed as JSON.

We don't support references and compound types yet.

Push to bucket

Summarize

Adding a quilt_summarize.json file to a data package (or S3 directory path) will enable content preview right on the landing page.

Colocating data with context in this way is a simple way of making your data projects approachable and accessible to collaborators.

quilt_summarize.json can be a list of paths to files in S3 that you want to include in your summary. For example: ["description.md", "../notebooks/exploration.ipynb"]. Additionally, note that if a README.md file is present, it will always be rendered as well.

There are currently some small limitations with preview:

  • Objects linked to in quilt_summarize.json are always previewed as of the latest version, even if you are browsing an old version of a package.

  • Object titles and image thumbnails link to the file view, even if you are in the package view.

Admin UI

Catalogs also enable you to search the contents of your bucket. We support both unstructured (e.g. "San Francisco") and structured with (e.g. "metadata_key: metadata_value") search. Hits are previewed right in the search results.

You can upload a new package providing the name of the package, commit message, files, metadata, and .

You can push the existing package from one bucket to another. To use this feature consult page.

The Quilt catalog includes an admin panel that allows you to manage users and buckets in your stack and to customize your Quilt catalog. See for details.

.

Query String Queries
workflow
workflows
Admin UI docs
To learn more, check out the public demo catalog
For a hands-on demo, check out the public demo catalog