Packaging Engine

This feature requires Quilt Platform version 1.58.0 or higher

Overview

The Quilt Packaging Engine in the Quilt Platform allows administrators and developers to automate the process of creating Quilt packages from data stored in Amazon S3. It serves as a key component of Quilt's functionality as a Scientific Data Management System, enabling automated data ingestion and standardization. It currently consists of:

1. Admin Settings GUI to enable package creation based on notifications from:

   1. AWS Health Omics

   2. Nextflow workflows using the WRROC (Workflow Run RO-Crate) format from nf-prov.

2. SQS queue that will process package descriptions

3. Documentation for creating custom EventBridge rules to invoke that queue

Admin Settings

The simplest way to enable package creation is through the Admin Settings GUI, which supports the following built-in event sources:

Admin Settings

AWS Health Omics

When enabled, this will create a package from the runOutputUri provided in a aws.omics completion event. For example, if the runOutputUri is s3://quilt-example/omics-quilt/3395667, the package will be created in that same bucket with the name omics-quilt/3395667.

Workflow Run RO-Crate

When enabled, this will create a package from the crate when an ro-crate-metadata.json file is written to a bucket that is already part of the stack. The engine is a consumer of the Quilt RO-Crate profile, which defines how a crate states its package name, files, people, instrument and lab notebook entry.

A crate is only read when the message names it in metadata_uri, which the built-in rule does. A source_prefix alone — even one pointing at the ro-crate-metadata.json itself — packages the enclosing folder as an ordinary prefix and never reaches the crate. Nothing fails on that account, so the usual symptom is a package with no crate entries or metadata — unless a workflow then rejects the package for the metadata it is missing.

The crate's graph decides what the package contains:

• Entries are exactly the root dataset's hasPart list, plus the crate itself, which records the graph the package was built from. A relative @id resolves against the crate's folder and keeps its path; an s3:// URI may point outside it and is entered under its file name alone, so two such parts that share a file name collide and the crate is rejected. A directory part (a Dataset entity, or an @id ending in /) includes everything under it. Paths that climb out of the folder are rejected, as is a part whose scheme names data the packager cannot read. An @id that identifies rather than locates — a URL, a URN, a DOI — has no object to package and is skipped.

• Entry metadata for each File entity is every property other than @id, @type and name, so dateCreated, dateModified and sha256 survive the upload. This holds whether the file is listed in hasPart itself or reached by expanding a directory part; an object with no File entity gets none.

• Package metadata is a flat projection keyed by the role an entity plays for the root dataset, not by its @type, with human-readable values:

  • package_name: the name the package is published under

  • creator: each root creator

  • producer: each root producer, then its parentOrganization chain

  • instrument: the instrument of each action in the root's mentions

  • instrument_id: the identifier of those same instruments

  • eln_entry: each root subjectOf whose additionalType is the profile's ELNEntry

  Values are entity names except instrument_id. Roles the crate does not state are omitted.

• Package name comes from a PropertyValue under the root's identifier: a packageName gives the full name, which must already be a valid package name and is rejected otherwise; a packageNamespace is joined to the root dataset's name, sanitized to the package-name grammar. Without either, the name is inferred from the S3 key (below).

A crate is rejected only when it cannot be packaged as written: a hasPart member that cannot be resolved, an invalid explicit package name, or two conflicting entities with one @id. Nothing is packaged in that case, rather than part of the crate. How the crate models people, instruments, actions and anything else is up to its producer; the profile's recommendations only decide which of the metadata keys above can be filled in. A metadata file that is not an RO-Crate (no ./ entity of type Dataset) packages the whole enclosing folder and is used verbatim as package metadata.

RO-Crate is a metadata standard for describing research data. The Workflow Run working group adds three additional profiles, which are supported in the latest versions of nf-prov. You will need to explicitly configure nf-prov to use wrroc, by using a nextflow.config file like this:

plugins {
 id 'nf-prov@1.4.0'
}

prov {
 enabled = true
 formats {
   wrroc {
     file = "${params.outdir}/ro-crate-metadata.json"
     overwrite = true
     agent {
       name = "John Doe"
       orcid = "https://orcid.org/0000-0000-0000-0000"
     }
      license = "https://spdx.org/licenses/MIT"
      profile = "provenance_run_crate"
   }
 }
}

Note that Research Objects identify people using an ORCID iD, which anyone can get for free at the ORCID website.

The package will be created in the same bucket as the outdir. Unless the crate names the package as described above, the name is inferred from the S3 key: for my/s3/folder/ro-crate-metadata.json it is my_s3/folder.

Architecture

The Quilt Packaging Engine is built on top of the existing packaging lambdas used by the Quilt Platform, including the ability to parallelize creation of S3 Checksums for existing objects (see Checksums and Hashing for the algorithm and how to enable faster CRC64/NVME checksums). We have exposed this functionality to customers via an SQS queue, which is invoked by the EventBridge rules created by the Admin Settings GUI.

SQS Parameters

You can also send messages directly to the SQS queue, which is part of the Quilt stack. The queue information will be listed as PackagerQueueArn and PackagerQueueUrl under the Outputs tab in CloudFormation section of the AWS Console. The URL will be something like:

https://sqs.REGION.amazonaws.com/ACCOUNT_ID/stack-name-PackagerQueue-XXX

Where REGION and ACCOUNT_ID will be the same as for the Quilt stack.

PackagerQueueArn and PackagerQueueUrl in the CloudFormation Outputs tab

The body of the message is the stringified JSON of a package description. There is only one required parameter:

{
  "source_prefix": "s3://data_bucket/source/folder/"
}

This is assumed to be a folder if it ends in a /; otherwise, we will remove the last component of the path to get the folder. That last component is only discarded, never read: a metadata document — an RO-Crate included — is read only from metadata_uri. The contents of the folder will be used to create a package in the same bucket as the source folder, with the package name being inferred from the source URI.

Optionally, you can control the package name, metadata, and other settings by explicitly specifying any of the following fields:

{
  "source_prefix": "s3://data_bucket/source/folder/", // trailing '/' for folder
  "registry": "package_bucket", // may be the same as `data_bucket`
  "package_name": "prefix/suffix",
  "metadata": { "key": "value" }, // object (or metadata URI, but not both)
  "metadata_uri": "metadata.json", // S3 URI to read, relative or absolute
                                   // (required for an RO-Crate)
  "commit_message": "Commit message for the package revision", // string
  "workflow": "alpha", // name of a valid metadata workflow
}

The job will fail if you try to specify both metadata and metadata_uri.

Notes on individual fields:

• package_name must match ^[\w-]+/[\w-]+$ (a namespace and a name consisting of letters, digits, underscores, and hyphens, separated by a single /). When the name is inferred from source_prefix, characters outside that set are replaced with hyphens.

• metadata_uri is what selects RO-Crate ingestion; a crate that is not named here is packaged as an ordinary file, and the same graph passed inline in metadata is stored verbatim rather than parsed. A relative value resolves against source_prefix.

• workflow has three-way semantics:

  • omitted — the registry's default workflow (if one is configured) is applied, and package creation fails if the package does not validate against it;

  • "" (empty string) — no workflow is applied, even if a default exists;

  • a workflow name — that specific workflow is applied.

SendMessage API

If you have appropriate IAM permissions, and the SQS URL, you can send a message to the queue using the AWS SDK or the AWS CLI. Here is an example using the AWS CLI:

export QUEUE_URL=https://sqs.us-east-1.amazonaws.com/XXX/PackagerQueue-XXX
aws sqs send-message --queue-url $QUEUE_URL \
--message-body '{"source_prefix":"s3://data_bucket/source/folder/"}'

Custom EventBridge Rules

EventBridge rules can be used to transform EventBridge events from any bus in your account into a conforming SQS message.

Example: Event-Driven Packaging (EDP)

Event-Driven Packaging, currently in private preview, coalesces multiple S3 uploads into a single package-objects-ready event, which infers the appropriate top-level folder. When ready, it creates an event like this on its own EventBridge bus:

{
  "version":"0",
  "id":"XXXXXXXXXXXXXX",
  "detail-type":"package-objects-ready",
  "source":"com.quiltdata.edp",
  "account":"XXX",
  "time":"2022-12-08T20:01:34Z",
  "region":"us-east-1",
  "resources":[
    "arn:aws:s3:::bucket-name"
  ],
  "detail":{
    "version":"0.1",
    "bucket":"bucket-name",
    "prefix":"prefix-path-1/prefix-path-2/"
  }
}

The following Python code creates an EventBridge rule that targets the packager queue when matching that event:

import boto3
import json

# AWS Region and Account Details
region = "us-east-1"
account_id = "XXXXXXXXXXXX"
sqs_queue_name = "PackagerQueue-XXXXXXXXXXXX"
sqs_arn = f"arn:aws:sqs:{region}:{account_id}:{sqs_queue_name}"

# Initialize AWS Clients
eventbridge = boto3.client("events", region_name=region)

# Step 1: Create EventBridge Rule

event_pattern = {
    "source": ["com.quiltdata.edp"],
    "detail-type": ["package-objects-ready"]
}

response = eventbridge.put_rule(
    Name="RouteEDPEventsToSQS",
    EventPattern=json.dumps(event_pattern),
    State="ENABLED",
    Description="Routes package-objects-ready events to SQS",
)

print(f"EventBridge Rule Created: {response['RuleArn']}")

# Step 2: Attach the SQS Target with Input Transformation

targets = [
    {
        "Id": "SQS_PackagerQueue",
        "Arn": sqs_arn,
        "InputTransformer": {
            "InputPathsMap": {
                "bucket": "$.detail.bucket",
                "prefix": "$.detail.prefix"
            },
            "InputTemplate": """{
                \"source_prefix\": \"s3://<bucket>/<prefix>\"
            }"""
        }
    }
]

response = eventbridge.put_targets(
    Rule="RouteEDPEventsToSQS",
    Targets=targets
)

print("SQS Target Attached to EventBridge Rule:", response)

Limits

Server-side package operations (creating, copying, and hashing packages) are subject to size and file-count limits. The limit values are configured per-stack by the CloudFormation template; the error codes below are what you see when an operation exceeds one of them:

• ManifestTooLarge — the source package manifest exceeds the maximum manifest size when copying a package across buckets

• PackageTooLargeToCopy — the total size of package data exceeds the maximum when copying a package's objects

• TooManyFilesToCopy — the number of files exceeds the maximum when copying a package's objects

• PackageTooLargeToHash — the total bytes needing new checksums exceed the maximum for a single package operation

• TooManyFilesToHash — the number of files needing new checksums exceeds the maximum

• FileTooLargeForHashing — a single file exceeds the maximum size the hashing lambda accepts

• RequestTooLarge — the request payload passed via S3 exceeds the maximum request size

If you hit one of these limits, contact your Quilt administrator or Quilt support — most limits can be adjusted for your stack.

Caveats

1. The package creation process is asynchronous, so you may need to wait a few minutes before the package is available (longer if the source data is large).

2. If you send the same message multiple times before the folder is updated, it will not actually create a new revision, since the content hash will be the same. However, that would still waste computational cycles, so you should avoid doing so.