LogoLogo
HomeGitHub RepoBook Demo
dev
dev
  • About Quilt
  • Architecture
  • Mental Model
  • Metadata Management
  • Metadata Workflows
  • Quilt Platform (Catalog) User
    • About the Catalog
    • Bucket Browsing
    • Document Previews
    • Embeddable iFrames
    • Packaging Engine
    • Query
    • Quilt+ URIs
    • Qurator Omni
    • Search
    • Visualization & Dashboards
    • Advanced
      • Athena
      • Elasticsearch
      • Removing Stacks
  • Quilt Platform Administrator
    • Admin Settings UI
    • Catalog Configuration
    • Cross-Account Access
    • Enterprise Installs
    • quilt3.admin Python API
    • Advanced
      • Package Events
      • Private Endpoints
      • Restrict Access by Bucket Prefix
      • S3 Events via EventBridge
      • SSO Permissions Mapping
      • Tabulator
      • Troubleshooting
        • SSO Redirect Loop
    • Best Practices
      • GxP for Security & Compliance
      • Organizing S3 Buckets
  • Quilt Python SDK
    • Installation
    • Quick Start
    • Editing a Package
    • Uploading a Package
    • Installing a Package
    • Getting Data from a Package
    • Example: Git-like Operations
    • API Reference
      • quilt3
      • quilt3.Package
      • quilt3.Bucket
      • quilt3.hooks
      • Local Catalog
      • CLI, Environment
      • Known Limitations
      • Custom SSL Certificates
    • Advanced
      • Browsing Buckets
      • Filtering a Package
      • .quiltignore
      • Manipulating Manifests
      • Materialization
      • S3 Select
    • More
      • Changelog
      • Contributing
      • Frequently Asked Questions
      • Troubleshooting
  • Quilt Ecosystem Integrations
    • Benchling Packager
    • Event-Driven Packaging
    • Nextflow Plugin
Powered by GitBook
On this page
  • Quick Start
  • Using Earlier Versions
  • Using Prerelease Versions
  • Output and Input URIs
  • Additional Features

Was this helpful?

  1. Quilt Ecosystem Integrations

Nextflow Plugin

PreviousEvent-Driven Packaging

Last updated 1 month ago

Was this helpful?

  • is a bioinformatics workflow manager that enables the development of portable and reproducible workflows. The software is used by scientists and engineers to write, deploy and share data-intensive, highly scalable, workflows on any infrastructure.

  • nf-quilt is a developed by Quilt Data that enables you read and write directly to instead of just S3 locations.

Quick Start

All you need to do is add the nf-quilt plugin to a Nextflow pipeline that writes to Amazon S3. The plugin will automatically create a Quilt package with metadata from each run. You can do this in one of three ways.

  1. Add it to the command-line:

    nextflow run nf-core/rnaseq -plugins nf-quilt --outdir "s3://quilt-example-bucket/test/nf_quilt_rnaseq"
  2. Include it in the nextflow config file (e.g., main.nf):

    plugins {
        id 'nf-quilt'
    }
  3. Specify it in the Advanced Options for a Seqera Platform job:

Advanced Options > Nextflow config file

Using Earlier Versions

To use older versions of nf-quilt, you can specify the version number of the plugin using the '@' sign:

nextflow run main.nf -plugins nf-quilt@0.7.16

Using Prerelease Versions

To use unreleased versions of the nf-quilt plugin, you must also set the location using environment variable. For example, to use version 0.8.6, setNXF_PLUGINS_TEST_REPOSITORY from the command-line or the "Pre-run script" of the Seqera Platform:

# export NXF_VER=23.04.3
export LOG4J_DEBUG=true  # for verbose logging
export NXF_PLUGINS_TEST_REPOSITORY=https://github.com/quiltdata/nf-quilt/releases/download/0.8.6/nf-quilt-0.8.6-meta.json

Output and Input URIs

The canonical reference to a package is defined by a quilt+ URI. For example, the s3://quilt-example-bucket/test/nf_quilt_rnaseq S3 URI will create a package with the Quilt URI:

quilt+s3://quilt-example-bucket#package=test/nf_quilt_rnaseq

You can then use that URI as input to future jobs, and similar URIs for the output, e.g.,

nextflow run my/analysis \
 --indir quilt+s3://quilt-example-bucket#package=test/nf_quilt_rnaseq.csv \
 --outdir quilt+s3://prod-bucket#package=experiment/analysis

Additional Features

The nf-quilt plugin supports a wide range of additional options for configuring input, output, and metadata. For more details, or to participate in the development, please visit the GitHub repository.

quiltdata/nf-quilt
Nextflow
Nextflow plugin
Quilt data packages