LogoLogo
HomeGitHub RepoBook Demo
dev
dev
  • About Quilt
  • Architecture
  • Mental Model
  • Metadata Management
  • Metadata Workflows
  • Quilt Platform (Catalog) User
    • About the Catalog
    • Bucket Browsing
    • Document Previews
    • Embeddable iFrames
    • Packaging Engine
    • Query
    • Quilt+ URIs
    • Qurator Omni
    • Search
    • Visualization & Dashboards
    • Advanced
      • Athena
      • Elasticsearch
      • Removing Stacks
  • Quilt Platform Administrator
    • Admin Settings UI
    • Catalog Configuration
    • Cross-Account Access
    • Enterprise Installs
    • quilt3.admin Python API
    • Advanced
      • Package Events
      • Private Endpoints
      • Restrict Access by Bucket Prefix
      • S3 Events via EventBridge
      • SSO Permissions Mapping
      • Tabulator
      • Troubleshooting
        • SSO Redirect Loop
    • Best Practices
      • GxP for Security & Compliance
      • Organizing S3 Buckets
  • Quilt Python SDK
    • Installation
    • Quick Start
    • Editing a Package
    • Uploading a Package
    • Installing a Package
    • Getting Data from a Package
    • Example: Git-like Operations
    • API Reference
      • quilt3
      • quilt3.Package
      • quilt3.Bucket
      • quilt3.hooks
      • Local Catalog
      • CLI, Environment
      • Known Limitations
      • Custom SSL Certificates
    • Advanced
      • Browsing Buckets
      • Filtering a Package
      • .quiltignore
      • Manipulating Manifests
      • Materialization
      • S3 Select
    • More
      • Changelog
      • Contributing
      • Frequently Asked Questions
      • Troubleshooting
  • Quilt Ecosystem Integrations
    • Benchling Packager
    • Event-Driven Packaging
    • Nextflow Plugin
Powered by GitBook
On this page
  • Quick Start
  • Using Earlier Versions
  • Using Prerelease Versions
  • Output and Input URIs
  • Additional Features

Was this helpful?

  1. Quilt Ecosystem Integrations

Nextflow Plugin

PreviousEvent-Driven Packaging

Last updated 2 months ago

Was this helpful?

  • Nextflow is a bioinformatics workflow manager that enables the development of portable and reproducible workflows. The software is used by scientists and engineers to write, deploy and share data-intensive, highly scalable, workflows on any infrastructure.

  • nf-quilt is a Nextflow plugin developed by Quilt Data that enables you read and write directly to Quilt data packages instead of just S3 locations.

Quick Start

All you need to do is add the nf-quilt plugin to a Nextflow pipeline that writes to Amazon S3. The plugin will automatically create a Quilt package with metadata from each run. You can do this in one of three ways.

  1. Add it to the command-line:

    nextflow run nf-core/rnaseq -plugins nf-quilt --outdir "s3://quilt-example-bucket/test/nf_quilt_rnaseq"
  2. Include it in the nextflow config file (e.g., main.nf):

    plugins {
        id 'nf-quilt'
    }
  3. Specify it in the Advanced Options for a Seqera Platform job:

Advanced Options > Nextflow config file

Using Earlier Versions

To use older versions of nf-quilt, you can specify the version number of the plugin using the '@' sign:

nextflow run main.nf -plugins nf-quilt@0.7.16

Using Prerelease Versions

To use unreleased versions of the nf-quilt plugin, you must also set the location using environment variable. For example, to use version 0.8.6, setNXF_PLUGINS_TEST_REPOSITORY from the command-line or the "Pre-run script" of the Seqera Platform:

# export NXF_VER=23.04.3
export LOG4J_DEBUG=true  # for verbose logging
export NXF_PLUGINS_TEST_REPOSITORY=https://github.com/quiltdata/nf-quilt/releases/download/0.8.6/nf-quilt-0.8.6-meta.json

Output and Input URIs

The canonical reference to a package is defined by a quilt+ URI. For example, the s3://quilt-example-bucket/test/nf_quilt_rnaseq S3 URI will create a package with the Quilt URI:

quilt+s3://quilt-example-bucket#package=test/nf_quilt_rnaseq

You can then use that URI as input to future jobs, and similar URIs for the output, e.g.,

nextflow run my/analysis \
 --indir quilt+s3://quilt-example-bucket#package=test/nf_quilt_rnaseq.csv \
 --outdir quilt+s3://prod-bucket#package=experiment/analysis

Additional Features

The nf-quilt plugin supports a wide range of additional options for configuring input, output, and metadata. For more details, or to participate in the development, please visit thequiltdata/nf-quilt GitHub repository.