Query

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

The Catalog's Queries tab allows you to run Athena queries against your S3 buckets, and any other data sources your users have access to. There are prebuilt tables for packages and objects, and you can create your own tables and views. See, for example, Tabulator.

NOTE: This page describes how to use Athena for precise querying of specific tables and fields. For full-text searching using Elasticsearch, see the Search page.

Basics

"Run query" executes the selected query and waits for the result.

ui

Individual users will also see their past queries, and easily re-run them.

Example: query package-level metadata

Suppose we wish to find all packages produced by algorithm version 1.3 with a cell index of 5. As of Quilt Platform version 1.70, package-level metadata lives in the per-bucket Iceberg package_manifest table, which replaced the old *_packages-view Athena view:

SELECT * FROM "YOUR-BUCKET_package_manifest"
-- extract and query package-level metadata
WHERE json_extract_scalar(metadata,
  '$.user_meta.nucmembsegmentationalgorithmversion') LIKE '1.3%'
AND json_array_contains(json_extract(metadata, '$.user_meta.cellindex'), '5');

Example: query object-level metadata

Suppose we wish to find all .tiff files produced by algorithm version 1.3 with a cell index of 5. Object-level (file entry) metadata is now in the per-bucket Iceberg package_entry table, which replaced the old *_objects-view Athena view:

SELECT * FROM "YOUR-BUCKET_package_entry"
WHERE substr(logical_key, -5) = '.tiff'
-- extract and query object-level metadata
AND json_extract_scalar(metadata,
  '$.user_meta.nucmembsegmentationalgorithmversion') LIKE '1.3%'
AND json_array_contains(json_extract(metadata, '$.user_meta.cellindex'), '5');

Configuration

Athena queries saved from the AWS Console for a given workgroup will be available in the Quilt Catalog for all users to run.

Administrators can hide the "Queries" tab by setting ui > nav > queries: false (learn more).