build.yml

A build.yml file specifies the structure, type, and names for package contents.

Below is the general syntax of the build.yml file:

contents:
  GROUP_NAME:
    # transform: optional. applies recursively to child nodes. If 'transform'
    #   is omitted, Quilt will select a transform by file extension, falling
    #   back on the "id" transform, which stores the raw data.
    transform:  {id | csv | tsv | ssv | xls | xlsx}
    # kwargs: optional.  applies recursively to child nodes.
    kwargs:
        # Each KEYWORD: VALUE pair is passed directly to the transform function.
        #   For example, with csv this is pandas.read_csv.
        KEYWORD: VALUE  # keyword arguments for transform function/method
    DATA_NAME:
      # file: required.  Relative path from base of package dir.
      file: FILE_PATH
      # transform: optional.  If given, overrides GROUP_NAME's transform.   
      transform: {id, csv, tsv, ssv, xls, xlsx} # optional
      # if transform is omitted, 
      kwargs:  # overrides GROUP_NAME's kwargs for this node.
        KEYWORD: VALUE  # optional. keyword arguments to 
    GLOB_GROUP:     # Container node for nodes created from matching files
      QUOTED_GLOB_PATH:  # standard glob path, such as "*.csv" 
        transform: csv   # set transform for all matched files (optional)
        kwargs:          # set kwargs for all matched files (otional)
          KEYWORD_ARG: VALUE

Example build.yml:

contents:
  data_example:     # create a node named 'data_example'
    transform: csv  # Read files using csv reader
    kwargs:         # optional
      header:       # optional: no header row
      sep: ","      # optional: set field separator
    child:
      file: data/foo.txt # parsed as CSV, no header
    another_child:
      file: data/bar.txt # parsed as CSV, no header
    child_from_elsewhere:
      # parsed as TSV (from kwargs set here) no header (from parent's kwargs)
      file: data2/bar.txt
        kwargs:
          sep: '\t'  # Use tab as separator.
    glob_example:   # create a node named 'glob example'
      # assuming these files exist: 'somedir/foo.xls', 'somedir/subdir/bar.csv', 'somedir/baz.tsv'
      'somedir/**/*':   # create nodes 'foo', 'bar', and 'baz'
        # matched files parsed as CSV, no header
    glob_example_2:  # create a node named 'glob_example_2'
      # assuming these files exist: 'chars/bella.txt', 'chars/edward.txt', 'chars/old/esme.txt'
      'chars/*.txt':  
        transform: tsv  # create nodes 'bella' and 'edward'    
        # matched files parsed as TSV, no header

Reserved words

• file - required for leaf nodes; specifies where source file lives on disk

• transform - specifies how the file will be parsed

• kwargs - these options are passed through to the parser (usually pandas.read_csv so that users can skip lines, type columns, specify delimiters, and much more)

• checks - experimental data unit tests

• environments - experimental environments for checks

• package - experimental source specifier includes an existing package or sub-package in the build tree (see Package Composition)

• *?[!] - any character in this group will initiate glob-style pattern matching

transform and kwargs can be provided at the group level, in which case they apply to all descendants until and unless overridden.

Column types

By default, quilt build converts some file types (e.g., csv, tsv) to Pandas DataFrames using pandas.read_csv. Sometimes, usually due to columns of mixed types, pandas will throw an exception during quilt build. In such cases it's helpful to include column types in build.yml by adding a dtype parameter:

  contents:
    iris:
      file: iris.data
      transform: csv
      kwargs:
        header:
        dtype:
          sepal_length: float
          sepal_width: float
          petal_length: float
          petal_width: float
          class: str

dtype takes a dict where keys are column names and values are valid Pandas column types:

• int

• bool

• float

• complex

• str

• unicode

• buffer

See also dtypes.

Glob and wildcard matching

If a string containing wildcards is used as a node name, it will be matched against the build directory. The filename of any matching path, minus the extension, will be used as the nodename. As when specifying a single data node, kwargs and transform may be used to specify how the file should be read.

The following standard wildcard strings are accepted:

• ** - Match current dir and all subdirs, recursively

• * - Match any one or more characters

• ? - Match any single character

• [X] (where X is any number of characters) - Match any one character contained in X

• [!X] (where X is any number of characters) - Exclude any one character contained in X

All wildcards except ** act in the current directory alone, so * does not match subdir/foo/file.ext, but subdir/foo/* and **/*.ext do.

If provided, transform and any specific kwargs are used with each matched file. Otherwise, the parent (or default) transform and kwargs will be used.

Finally, if matching results in identical node names, the nodes are renamed in a consistent manner (paths are sorted lexicographically), and any duplicate names are numbered. So for files "foo.txt" and "subdir/foo.txt", the result is "foo" (from foo.txt) and "foo_2" (from "subdir/foo.txt"). The naming behavior is consistent across platforms.