Refgenie

reference genome manager

What is refgenie?

Refgenie manages storage, access, and transfer of reference genome resources. It provides command-line and Python interfaces to download pre-built reference genome “assets”, like indexes used by bioinformatics tools. It can also build assets for custom genome assemblies. Refgenie provides programmatic access to a standard genome folder structure, so software can swap from one genome to another.

In a hurry? Check out the demo videos that present the most relevant refgenie features in 3 minutes!

What makes refgenie better?

1. It provides a command-line interface to download individual resources. Think of it as GitHub for reference genomes. You just type refgenie pull hg38/bwa_index.

2. It’s scripted. In case you need resources not on the server, such as for a custom genome, you can build your own: refgenie build custom_genome/bowtie2_index.

3. It simplifies finding local asset locations. When you need a path to an asset, you can seek it, making your pipelines portable across computing environments: refgenie seek hg38/salmon_index.

4. It provides remote operation mode, useful for cloud applications. Get a path to an asset file hosted on AWS S3: refgenie seekr hg38/fasta --remote-class s3.

5. It includes a Python API. For tool developers, you use rgc = refgenconf.RefGenConf("genomes.yaml") to get a Python object with paths to any genome asset, e.g., rgc.seek("hg38", "kallisto_index").

6. It strictly determines genomes compatibility. Users refer to genomes with arbitrary aliases, like “hg38”, but refgenie uses sequence-derived identifiers to verify genome identity with asset servers.

Quick example

Install

Refgenie is a Python package package, install from PyPi:

pip install --user refgenie

Or conda:

conda install refgenie

And that’s it! If you wish to use refgenie in remote mode See further reading on remote mode in refgenie.

If you’re connected to the Internet, call a test command, e.g.:

refgenie seekr hg38/fasta

Initialize to use refgenie locally

Refgenie keeps track of what’s available using a configuration file initialized by refgenie init:

export REFGENIE='genome_config.yaml'
refgenie init -c $REFGENIE

Download indexes and assets for a remote reference genome

Use refgenie pull to download pre-built assets from a remote server. View available remote assets with listr:

refgenie listr

Response:

                        Remote refgenie assets
                 Server URL: http://refgenomes.databio.org
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ genome              ┃ assets                                       ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ mouse_chrM2x        │ fasta, bwa_index, bowtie2_index              │
│ hg38                │ fasta, bowtie2_index                         │
│ rCRSd               │ fasta, bowtie2_index                         │
│ human_repeats       │ fasta, hisat2_index, bwa_index               │
└─────────────────────┴──────────────────────────────────────────────┘

Next, pull one:

refgenie pull rCRSd/bowtie2_index

Response:

Downloading URL: http://rg.databio.org/v3/assets/archive/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index
94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index:default ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 128.0/117.0 KB • 1.8 MB/s • 0:00:00
Download complete: /Users/mstolarczyk/Desktop/testing/refgenie/data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index/bowtie2_index__default.tgz
Extracting asset tarball: /Users/mstolarczyk/Desktop/testing/refgenie/data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index/bowtie2_index__default.tgz
Default tag for '94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index' set to: default
Created alias directories:
 - /Users/mstolarczyk/Desktop/testing/refgenie/alias/rCRSd/bowtie2_index/default

See further reading on downloading assets.

Build your own indexes and assets for a custom reference genome

Refgenie assets are scripted, so if what you need is not available remotely, you can use build it locally:

refgenie build mygenome/bwa_index

See further reading on building assets.

Retrieve paths to local refgenie-managed assets

Once you’ve populated your refgenie with a few assets, use seek to retrieve their local file paths:

refgenie seek mm10/bowtie2_index

This will return the path to the particular asset of interest, regardless of your computing environment. This gives you an ultra-portable asset manager! See further reading on retrieving asset paths.

Retrieve paths to remote refgenie-managed assets

Use seekr (short for “seek remote”) to retrieve remote seek_key targets:

refgenie seekr mm10/fasta.fai

This will return the path to the particular remote file of interest, here: FASTA index file, which is a part of mm10/fasta asset.

See further reading on using refgenie in remote mode.

---

If you want to read more about the motivation behind refgenie and the software engineering that makes refgenie work, proceed next to the overview.