RefgetStore sequence retrieval flows

A RefgetStore can be read from three places: sequence bytes may be resident in memory, sitting in a local .seq file on disk, or available only from a remote HTTP store. Because moving whole chromosomes is expensive, the store offers three distinct retrieval flows so you can match the cost to your access pattern instead of always paying for a full download.

This page is a reference for those flows: what each one fetches, what it costs, and when to use it.

The three flows

Flow	What it moves	Peak memory	Persists locally?	Repeated remote reads	Best for
1. Partial read (get_substring, get_substrings)	Only the bytes covering [start, end), returned as a string	O(region)	No	(resident/local only)	Sparse, random-access extraction from data that is already resident or on local disk
2. Streaming (stream_sequence)	Only the bytes covering [start, end), as a byte/character stream — local seek or remote HTTP Range	O(1) in region length	No	Re-fetches each call	One-off or large region pulls from a remote store without downloading whole sequences
3. Load & cache (load_sequence, load_all_sequences)	The whole sequence .seq, downloaded once	O(sequence)	Yes (under the local cache)	Served locally after first load	Repeated access to the same sequences; warm reuse across sessions

The key trade-off is flow 2 vs flow 3 for remote data:

• Flow 2 (streaming / byte-range) fetches only the bases you ask for. A 50 bp lookup transfers ~50 bytes. Nothing is stored, so a second read of the same region fetches again. Ideal for sparse extraction — a handful of loci scattered across a genome.
• Flow 3 (load & cache) downloads the entire sequence (a whole chromosome can be tens of MB even encoded), persists it to the local cache, and holds it in memory. The first touch is expensive; every read afterward is local and fast. Ideal for dense or repeated access — e.g. converting an entire VCF against one assembly.

Flow 1 is the convenience layer over already-available data: once a sequence is resident (flow 3) or its .seq is on local disk, get_substring reads just the covering bytes with no full-sequence load.

Source resolution

get_substring / stream_sequence resolve their byte source in order:

1. Resident — if the sequence is fully loaded (Full), read from the in-memory buffer.
2. Local .seq — if a local store path holds the file, do a positioned read of just the covering bytes (the whole sequence never enters RAM).
3. Remote byte-range — streaming only: if the file is remote, issue an HTTP Range: request for the covering bytes. This requires the build’s http feature.

get_substring stops at step 2: against a remote-only sequence it raises an error rather than silently downloading. To read a remote region, use flow 2 (stream the region) or flow 3 (load_sequence to download and cache the whole sequence, after which flow 1 serves it from memory).

Choosing a flow

Need bases for a region of a sequence?
│
├─ Is the sequence resident or on local disk?
│     └─ Yes → Flow 1: get_substring / get_substrings
│
└─ Is it remote-only?
      ├─ A few sparse regions, read once → Flow 2: stream_sequence (byte-range; no download)
      └─ Many regions / repeated reads    → Flow 3: load_sequence, then Flow 1

Binding interface

The same three flows are exposed through a consistent interface across the bindings — Python, R, Node.js (native), and WebAssembly:

Method	Flow	Notes
get_substring(digest, start, end)	1	Single region → string
get_substrings(digest, ranges)	1	Many regions of one sequence in a single call
stream_sequence(digest, start, end)	2	Region as a stream; the remote byte-range path
load_sequence(digest)	3	Download + cache + make resident
load_all_sequences()	3	Eagerly promote every sequence

In the browser, WebAssembly has no synchronous disk or network access, so the fetch-and-cache machinery for flows 2 and 3 is driven from JavaScript (fetching .seq bytes, caching them in the Origin-Private File System) and handed to the in-memory wasm store. The method names mirror the native bindings so the mental model carries over.

Note

On-demand remote retrieval through the byte-range path (flow 2) and uniform coverage of all three flows across every binding are being rolled out. The flow definitions and the trade-offs above are stable; consult your binding’s release notes for the exact methods currently available.

Related reference

• RefgetStore file format — the on-disk/remote directory layout, including the sequences/<ab>/<digest>.seq files these flows read.
• RefgetStore encoding — how .seq bytes map to bases, which determines the byte range a region read must fetch.