How RefgetStore defers loading

A RefgetStore can describe far more data than you want to move. A store holding a few thousand reference genomes has a sequence index of tens of megabytes and hundreds of gigabytes of sequence bytes behind it. Opening such a store and asking “which collections are here?” should not require downloading any of that.

RefgetStore handles this with deferral at two levels: individual records exist as metadata-only placeholders until something asks for their data, and for remote stores the index files themselves are fetched on demand. This page explains both mechanisms and what they imply for how a store behaves.

Records: Stub and Full

Every sequence and every collection in a store is held as one of two variants.

Variant	Sequence record	Collection record
Stub	Metadata only: name, length, alphabet, sha512t24u, md5, description	Metadata only: collection digest and the attribute digests (names, lengths, sequences, and the ancillary digests)
Full	Metadata plus the sequence bytes	Metadata plus the list of member sequence records

Loading a store parses index files, which contain only metadata, so every record starts as a Stub. Nothing has been read from a .seq file and no per-collection .rgsi file has been opened.

A record is promoted to Full on first access:

• A sequence is promoted when its bytes are fetched, either by an explicit load_sequence(digest) or lazily when a read method needs them. The bytes are read from the local .seq file, or downloaded from the remote store and written into the local cache.
• A collection is promoted when its collections/<digest>.rgsi file is read, which also populates the name lookup table that maps sequence names to digests within that collection.

Promotion is sticky. Once a record is Full it stays resident until you call clear(), which drops all sequence records and preserves the metadata, collections, name lookups, MD5 lookups, aliases, and FHR metadata.

You can inspect the state of the store directly:

from refget.store import RefgetStore

store = RefgetStore.open_local("/path/to/store")

info = store.stats()
print(info["n_sequences"], info["n_sequences_loaded"])
print(info["n_collections"], info["n_collections_loaded"])

record = store.get_sequence("2i7v3vX9EDUnPWlkZMxaMFI2P0hDMOFC")
print(record.is_loaded)

Note that is_loaded reports whether the record holds bytes in memory, not whether the sequence exists in the store. A Stub in a disk-backed store is fully retrievable; it just has not been read yet.

Indexes: what a remote store downloads when

For a local store, the index files are already on disk and reading them is cheap, so both are parsed at open time. For a remote store the same files must be fetched over the network, and the sequence index is by far the larger of the two: sequences.rgsi carries one line per sequence, which reaches tens of megabytes for a store holding thousands of assemblies.

Opening a remote store therefore fetches only:

• rgstore.json, the manifest
• collections.rgci, the collection index, which yields one Stub collection record per collection

sequences.rgsi is left alone. It is downloaded the first time a sequence is actually requested. Until then the store knows every collection that exists, can list and filter them, and can serve level 1 attribute digests, all without having transferred a sequence index or a single base.

from refget.store import RefgetStore

# Fetches rgstore.json and collections.rgci only.
store = RefgetStore.open_remote("/tmp/refget-cache", "https://example.org/store/")

page = store.list_collections(page=0, page_size=10)
for meta in page["results"]:
    print(meta.digest, meta.n_sequences)
print(page["pagination"]["total"])

# This first sequence request triggers the sequences.rgsi download.
seq = store.get_sequence_by_name(page["results"][0].digest, "chr1")

Downloaded index files land in the local cache directory, so the deferral cost is paid once per cache rather than once per session.

Store size without the sequence index

Deferring the sequence index creates a gap: a client that has not downloaded sequences.rgsi cannot sum sequence lengths, so it cannot answer “how big is this store?”

The manifest closes the gap. At index-write time the store computes the total logical size of all sequence payloads (each sequence’s length converted through the storage mode’s bits-per-symbol) and records it in rgstore.json as logical_sequence_bytes. A client reads it straight from the manifest it already downloaded:

meta = store.store_metadata()
print(meta.get("logical_sequence_bytes"))

This value covers sequence payloads only. It excludes index files, alias files, FHR sidecars, and the manifest itself, and it is computed arithmetically rather than by walking the directory, so it reflects what the sequences would occupy rather than the exact current on-disk footprint.

Where deferral stops

Lazy loading is a property of RefgetStore, the mutable wrapper type. Promoting a Stub to Full mutates the store, so every lazy-loading read method takes a mutable borrow.

ReadonlyRefgetStore, the variant intended for concurrent serving, does not lazy-load: its read methods borrow immutably and therefore cannot promote a record. Data must be preloaded before conversion. See The readonly store and concurrent access for what to preload and why.

One exception is worth knowing. Substring retrieval against a local disk-backed store reads only the bytes covering the requested range directly from the .seq file, without promoting the record. A Stub is enough for get_substring and get_substrings in that case. Against a remote-only sequence, those methods raise rather than silently downloading a whole chromosome. The retrieval flows reference covers the trade-offs between partial reads, streaming, and full loads.

Related reading

• RefgetStore file format for the manifest fields and index file layouts described here
• RefgetStore retrieval flows for choosing between partial reads, streaming, and loading
• The readonly store and concurrent access for preloading requirements when serving