Remote datasets

BetaWorks, but under active development — APIs and results may shift. Needs a noeira cloud account, or a catalog service and object store you run yourself.

Training happens on rented GPU boxes that are created and destroyed. Without a remote store, every dataset is either regenerated on each box or pushed from a laptop each time. RemoteCatalog (noeira/data/remote.mojo) closes that gap: it resolves a dataset name to bytes at a local path, fetching only if the local copy is missing or stale. The catalog it talks to is noeira cloud, the same service that records runs.

Requires noeira cloud

noeira cloud is not open yet. The dataset and asset-pack syncing on this page needs it; the Hugging Face downloader does not. To join the private beta, write to denis@noeira.ai.

Three kinds of thing move through it:

What	Where it lives locally	Commands
A catalog dataset (a trajectory store)	any path	RemoteCatalog.pull / push
A project recording (LeRobot v3)	projects/<project>/datasets/<dataset>/	pixi run dataset-push / dataset-pull / dataset-list
An environment asset pack	beside the env’s assets.kv	pixi run assets-status / assets-pull / assets-where

from noeira.data.remote import RemoteCatalog

var cat = RemoteCatalog.from_env()                       # reads .env
var path = cat.pull(String("walker-random-v1"), String("data/walker.h5"))
var store = TrajectoryStore(path^)

Credentials come from a dotenv file, the same two keys run logging uses:

NOEIRA_CLOUD_URL=https://cloud.noeira.ai
NOEIRA_CLOUD_API_KEY=…

from_env raises with the missing key named, rather than letting it surface later as a 401 — an empty key produces a bare Authorization: Bearer header, which is indistinguishable at the server from a wrong one.

The service is never in the data path

box ──1. GET /datasets/:id  (Bearer API key)──> catalog service ──> metadata DB
    <─2. metadata + presigned URL ─────────────┘
    ──3. GET / PUT the bytes ──────────────────> object store (direct)

The service returns a presigned URL; the client transfers directly against the object store. This is a deliberate shape, not an implementation detail:

• there is no custom transfer protocol, and no request-duration limit in the data path — proxying a 40 GB body through an edge function is the obvious wrong turn;
• on the box, curl, rclone, aws s3 cp and the built-in downloader all work unchanged;
• multipart upload for large objects is never implemented by us — those tools do it.

noeira cloud stores the bytes in Cloudflare R2 behind its Worker, chosen because egress is free: pulling datasets to GPU boxes as often as you like costs nothing, which is the entire argument when boxes are torn down and re-pull everything.

pull is a cache, not a toll booth

The catalog stores each dataset’s sha256 and size. If the destination already exists and its hash matches, no bytes move. That makes repeated runs on the same box free, and it makes the remote store a cache rather than a single point of failure.

The catalog also records the generating recipe — seed plus source commit — mirroring the store’s own manifest. For state-only datasets, regeneration is sometimes cheaper than transfer: walker at 10 M transitions is 992 MiB but only a couple of minutes of CPU. That argument fails completely for motion-capture data, pixel datasets and exploration-driven collection, which is exactly why the bucket exists.

Pushing

var id = cat.push(
    String("data/walker.h5"),
    name=String("walker-random"),
    version=String("v1"),
    env_id=String("walker-walk"),
    n_rows=10_000_000,
    seed=7,
)

push is three steps — register, upload, complete — and they are separate on purpose. The catalog row stays pending between them, and describe refuses to serve a pending row. So an interrupted upload leaves a row that cannot be downloaded, rather than one advertising a truncated object.

Project recordings

A recording made under a project lives at projects/<project>/datasets/<dataset>/. The projects root is projects/ by default; NOEIRA_PROJECTS, from the environment or from .env, moves it. A recording is mirrored file by file, including while it is still being recorded:

pixi run dataset-push -- --project so101-tower --dataset cube-in-bowl --watch
pixi run dataset-pull -- --project so101-tower --dataset cube-in-bowl
pixi run dataset-list -- --project so101-tower

Every file is offered with its digest, and the server answers unchanged for one it already holds, so a --watch cycle costs one small request per file. The video still being written is held back and metadata goes last. The details are on LeRobot datasets.

Asset packs

Meshes and textures an environment needs can be declared as a pack in an assets.kv next to the environment, rather than tracked in git. LIBERO’s assets are the pack declared so far (noeira/envs/libero/assets.kv, 189 MB).

pack=libero@v1
provider=hf
url=https://huggingface.co/datasets/DenisLabs/mojo-rl-assets/resolve/main/libero_v1.tar.zst
sha256=8c7616b2…
bytes=188991117
dest=assets

pixi run assets-status          # what is on this box, what is not
pixi run assets-pull            # fetch everything declared
pixi run assets-pull libero     # only the assets.kv files whose path matches
pixi run assets-pull --force    # re-fetch even if present
pixi run assets-where           # the URL each pack comes from, and whether a token is sent

provider= decides which credential is sent, which a URL alone cannot tell:

Provider	Locator	Credential	How the bytes are reached
https	url=	none, or the variable named by token_env=	a plain GET
hf	url=	HF_TOKEN, or the hf CLI’s token	a bearer on the GET
noeira	id=	NOEIRA_CLOUD_API_KEY	catalog call → presigned URL → an unauthenticated GET

The archive is fetched to the cache, verified against its sha256, extracted once, and hard-linked into place (copied if a hard link is impossible, never symlinked). The cache is keyed by the archive’s hash, not the pack name, so two declarations of the same bytes share one download. It lives at ~/.cache/noeira/assets, or under NOEIRA_CACHE.

Run assets-pull as a preflight on a fresh box. A missing mesh found forty minutes into training is the failure it exists to prevent. One pack failing does not stop the others, and the command exits non-zero if any failed.

The downloader

noeira/io/fetch.mojo handles the transfer, on top of libcurl through the HTTP shim in noeira/io/http.mojo (build it once with pixi run build-http). The behaviour worth knowing:

• Resume by HTTP Range across a bounded number of retries, so a multi-GB transfer survives a dropped connection instead of restarting. A server that ignores Range answers 200 with the whole file. Appending that to a partial download would corrupt it silently, so those bytes are refused and the transfer restarts from zero.
• F_NOCACHE on macOS, periodic fsync + POSIX_FADV_DONTNEED on Linux, so a multi-GB write does not evict everything else from the page cache.
• No total timeout. The guard is on throughput instead: under 1 KB/s for 60 s is treated as a dead connection, at any file size.
• Downloads to <dest>.part, renames only on success. A crashed transfer must never leave a truncated file where the next run would treat it as complete.

It used to be Python

The loop was once a Python program run from Mojo. Driving its chunk loop through PythonObject leaked every chunk until the function returned, 23 GB at 34% of one download. The libcurl shim retired the question: the loop is C, writing straight to a file descriptor.

Working example

examples/data/dataset_remote_smoke.mojo round-trips a real store through the catalog: write locally → push → pull to a fresh path → reopen and verify the contents survived. It needs credentials, and it says so clearly if they are absent.

Deployment notes, if you stand one up yourself

Presigning requires an S3 API key pair — an in-worker bucket binding cannot presign. region: 'auto' is required by every S3 signer against R2, and omitting it breaks all calls. Presigned URLs cap at 7 days; return the expiry to the client rather than letting it discover a 403.

Do not attach a custom domain to the bucket. A presigned URL cannot use one — the signature binds to the host — and attaching a domain makes the bucket publicly readable, with no private variant. Object keys like datasets/<name>/<version>.h5 are entirely guessable, so that combination is the worst of both: no benefit in the data path, every dataset exposed.