041 · ARTIFACTS

Artifacts

Checkpoint storage for spot training, in every cloud

RUNNING ON SPOTREADY TO RESUME

write next to the GPU · read once · keep training

Train where GPUs are cheapest. Resume anywhere.Your own buckets in AWS, GCP, Azure or any S3 endpoint, as one checkpoint store. Every write lands next to the GPUs, so following spot capacity across clouds costs one read, not an egress bill.

Read the docsAgents: /llms.txt · curl any page

to write a checkpoint next to the GPUs

≈ $2

to resume a 20 GB checkpoint in another cloud

≈ $700

for a week of the same checkpoints sent to one central bucket

01 · The egress math

Write where you train. Read across clouds, rarely.

A run writes a checkpoint every half hour and reads one back only when it resumes on a new box. A central bucket bills you for the wrong half: every write leaves the cloud. Artifacts puts each write in your bucket next to the GPUs, for free. The only bytes that cross clouds are the one checkpoint a resumed run reads. Nothing is replicated ahead of time.

Where the writes go	Per GB	For the run
A bucket in the box's own region	$0	$0
Same cloud, another region	≈ $0.02	≈ $130
Out of the cloud: another cloud, Tigris, R2	≈ $0.08–0.12	≈ $550–800

The run: a 20 GB checkpoint, weights plus Adam state, every 30 minutes for a week. 336 checkpoints, 6.7 TB written. Each cross-cloud resume costs about $2. Approximate list prices.

02 · Built for spot

Your run hops clouds. Its checkpoints follow.

01aws us-east-1
step 100 … 500
Writes to S3 in us-east-1. Preempted.
02gcp europe-west4
step 510 … 900
Reads step 500 from AWS, once. Writes to GCS in europe-west4.
03azure westeurope
step 910 …
Reads step 900 once. Writes to Blob in westeurope.

Spot GPUs cost a fraction of on-demand because the cloud can take them back. Let it. When a box is preempted, relaunch the same command wherever capacity is free. --resume latest hands the new box the latest checkpoint and locks out the old one in case it is still alive. One read crosses clouds; every checkpoint after that lands in your bucket in the new region.

The run here is SexpGPU: the world's first ML framework designed for agents, and ours. One experiment file, one binary for any GPU box, and a --resume latest made for spot. Pointing it at o41:// takes one environment variable.

zsh

# The only change to a run: the location string.
$ export O41_ARTIFACTS_API_KEY=ak_…
$ SEXPGPU_CHECKPOINT_DIR=o41://experiments/run-1234 \
    sexpgpu run train.sx --resume latest
# Preempted. Relaunched on GCP spot. Same command.
resumed step-00000500 from aws us-east-1 (cross-cloud)
saved step-00000510 to gcp europe-west4 (region)

03 · Four ways in

SexpGPU, Rust, the shell, or plain HTTP.

Every operation comes down to PUTs and GETs straight against S3, GCS, Azure Blob or an S3 endpoint, each URL signed for one object. Anything that speaks HTTP can read and write checkpoints, and no box ever holds a bucket key.

O41-CHECKPOINT · RUST

use o41_checkpoint::{open, step_name, Config};

let checkpoints =
    open("o41://experiments/run-1234", Config::from_env()?).await?;

// The latest, or None on an empty artifact, and the one writer.
let (resumed, mut writer) = checkpoints.resume().await?;
if let Some(at) = &resumed {
    let state = at.file("state.json")?.bytes().await?;
}

writer
    .save(&step_name(510), files, meta)  // lands in this box's region
    .await?;

CLI · @O41/ARTIFACTS

npx @o41/artifacts login
npx @o41/artifacts ls experiments/
npx @o41/artifacts versions experiments/run-1234
npx @o41/artifacts get o41://experiments/run-1234/latest ./ckpt

HTTP · ANY LANGUAGE

curl -H "Authorization: Bearer $O41_ARTIFACTS_API_KEY" \
  "https://artifacts.041.io/api/v1/versions/read?artifact=experiments/run-1234&version=latest&urls=1"
# → { "version": "step-00000900", "transfer": "cross-cloud",
#     "files": [{ "name": "params.safetensors", "url": "https://…" }] }

04 · A control plane, not a store

Your buckets, your bill

Bytes never pass through us. Boxes talk straight to S3, GCS, Azure Blob or any S3 endpoint, over URLs signed for one object.

No cloud keys to leak

Clouds connect by OIDC federation, so there is no long-lived key to steal. Revoking access is deleting one role.

Training never stops

No bucket in this region? The write goes to the cloud's default, then your fallback, with a warning. It costs some egress, never the run.

Self-describing buckets

Every version carries a _manifest.json beside its files. If we are ever down, read your bucket with your own keys.

Bring your buckets.

Connect a cloud, pick a fallback, point a run at o41://. Questions: niko@041.io