# o41-checkpoint

A Rust crate for training code: save, resume, fork and list checkpoints on a
local directory, a raw `s3://`, `gs://` or `az://` bucket, or `o41://`. It
speaks checkpoints and knows nothing about what their files mean. SexpGPU
checkpoints through it from its next release.

```rust
use o41_checkpoint::{open, step_name, Config, Name};

// The location picks the backend: a directory, s3:// gs:// az://, or o41://.
let checkpoints = open("o41://experiments/hoonshot/1234", Config::from_env()?).await?;

// Before the first step: the latest, or None on an empty location, and the writer.
let (resumed, mut writer) = checkpoints.resume().await?;
if let Some(at) = &resumed {
    let state = at.file("state.json")?.bytes().await?;
    let header = at.file("params.safetensors")?.range(0..8).await?; // lazy, ranged
}

writer.save(&step_name(510), files, meta).await?;

// A fresh start, refused on a location that has checkpoints.
let writer = checkpoints.start().await?;

// A fork into this empty location, from any checkpoint by its full path.
let (source, writer) = checkpoints.fork("o41://experiments/hoonshot/1234/step-00000300").await?;

// Readers.
let latest = checkpoints.load(Name::Latest).await?;
let listed = checkpoints.list().await?;
let note = checkpoints.note("evaluated.json").await?;
```

## Configuration

`Config::from_env()` reads:

| Variable                                      | Is                                                                                    |
| --------------------------------------------- | ------------------------------------------------------------------------------------- |
| `O41_ARTIFACTS_API_KEY`                       | An organization API key; an `o41://` location is refused without it                   |
| `O41_ARTIFACTS_URL`                           | The service, `https://artifacts.041.io` by default                                    |
| `O41_ARTIFACTS_CLOUD`, `O41_ARTIFACTS_REGION` | Override the detected place; see [placement](https://artifacts.041.io/docs/placement.md#finding-the-place-on-a-box) |

## Three backends, one contract

| Location                  | Backend                                  | Resume, start, fork                                                                     |
| ------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------- |
| `/path/run`               | Local filesystem                         | By listing; the manifest written last, by rename                                        |
| `s3://`, `gs://`, `az://` | `object_store` with your own credentials | By listing; no fencing. Also how you read your Artifacts buckets if the service is down |
| `o41://artifact`          | 041 Artifacts: the API, then signed URLs | Opening a writer; fenced                                                                |

| o41-checkpoint              | 041 Artifacts                                      |
| --------------------------- | -------------------------------------------------- |
| Checkpoint location         | Artifact                                           |
| Checkpoint, `step-00000510` | Version                                            |
| `resume`                    | open with mode `latest`                            |
| `start`                     | open with mode `new`                               |
| `fork(path)`                | open with mode `fork`                              |
| `save`                      | begin, the PUTs, commit                            |
| `load`, `list`, `note`      | `versions/read`, `versions/list`, `artifacts/note` |

- `step_name(510) == "step-00000510"`: zero-padded, so byte order is step order.
- Files are bytes or a path on disk; sizes are known before a save starts.
- Transfers: parts of 32 MiB or more, 16 in flight, ranged GETs in parallel,
  every upload aborted on failure.
- A save from a fenced writer fails with `Superseded`; stop, another box owns
  the run now.

---

Artifacts by 041 documentation. Every page: https://artifacts.041.io/llms.txt