Delta Weight Sync#
Delta weight sync keeps non-colocated rollout engines up to date by shipping only the bytes that changed between two syncs, instead of a full checkpoint each time. It targets large-model training/inference disaggregation across clusters or datacenters, where writing the whole actor every sync is the dominant cost.
Two transports are available: disk for shared-storage and cross-cluster deployments, and
sparse_hccl for online transfer between Ascend trainers and vLLM-Ascend rollout workers.
The first sync
sends every checkpoint coordinate through the public sparse protocol; later syncs
send only changed coordinates.
For disk transport, the trainer publishes each sync as a canonical HF checkpoint
directory; the engine’s pull_weights worker RPC (shipped in vime’s vllm patch) fans the
apply out to every host the engine spans and verifies it, then the engine reloads the
patched local checkpoint through the ordinary update_weights_from_disk adapter. vime
only ever talks to one endpoint per engine, so multi-node serving and external rollout engines
need nothing extra on the vime side.
Configuration#
Sparse HCCL:
With --offload-train --use-critic and without --colocate, training sleep
releases the weight-transfer connection. Reconnection preserves snapshots only
when the same rollout actors still report the last completed weight version;
otherwise it sends a complete seed. Receiver replacement, failed version queries,
and partial transmission failures require a fresh seed. BF16 seeds carry a four-byte
index plus a two-byte value per element, approximately three times the dense payload.
The resident-actor performance measurements do not cover this configuration.
This path uses the sparse HCCL engines included in vLLM-Ascend.
VIME exports weights into HF checkpoint coordinates, tracks shard snapshots,
gathers changed entries in a consistent order, and manages update versions.
Full export and sparse coordinate conversion use the same convert_to_hf() rules.
Coordinate conversion requires dense temporary buffers and output scans; temporary memory can grow with TP size. Supported weights are unquantized floating-point tensors with value-preserving conversion. Draft/MTP weights are unsupported. Validate sparse reconstruction against full export when adding a model layout.
--update-weight-mode delta
--update-weight-transport sparse_hccl
--update-weight-delta-batch-diff 32
--update-weight-delta-batch-gather 32
This path requires non-colocated, unquantized ND rollout weights and inference
PP=1, with EPLB disabled. Configure --vllm-pipeline-parallel-size 1.
The receiver handles TP/DP/EP loading; inference PP>1 is rejected before
communication initialization. Training-side PP/TP/EP export remains supported.
The script scripts/run-qwen3-30B-A3B-sparse-hccl.sh uses training
TP2/PP2/EP2 and rollout TP2/PP1/EP2, on four training plus two rollout NPUs.
Set DATA_ROOT to the directory containing weights/ and datasets/, and load
the NPU environment before running the script.
VIME normalizes contiguous payloads onto the communicator device and validates
them before invoking the trainer engine.
Disk transport configuration:
--update-weight-mode delta
--update-weight-transport disk
--update-weight-disk-dir /shared/fs/delta-updates
--update-weight-local-checkpoint-dir /local/nvme/rollout-ckpt
--update-weight-delta-encoding xor # or: overwrite
--update-weight-delta-checksum xxh3-128 # or: blake3, adler32
Table 1: Disk delta transport flags and their roles.
Flag |
Role |
|---|---|
|
Shared filesystem directory the trainer publishes deltas to and the rollout hosts read from. |
|
Host-local (e.g. NVMe) full HF checkpoint that |
|
On-disk delta encoding: |
|
Per-tensor integrity checksum: |
Deltas are always zstd-compressed (level 1); profiling showed it dominates lz4 / gzip / snappy / brotli on both wire size and decompress speed for this data, so it is not a knob.
How it works#
Sparse HCCL#
Initial updates and unverified reconnections use Seed; subsequent updates use Diff. An explicitly suspended connection resumes Diff only after matching receiver identity and version. Both paths share Publish and Apply.
Seed. On the first update or after an unverified reconnection, export all current training weights in HF format and produce complete checkpoint coordinates and target values for Publish/Apply. After successful seeding, capture local shard snapshots as the baseline for later diffs. Reconnection fully overwrites worker weights.
Diff. On later updates, compare current training shards with their snapshots bitwise. Convert changed positions and values into HF checkpoint coordinates using the same conversion rules as full export.
Publish. For Diff entries, first gather variable-length payloads to rank zero in a consistent directory order. For either Seed or Diff, rank zero validates bounds, duplicate indices and NaNs, normalizes device and contiguity, and splits payloads by the byte budget. The sender publishes metadata through
send_weight_chunkand runs receive RPCs and HCCL broadcasts concurrently.Apply. Rollout workers receive the checkpoint coordinates and target values broadcast through HCCL in Publish. The checkpoint loading path uses the model’s native loader to map these updates to each worker’s TP shards and local experts, then overwrites the corresponding weights in place with the received values, including zeros.
Wire indices are int32; each checkpoint tensor must contain fewer than 2**31
elements. The receiver requires full-shape NaN staging. Seed sends both indices
and values, so it uses more wire bytes than a dense values-only transfer.
The gather byte budget bounds aggregate receive scratch across trainer ranks. Oversized slots are split into element ranges. Input delta buffers, merged return values, local conversion probes and receiver staging are outside this budget; it is not a bound on total weight-update peak memory.
Disk transport#
Seed. The first sync, including after trainer recovery or an engine replacement, publishes a full HF checkpoint from the trainer’s current weights. It captures the CPU snapshot from those exact tensors and reloads every engine before generation resumes. The version exceeds every existing version directory, including an update whose reply was lost. Existing versions are retained; restarting does not delete the stream.
Publish. On every later sync the trainer diffs each gathered HF tensor against the snapshot, encodes and compresses the change, and writes a new version directory
weight_v{N:06d}/under--update-weight-disk-dir. The directory is a canonical HF checkpoint —model-NNNNN.safetensorsfiles holding the compressed diff tensors plus amodel.safetensors.index.json(tensor name → file) carrying the apply metadata — so the artifact is portable, not tied to the trainer’s parallelism layout. The snapshot is then advanced to the new values for the next diff.Pull. The trainer calls
pull_weightson each engine. Inside the engine the request is broadcast to every rank on every node; each host applies the new version’s delta into its local checkpoint in place (a per-host file lock collapses co-located ranks to one apply). The apply is parallelized across tensors and verified per-tensor (see Integrity); the call only reports success once every host holds a checksum-verified checkpoint.pull_weightsis not delta-specific: each published version is self-describing, and a version that is an ordinary full HF checkpoint (no delta metadata in its index) is pulled by copying it as-is — resetting the chain, so a fresh host joining late seeds from the newest full version instead of replaying every delta, and older deltas can be pruned. vime’s full-mode disk sync uses exactly this when--update-weight-local-checkpoint-diris set.Reload. The engines reload the patched local checkpoint through the vanilla
update_weights_from_diskpath — the weight-loading code never sees the delta format.
The snapshot and the engines start from the same published HF bytes, including when the
trainer restored a newer model than --hf-checkpoint. Megatron→HF conversion differences
(such as trimmed embedding / LM-head padding) therefore do not corrupt the next delta.
Encodings#
Both encodings are byte-level and dtype-blind, so the same path works for quantized checkpoints. The engine reads the choice from each version’s index metadata.
xor(default): writesnew ^ old. Smallest wire and fastest to apply (sequential, cache-friendly; the unchanged bytes are zeros the compressor crushes). It is an involution, so it must be applied exactly once against the correct base — applying it twice reverts.overwrite: writes the changed positions and their new absolute values. Larger on the wire and a less cache-friendly scattered apply, but idempotent: re-applying it (or finishing a partially-applied delta) converges to the same state regardless of how many times it runs. Use it when re-applicability matters more than wire size.
Integrity#
The trainer stores a per-tensor checksum of each tensor’s new state in the version. After
applying, every host recomputes the checksum and raises on any mismatch — the failure
propagates through the pull_weights response, so a corrupt delta or a wrong base fails loud
instead of serving bad weights. The apply also refuses to run out of order: a version only
applies on top of its declared base version.
--update-weight-delta-checksum selects the algorithm. The checksum is not the apply bottleneck
(the apply is decompress + XOR bound), so this is a digest-property choice, not a speed one:
xxh3-128 (default) is the widest fast non-cryptographic digest; blake3 is cryptographic, for
untrusted storage; adler32 is for interop with systems that expect it.