Repository navigation
feat: remote CXL pool storage backend for nodes without CXL - #98
youngrok-XCENA wants to merge 68 commits into
Conversation
A pool node exports its Maru CXL regions to handlers on other nodes. maru-remote-server owns a CXL-backend MaruHandler, registers the mapped regions with NIXL and serves sized page reservations for remote WRITE, publish, existence checks, protected lookups for remote READ, and quarantine for pages a timed-out WRITE may still reach. Every reply carries the server's start-up generation so clients notice a restart. Keys an earlier server run left in regions this run does not own are reported missing and replaced by the next store. The NIXL wrapper hands back a still-running transfer after its deadline instead of releasing it, so callers keep its buffers out of circulation until it ends. NIXL is an optional dependency: pip install 'maru[remote]'.
MaruConfig(storage_backend="remote") keeps KV in another node's CXL pool. The backend implements the CPU backend's handler contract: alloc returns a staging slot, batch_store reserves pool pages, RDMA WRITEs the slots and publishes the keys, and batch_retrieve pins, RDMA READs into staging, unpins and returns read leases the caller releases after its copy. Keys are scoped by cache_namespace, so engines with the same namespace share. After a failed control call the backend stops calling the server for remote_retry_s (lookups miss, stores are skipped, loads fail and are recomputed) and reconnects afterwards; a new server generation reloads the NIXL peer and forgets locally known keys. Staging slots of a timed-out transfer stay isolated until NIXL reports its end, and a timed-out WRITE's pool pages are quarantined on the server until then. The vLLM connector selects it with maru_storage_backend="remote" and the maru_remote_* settings and runs it on the lease path #88 built for CPU storage. That path's CPU-specific names now describe what they are (_lease_mode, _bind_lease_namespace, _unkeyed_request, _external_load_range); CPU error messages are unchanged.
Adds the remote backend's launch commands, settings, failure behaviour and trust model, a vLLM integration guide section next to the CPU and mixed modes, and two hardware tools: remote_storage_g0.py checks byte round trips, duplicate handling and RDMA bandwidth through MaruHandler, and remote_storage_g2.py drives two vLLM workers to show cross-worker reuse.
The per-layer packed-store fallback (used by the CPU, mixed and remote backends, and by CXL when the maru_kv_ops kernel is unavailable) copied a layer's [2, tokens, hidden] extract into slab[:, layer] in one copy_. That destination is strided across layers, and the GPU->host copy into it took about 6 ms per (chunk, layer): 1.7 s for 12 chunks x 24 layers of Qwen2.5-0.5B on an RTX PRO 6000. Each plane slab[k, layer] is contiguous, so two copies per layer write the same bytes in about 20 ms in total. The same change also fixes a cross-server visibility problem on shared CXL: the strided copy is staged through CPU stores that can stay in the writing server's cache, so another server read stale bytes in a byte comparison between two servers sharing one CXL device, while plane-wise copies matched.
…amespace Lease-backed loads (cpu, mixed, remote) retrieved every chunk of every request in a step at once. A backend with bounded read buffers (the remote backend's staging slots) then failed long or concurrent loads although the scheduler had reported hits. Each request is now loaded in batches of at most MaruHandler.retrieve_capacity() chunks, each released before the next; a batch that cannot be read fails the request from that batch on. With vLLM's asynchronous scheduling a failed synchronous load crashed the engine (device-side assert in the next step on vLLM 0.22.1, reproduced on hardware); with --no-async-scheduling the same failure recomputes. The lease path now refuses to start with async scheduling enabled. The sharing namespace no longer hashes the local model path, _name_or_path or transformers_version, so nodes that load the same weights from different paths share KV; the resulting value is logged at start-up. The remote staging buffer defaults to at least 64 KV objects. A chunk whose slab cannot be allocated in a step is skipped for that step's remaining layers instead of re-allocating a slab that can never complete, and expected skips (full or unavailable backend) are logged at most once a minute.
- Requests carry the server generation the client connected to; the server refuses requests addressed to an earlier run without executing them, so the client's single retry after a restart is safe (a retried lookup used to collide with the ticket the first attempt had pinned). Each lookup attempt uses a fresh ticket. - Keys a worker remembers storing are confirmed with a ping once it has not heard from the server for 5 s; a restart forgets them so they are stored again instead of staying misses. - Abandoning quarantined pages waits for the cool-down and trips the breaker on timeouts; close() abandons pages of WRITEs that have ended; the server frees quarantined pages never abandoned after --quarantine-ttl (600 s). - A full pool answers with code POOL_FULL; the worker skips stores, not loads, for the retry period. - A stale key the server cannot delete (pinned by a dead run) is REJECTED rather than reported stored. The server warns about pool regions it does not own at start-up. - Staging is allocated after hello, and local setup failures are configuration errors; read-protection lifetime must also cover two transfer timeouts. Buffer and lease types move to remote_buffers.py.
…period A failed remote handler connection costs a control timeout, and the connector retried every 5 s, so while the pool was down one request in every 5 s waited for it. The remote backend now waits maru_remote_retry_s (30 s by default) between connection attempts, like its own breaker; other backends keep 5 s.
…s full, hold refused keys - The lease-path load stops at the first batch whose GPU copy fails and reports the rest of the range, instead of reading later batches vLLM will recompute anyway. - The pool server reports POOL_FULL only for allocator exhaustion; other allocation faults are plain errors. - A key the pool refuses to publish is not written again for the retry period (cleared on reconnect or restart). - Document the read-protection lifetime check, refused keys and the untested restart-with-pending-transfer ordering; correct the batch_store ownership comments.
The per-layer packed-store fallback (used by CPU and mixed mode, and by CXL when the maru_kv_ops kernel is unavailable) copied a layer's [2, tokens, hidden] extract into slab[:, layer] in one copy_. That destination is not contiguous: other layers sit between the layer's K and V planes. Measured on its own (save_kv_layer over 24 layers, Qwen2.5-0.5B geometry, 12 chunks, one RTX PRO 6000, pageable host buffer), the store took 1.73-1.81 s, about 6 ms per (chunk, layer). Each plane slab[k, layer] is contiguous, so two copies per layer write the same bytes in about 21 ms in total.
…only Remote loads fail in normal operation (pool outage or restart), and with async scheduling a failed synchronous load crashes the engine. CPU and mixed mode keep main's start-up rules: their loads fail only on metadata server faults, which does not justify giving up async scheduling. The CPU/mixed docs and examples are restored to main.
The remote backend shared the CPU/mixed lease path, which only copies synchronously, so remote loads and stores ran inside the step and the engine had to give up async scheduling. The lease rules (stored keys are re-checked in the pool, external token ranges, leases released after the copy) stay; the synchronous-only restriction now applies to cpu and mixed alone: - The remote staging buffer is page-locked for CUDA, so the packed load and store kernels and async copies use the slots directly (MaruHandler.storage_gpu_accessible). - maru_async_load and maru_async_store are accepted for remote. A parked request is loaded on the loader thread in batches the staging buffer can hold, each released once its copies completed, and unparks only after the last batch or with its failure. - --no-async-scheduling is required only when remote loads are synchronous; an async load reports failures before the request is scheduled. - A partial chunk (strided token slice) uses the per-layer copy; the kernel reads whole chunks.
…ansfer mode An async store stages every chunk of a prompt at once, so a staging buffer smaller than the longest prompt skipped the excess chunks. The default staging size now also covers one max_model_len prompt. Expected skips are logged at most once a minute, the remote transfer mode (loads, stores, vLLM async scheduling) is logged at start, and a deferred leased load logs its batches.
vLLM asks the connector again for a request whose asynchronous load failed from its first block. The keys are still in the pool, so the same hit was reported and the failing load retried without end (seen with an injected retrieve failure: the request never completed while the engine stayed healthy). A request handed an async lease-backed load now gets no external tokens on the second ask and is recomputed.
alloc, free and lease release took the backend lock that a store or load holds across its control requests and RDMA transfer. With async loads and stores on other threads the engine thread then waited for those transfers: during a pool outage one allocation waited ~2 s behind a timed-out reserve. Slot bookkeeping now has its own lock (the staging free list is locked internally), and alloc only returns the slots of ended timed-out transfers when the transfer lock is free; returning their pool pages stays with stores and loads.
The async store asks has_local on the engine thread before each chunk. It took the transfer lock (and could ping), so during a pool outage a request waited ~2 s behind another thread's timed-out reserve. It now answers from the remembered keys, and when a ping is due but the transfer lock is busy it answers False; the store is then retried and the pool reports the key present.
Maru has no eviction, so a full pool stopped admitting new KV. The remote server knows every key it published and when it was last read, so it now evicts: when a reservation would exceed --capacity (or the allocator runs out), it deletes the least recently read published keys, skipping pinned ones (being read), and retries. --eviction none keeps the previous behaviour (POOL_FULL, the client pauses stores). Stats report used and capacity pages, LRU keys and evictions.
A worker remembered only the keys it had stored itself. After a pool hit it stored the same prefix back: it reserved pages (which can evict other keys), wrote the bytes over RDMA, and the server dropped them as already present. - Keys read from the pool in this server run are remembered too, and keys a lookup no longer finds are forgotten. - Every server reply carries the number of keys evicted in this run. When it changes, the worker forgets its remembered keys, so an evicted key is stored again instead of being skipped for the rest of the run. - A store first asks which keys are present and writes only the others, so keys stored by another worker, or forgotten after an eviction, cost one control round trip instead of a page reservation and a WRITE.
After any eviction a worker forgot every key it remembered. A full pool evicts on almost every store, so each pool hit then copied its whole prefix back to host staging slots before the server reported it present, and those slots were missing for stores of new prefixes. The server keeps the latest evicted keys with their eviction numbers and answers evicted_since. A worker's load and store threads ask for the keys evicted since the last number they saw, only when a reply reports new evictions, and drop just those keys. When the server no longer lists all of them, the worker forgets everything as before. The engine thread never makes this call; a remembered key may be stale until the next load or store.
has_local is asked on the engine thread before every store. After a few seconds without contact it pinged the server there, so a dead server could stall the engine for the control timeout. The scheduler's existence check also reconnected inline once the retry period ended, stalling the scheduler for the timeout every retry period while the server stayed down. A maintenance thread per remote backend now reconnects when a reconnect is due, confirms the server run after silence (forgetting remembered keys on a restart) and drops keys the server reported evicted. has_local answers from memory only, and the existence check reports misses while a reconnect is pending instead of attempting it.
The maintenance thread pinged and reconnected while holding the I/O lock, so with the server down the scheduler's existence check waited for its 2 s timeouts every retry period. An existence check that met a restart also reconnected inline on the scheduler path. The maintenance thread now pings once a second with its own control client and no lock held. A failed ping stops calls for the retry period; a changed server run or a pending reconnect reconnects under the lock only after the server has answered. The ping reply carries the eviction count, so an idle worker drops evicted keys within a round. Existence checks that meet a restart report misses and leave the reconnect to that thread.
Loads and stores still reconnected inline once the retry period ended: with the server down, the store thread waited for a hello timeout under the I/O lock and re-opened stores in between. The maintenance thread did not probe while calls were stopped, so after the server returned one role stayed stopped until its retry period ran out. The maintenance thread now probes every round, also while calls are stopped: a failed probe keeps calls stopped, and the first answer reconnects at once. Loads, stores, stats and ping report the server unavailable while a reconnect is pending. Probes use a 500 ms deadline so an outage is noticed within about 1.5 s. A restart resets the eviction mark under the eviction lock.
The server serves one request at a time, and some requests hold it for a while (mapping a new region while reserving pages took 1.7 s). A single probe that missed its 500 ms deadline then stopped calls on every worker for a retry round. Two timeouts in a row now stop calls, so a server busy for up to about 2 s is not an outage; an outage is noticed within about 3 s. An error reply to a probe is logged without stopping calls, and a server whose layout cannot serve this handler is retried after the retry period instead of on every probe.
An error reply between two probe timeouts did not reset the count, so the second timeout stopped calls. A pending reconnect (an existence check had met a restart) also stopped calls on the first slow probe. Now an error reply resets the count, and a single timeout stops calls only while they are already stopped. Tests cover both and the retry period after a layout error.
…ransfer A probe tests only the control channel. After an RDMA READ or WRITE timed out or failed, a successful probe cleared the stop within a second, so a broken data path with a live control channel made every load wait for the transfer timeout again. A failed transfer now keeps calls stopped for the whole retry period; probes still detect a restart or a layout change.
Adding a region blocks the remote endpoint for about 21 ms per GiB. Workers wait for it, and past about 2 s they stop calling the pool, but they reconnect at the first answered ping, so the cost is a few seconds of misses and recomputed requests. Say that instead of telling users to keep growth regions well below 100 GiB.
…e-kv-backend Both sides changed MaruServer.request_alloc and its CLI: keep the remote access options and managed owners next to the client leases. Keep this branch's names in the vLLM connector (_lease_mode, _unkeyed_requests).
remote_storage_g0.py carried a development milestone name; it is now remote_storage_roundtrip.py. remote_storage_g2.py, the first two-worker reuse check, is removed: examples/vllm/remote/run_benchmark.py does the same check against the current server.
A request parked for an async load holds blocks for its loaded prefix only, and resumes only once its whole prompt fits. vLLM checks that fit for each request against the blocks free when it parks, so parked requests can together hold the cache while the oldest still needs more. With none of them running, no block is ever freed and the engine stops until clients abort requests. The scheduler side now tracks the blocks each parked request needs for its whole prompt and asks vLLM to retry a request later (None from get_num_new_matched_tokens) when parking it would overcommit the cache. Requests told to wait keep their place ahead of later ones, so a long prompt is not passed over.
vLLM keeps one GPU KV block aside as its null block, so requests can hold one block fewer than kv_cache_config.num_blocks. Size the budget on the blocks requests can actually hold.
…kv-backend #96 no longer carries the client lease commit; its other changes are already here, so the merge changes no file.
#96 removed its client lease commit (aac1426): reclaiming a region at lease expiry also reclaims the regions of a live handler that only stalled, and the handler's late write can land in KV that another handler stored in the reused range. This branch got the lease through the #96 merge; remove it here too. The managed-owner checks for remote access stay.
ce95214 to
4c7e6a6
Compare
Every remote handler's maintenance thread pinged the pool once a second, also while its calls succeeded, so the pool's one remote endpoint thread served a ping per handler per second for nothing: every reply already carries the server run and the eviction count. The thread now pings only while calls are stopped or a reconnect is due, and reconnects at the first answer as before. A healthy round calls the pool only when needed: it applies evictions a reply reported (the scheduler's existence check does not apply them itself) on the probe client, and when the engine skipped stores on remembered keys with no reply since the previous round, it asks once to confirm the run. A probe that meets a new run reconnects at once, and probe timeouts count like failed pings. An idle handler now learns of an outage, a restart or an eviction at its next call. The first call after an outage waits for the control timeout once; a handler that only skips stores learns within a round.
…room vLLM asks a request that the parked-load budget told to wait again on every step. Each ask ran the prefix match first, which sends an existence check to MaruServer (every time in CPU mode, and in CXL mode unless every chunk is a known key) and logs a "Maru KV hit" line, once per waiting request per step. A request already waiting now checks the budget first and answers from it alone until the budget has room; then the usual lookup runs. A waiter whose prefix disappeared meanwhile computes once it reaches the front instead of on the next step. The async-load docs and the maru_async_load docstring now describe the budget.
After a failed or timed-out control request the backend stops calling the pool only until the maintenance thread's ping is answered; the full remote_retry_s wait applies to a failed RDMA transfer (unless the pool restarted), a full pool, a refused key and a failed start-up connection. The vLLM integration page, the remote example README, the MaruConfig docstring, the handler design doc and the _trip docstring and log said every failure stopped calls for remote_retry_s. The README also still said fault recovery awaited hardware validation. Also from review of the pull request: - Drop OwnedRegionManager.add_region(prefault=): remote access maps the region just before, so map_region returned the existing mapping and the flag changed nothing. The test now checks that no prefault runs. - Fix stale docstrings (maintain, DaxMapper.map_region, _unkeyed_request) and the maintenance-thread arrow label. - Restore an unrelated comment in the per-layer store and an unrelated sentence in the architecture overview. - Move the round-trip tool next to the remote example (tools/ holds the marutop shims) and read the transfer counters through get_stats().
…_kv_at Remote access is its only caller and runs only with CXL allocation. The assert matches evict_kv_at and keeps the type checker from flagging the decrement on an optional allocation manager.
…ocks The block budget removed a request from the wait list as soon as the budget let it through. The budget does not count blocks held by running requests, so vLLM could still refuse the request blocks that step. Asked again next step, the request was treated as new and queued behind every later waiter, so a long prompt was passed over by each later short one. Remove the entry when vLLM gives the request blocks (update_state_after_alloc), when the lookup finds nothing to load, or when the request finishes. Add a real-scheduler test where a running request holds the blocks a let-through request needs, and a connector test of the same path; both fail without this change on vLLM 0.22.1 and 0.23.0.
Two store paths dropped the tickets of reserved pages without publishing or abandoning them, so the pool quarantined the pages at reservation expiry and kept them until shutdown, even though the worker was alive: - the WRITE ended, but a reconnect happened meanwhile or another thread had stopped calls (for example a lookup that timed out); - the publish request timed out, so it is unknown whether it ran. Keep those tickets in the quarantine instead. It abandons them once the server answers again; the pool skips tickets it already published, and the pages of an earlier server run are gone with it. Tests cover both paths and a publish that ran before its reply was lost.
The pool marks a key used when it is stored as well as when it is read, so "least recently read" understated what eviction looks at. Use "least recently used (stored or read)" in the option help, docstrings, log line and docs.
After a WRITE that ended in an error state, or a publish the server refused, the store abandoned the pages at once and dropped the tickets if that abandon failed, so a pool that did not answer just then kept the pages until shutdown. Abandon now; on a timeout or a failed socket, keep the tickets for _reap, which abandons them once the server answers. A reply from a new server run drops them (the earlier run's pages are gone), and so does an abandon the server refuses, in _reap as well, since a retry cannot change a refusal and would repeat every round.
Preserve remote load failure reporting and partial-chunk handling while incorporating the pinned/pageable slab fallback from main. Validation: 1343 passed, 4 skipped with CUDA and native extensions; Ruff 0.16.5 lint and format checks passed.
Give the remote endpoint its own ZMQ context and terminate it after closing the socket, including when bind fails. This waits for libzmq cleanup without interrupting other endpoints or clients. Verify immediate port reuse and retain the existing bind-failure cleanup check. The focused restart and cleanup cases passed 150 repetitions; full suites passed on Python 3.12 with CUDA and Python 3.14 with CI dependencies.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 909947fed0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for name in ("remote_transfer_timeout_s", "remote_retry_s"): | ||
| value = getattr(self, name) | ||
| if ( | ||
| isinstance(value, bool) | ||
| or not isinstance(value, int | float) | ||
| or value <= 0 |
There was a problem hiding this comment.
Reject non-finite values for remote_transfer_timeout_s, not just values <= 0. For example, MaruConfig(remote_transfer_timeout_s=float("nan"), ...) passes this validation, but _xfer() then computes a NaN deadline and time.perf_counter() > deadline is always false; any NIXL transfer that remains in PROC will therefore poll forever instead of timing out, potentially leaving an engine request permanently stuck. Infinity causes the same unbounded-wait behavior, so this should require a finite positive number.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This PR adds a remote storage backend that lets nodes without CXL share a pool node's CXL memory over RDMA. The design and failure-path handling are careful overall; I have four points: lost cache hits for preempted requests, unbounded growth of _rejected, a staging default that scales with max_model_len, and an empty load range being treated as a failure.
| self._load_starts.pop(rid, None) | ||
| self._load_ranges.pop(rid, None) | ||
| self._cpu_bypass_requests.discard(rid) | ||
| self._unkeyed_requests.discard(rid) |
There was a problem hiding this comment.
🟡 minor — _async_load_issued is missing from the preempt cleanup path. The stale-cleanup loop in build_connector_meta clears all per-request state (_load_starts, _load_ranges, _unkeyed_requests, _requests_need_load, ...) for finished_req_ids | preempted_req_ids, but _async_load_issued is only discarded in request_finished. A preempted request does not finish; it goes back to the waiting queue, so its id stays in the set. When it is rescheduled, get_num_new_matched_tokens hits the branch at line 1388 and returns (0, False). The request then recomputes its whole prompt even though the keys are still in the pool, and the log shows a misleading "returned after a failed load" warning. A failed deferred load is reported through a separate path (get_block_ids_with_load_errors), not through preemption, so adding self._async_load_issued.discard(rid) to the stale loop still prevents the infinite retry this set was meant to stop.
| self._stored.add(keys[i]) | ||
| self._rejected.pop(keys[i], None) | ||
| else: | ||
| self._rejected[keys[i]] = self._clock() + self.config.remote_retry_s |
There was a problem hiding this comment.
🟡 minor — Nothing removes expired entries from the _rejected dict. An entry only goes away when the same key later succeeds (_rejected.pop) or on reconnect (clear()). The skip check in _store_locked is self._rejected.get(key, 0.0) > now, so expired entries are never used again but stay in the dict. REJECTED comes from keys that the pool node's local client already holds in its own region, so in a deployment where the pool node shares a namespace with a local vLLM, one entry accumulates per distinct chunk key and the dict grows without bound in a long-running worker. Consider dropping entries past now in _reap or the maintain round, or capping the size.
| prompt_chunks = -(-int(extra_config.get(_MAX_MODEL_LEN_KEY, 0)) // ct) | ||
| reserve = float(extra_config.get("maru_remote_load_reserve", 0.5)) | ||
| store_slots = math.ceil(prompt_chunks / (1.0 - reserve)) | ||
| return max(DEFAULT_REMOTE_STAGING, 64 * page_bytes, store_slots * page_bytes) |
There was a problem hiding this comment.
🟡 minor — The default staging size grows with max_model_len and has no upper bound. It is store_slots = ceil(max_model_len / chunk_tokens / (1 - load_reserve)) times page_bytes, so Llama-3.1-8B (32 MiB objects at 256-token chunks) with --max-model-len 131072 defaults to 512 chunks / 0.5 = 1024 slots × 32 MiB = 32 GiB. The whole buffer is page-locked by cudaHostRegister in StagingBuffer.cuda_register() and by NIXL registration, so it is actually committed. If host RAM is short, startup silently falls back to pageable memory (losing the kernel path) or, worse, registration fails. It would be safer to cap the default and warn the user to set maru_remote_staging_size explicitly when the cap is hit, or at least log the computed size before allocating.
| first = start // ct | ||
| capacity = handler.retrieve_capacity() | ||
| step = len(chunk_keys) if capacity is None else capacity | ||
| if step <= 0: |
There was a problem hiding this comment.
🟡 minor — _load_leased treats an empty external load range as a failure instead of a success. When start == end (e.g. update_state_after_alloc receives num_external_tokens=0 while num_matched_chunks > 0) and start is on a chunk boundary, first_chunk == last_chunk, so chunk_keys is empty. On backends where retrieve_capacity() returns None (cpu/mixed), step = len(chunk_keys) = 0 and the step <= 0 guard fires. The request then goes to _fail_deferred_load with an unrelated "no read buffer free" warning, whereas the cpu path before this PR passed an empty key list through and marked the deferred load done. Handling if not chunk_keys: before the guard as a normal completion (registering _deferred_done if needed), and keeping step <= 0 for the case where there are keys but no free slots, would fix it.
seohui-XCENA
left a comment
seohui-XCENA
left a comment
There was a problem hiding this comment.
Thanks for the thorough work and the detailed write-up. The failure handling is careful, and on the pool side I found no lock-order or eviction-atomicity problems. I'm requesting changes for the issues below, which I could reproduce at 909947f with unit-level tests (the in-process pool with the fake NIXL agent; the real vLLM 0.23.0 scheduler for the connector issue). Several are about performance: for this backend, stalls and repeated recomputes matter as much as correctness. Details are inline.
Must fix before merge
- Async remote loads ignore the external load range (
maru_vllm/connector.py) - Reserved tickets are dropped when the metadata refresh fails (
maru_remote/client.py) - A multi-frame request stops maru-server (
maru_server/remote_access.py) - A failed region growth skips LRU eviction (
maru_server/remote_access.py) - A partly failed reconnect keeps the previous run's key hints (
maru_remote/client.py) - A store keeps making control calls after calls were stopped (
maru_handler/storage/remote.py) - Every evicting reserve scans all keys (
maru_server/remote_access.py)
Suggestions
- While the pool is unreachable or the store retry window is open, the synchronous store path (
_store_packed_slabs) logs an ERROR for every chunk on every step. RoutingMemoryError/StorageUnavailableErrorfromallocthrough_note_store_skipwould match the once-a-minute warning described in the PR. - Please state the single-pool scope in the docs.
| if self._cpu_mode: | ||
| if self._lease_mode: | ||
| start = self._load_starts.pop(request.request_id, 0) | ||
| self._load_ranges[request.request_id] = (start, start + num_external_tokens) |
There was a problem hiding this comment.
Must fix. This range never reaches an async load. The deferred MaruReqMeta built in build_connector_meta (around L1504) doesn't pass load_start_token / load_end_token, and _load_ranges is cleared at the end of that call (L1661). So _external_load_range returns (0, ...): the async lease load reads from chunk 0, writes into the request's GPU prefix-cache blocks, and on failure _fail_load reports those shared blocks as invalid.
With the real vLLM 0.23 scheduler and prefix caching, a failed async remote load reported {1, 6, 7} as invalid, where block 1 is shared with a decoding request, and that request's computed tokens went from 18 to 0. The sync path emits (4, 12) for the same case. Passing self._load_ranges[req_id] into the deferred meta would fix it, together with a test in which the scheduler itself builds the deferred meta with num_computed_tokens > 0.
| client_id=self.client_id, | ||
| sizes=list(sizes), | ||
| ) | ||
| self._refresh_if_stale(int(reply["md_version"])) |
There was a problem hiding this comment.
Must fix. If _refresh_if_stale raises here (for example, the metadata call times out while the endpoint is registering a new region, or add_peer fails), the pool has already created these reservations, but the tickets never reach _store_locked, which only calls _fail. After the reservation TTL, the pages are quarantined and nothing can abandon them, so they stay unusable until pool shutdown. Unlike the disclosed lost-reply case, the worker did receive the tickets here.
Reproduced with an injected refresh failure: reservations=1, then quarantined=1, with the worker's _quarantine empty. Returning the pages and letting _store_locked handle a refresh error with _abandon_or_keep would keep them. lookup() (L237) has the same shape and leaves the pin until the ticket TTL; releasing the ticket there would help too.
| last_sweep = time.monotonic() | ||
| while not stop_event.is_set(): | ||
| if poller.poll(timeout=int(sweep_interval_s * 1000)): | ||
| raw = sock.recv() |
There was a problem hiding this comment.
Must fix. recv() reads only the first frame. If a request has more frames, the REP socket is not ready to send, so send() raises ZMQError(EFSM). Nothing catches it, and _RemoteEndpoint.run stops the whole maru-server (exit 1), taking the local clients' metadata server with it. Reproduced with a REQ socket sending [encode("ping"), b"extra"]. Using recv_multipart() and rejecting anything but a single frame, and catching per-request errors so that only bind and poll failures are fatal, would fix it.
| break | ||
| if self._capacity_pages is not None and free + used >= self._capacity_pages: | ||
| break | ||
| if not self._add_region(): |
There was a problem hiding this comment.
Must fix. _add_region() returns False only when the RM refuses. A map or NIXL registration failure re-raises out of _op_reserve, so _evict below never runs and the reply is a generic error instead of POOL_FULL. With a full pool of evictable keys, every store then fails and repeats the growth attempt (RM alloc, mmap, failed registration, unmap) under the RemoteAccess lock, and the worker applies no retry backoff because the error isn't POOL_FULL. Treating a failed growth as "cannot grow" and falling through to _evict would fix it.
Related suggestion: with the default unlimited --remote-capacity, every reserve on a full pool also asks the RM for a new region before evicting. Remembering that the RM refused, and backing off, would keep that request off the steady-state store path.
| with self._md_lock: | ||
| if self.peer and str(reply["generation"]) != self.generation: | ||
| self._forget_peer() | ||
| self.generation = str(reply["generation"]) |
There was a problem hiding this comment.
Must fix (performance). The new run's generation is recorded here, before _apply_metadata loads the peer. If that load fails once after a pool restart, the next _ready() sees the same generation and takes the same-run path, so _stored, _rejected and _evictions_seen from the old run are kept. has_local() then stays True for keys the new run doesn't have, and the connector skips storing them for the rest of that run, so those prefixes are recomputed on every request. Reproduced by failing add_remote_agent once. Recording the generation only after _apply_metadata succeeds would fix it.
| if not self._ready(connect=False): | ||
| return [False] * len(keys) | ||
| assert self._staging is not None | ||
| self._sync_from_io() |
There was a problem hiding this comment.
Must fix (performance). If _sync_from_io() stops calls (for example, evicted_since times out), the store still sends exists. A synchronous store, which is the default, can then block the engine thread for two control timeouts (about 4 s) instead of one. Reproduced with a 300 ms timeout: the store took 0.60 s. Re-checking _ready(connect=False) right after _sync_from_io(), here and in _retrieve_locked (L1005), would bail out early.
| """ | ||
| if not self._evict_enabled or n <= 0: | ||
| return 0 | ||
| locations = [ |
There was a problem hiding this comment.
Must fix (performance). This builds a tuple for every key in the LRU each time a reserve evicts, although only n are needed. Once the pool is full, that happens on every store, on the endpoint's single thread under the RemoteAccess lock, so every worker's control requests wait. The same comprehension took about 52 ms with 256K keys (a 1 TiB pool with 4 MiB pages). Walking the LRU lazily and stopping once n unpinned keys are found (or letting evict_at take an iterator) would keep the cost proportional to n.
| and `batch_retrieve` returns read leases that the connector releases after its | ||
| copy. Unlike CPU mode, the staging buffer is page-locked for CUDA, so the | ||
| connector uses its default transfer path: the `maru_kv_ops` copy kernels (when | ||
| built) and asynchronous loads and stores. The remaining limits are CPU mode's: |
There was a problem hiding this comment.
Suggestion: could these limits also say that a worker uses exactly one pool node (maru_remote_url), and that pool nodes are not combined, so workers attached to different pool nodes don't share KV? The topology implies it, but it isn't stated as a limit. The same applies to the remote section of docs/source/integration/vllm.md.
| # the weights. They differ between nodes serving the same model and must not | ||
| # split the sharing namespace; the caller's maru_cache_namespace names the weights. | ||
| _NODE_LOCAL_CONFIG_FIELDS = frozenset( | ||
| {"_name_or_path", "name_or_path", "transformers_version", "_commit_hash"} |
There was a problem hiding this comment.
Question: dropping transformers_version here doesn't seem to enable sharing across transformers versions in practice, because hf_config.to_dict() itself differs between versions (for example, 5.12.1 emits dtype where older releases emit torch_dtype). Nodes on different versions would still get different digests and silently miss. That only splits the cache and never shares wrongly, but is cross-version sharing a goal here? If so, the digest may need a normalized subset of fields.
Thanks for the review. I think I need to refine the user scenarios and the high-level design. I'll let you know when the work is finished. |
This branch has not been deployed
3 participants
🤔 Background & Motivation (Why)
Today, Maru can share KV only among nodes with a CXL device attached. An engine on a GPU server without CXL redoes prefill from scratch, even for a document that another server has already computed.
cxl) keeps KV in CXL shared memory, and engines on nodes that map the same CXL device read and write it. A CXL device is visible only in the memory space of the node it is plugged into.cpubackend, which nodes without CXL can use, keeps KV only in that process's DRAM, so it is not shared with other engines or other nodes.This PR adds a storage backend,
remote, that lets engines on nodes without CXL share the CXL memory of one node with a CXL device (the pool node) over RDMA. On the vLLM side, it is used through the existingMaruKVConnectorwith additional settings, and the pool node runs with just the existing maru-server.🏗️ Design Changes
Structure. Remote access is added as a path within existing components. On worker nodes, MaruHandler gains one more storage backend; on the pool node, maru-server opens one more remote endpoint.
Before: only nodes with a CXL device attached share KV, and nodes without CXL keep it in their own process DRAM.
flowchart LR subgraph BN1["Node with CXL"] BE1["vLLM + MaruHandler (cxl)"] end subgraph BN2["Node without CXL"] BE2["vLLM + MaruHandler (cpu)"] BD2[("Own process DRAM")] BE2 --> BD2 end BCXL[("CXL shared memory")] BE1 --> BCXLAfter: nodes without CXL share the pool node's CXL memory over RDMA (dashed lines are control requests, the thick line is KV bytes, thin solid lines are connections within a node, and cylinders are memory).
flowchart LR subgraph AN2["Nodes without CXL (multiple)"] AE2["vLLM + MaruHandler (remote)"] AS2[("Staging buffer")] AE2 --- AS2 end subgraph AP["Pool node"] AMS["maru-server<br/>remote endpoint + remote access"] ACXL[("CXL shared memory")] AMS --> ACXL end AE2 -.->|"control requests"| AMS AS2 <==>|"RDMA"| ACXLremote. KV bytes move between the worker's staging buffer and the pool's CXL region by RDMA READ/WRITE (one-sided, NIXL), which bypasses the pool node's CPU; only small control requests such as page reservation, key publishing, lookup, and pin release go to the pool node. The worker opens no DAX device and runs no MaruServer of its own.remoteuses the same contract as thecpubackend: a read returns a read lease on a staging slot, and the caller returns the lease after copying. The vLLM connector uses the default transfer path on this backend as well (GPU copy kernel, async load and store).Operating rules.
Caller-visible changes.
cpuandmixed: CUDA graphs off (eager); tensor, pipeline, data, and context parallel sizes all 1; the recompute policy on load failure; and unquantized KV stored per chunk. LoRA, multimodal inputs, and MLA models are not supported.cpuandmixed, async load and store can be used, and are recommended. With synchronous load (the default), vLLM async scheduling must be disabled; with async load enabled, this requirement does not apply.cxlbackend is unchanged. However, in the fallback path that fills slabs layer by layer when the store kernel is unavailable, if a chunk's slab (the buffer holding the KV of all layers of one chunk) cannot be obtained or filled, that chunk is skipped in the remaining layers of that step.cxlalso uses this fallback path. Logging the warning at most once a minute when skipping because of a slot shortage or a storage failure applies only to the lease-based backends (cpu,mixed,remote); pool exhaustion incxlremains an error, as it is today.📝 Implementation Details
Pool node
maru_server/remote_access.py(new):RemoteAccesshandles allocating, mapping, and NIXL-registering remote regions; page reservation and publishing; pinned lookups and read tickets; quarantining of timed-out writes; and LRU eviction and the eviction log.serve_remote()is the remote endpoint (ZMQ REP) loop.maru_server/server.pydelete_kv_at, which deletes only when the location matches;evict_kv_at, which deletes the needed number of eviction candidates all at once or none at all;batch_register_or_lookup, which registers and looks up atomically; andbatch_pin_lookup, which pins and looks up atomically.--remote-*options and the remote thread. If the remote thread dies while serving, the server exits with exit code 1.mmap.madviseruns holding the GIL.maru_server/kv_manager.py,allocation_manager.py: added location-checked delete and evict (delete_at,evict_at), per-region pin (batch_pin_in), and owner lookup (get_owner).DaxMapper(cuda_register=False): the pool node does not register mapped regions with CUDA.Configuration and options
maru_storage_backend="remote",maru_remote_url,maru_remote_ucx_device,maru_remote_staging_size,maru_remote_load_reserve,maru_remote_timeout_s,maru_remote_retry_s. The remote backend requiresmaru_cache_namespace.--remote-bind,--remote-pool-size,--remote-page-bytes,--remote-ucx-device,--remote-pool-id,--remote-capacity,--remote-eviction,--remote-reservation-ttl,--remote-ticket-ttl,--remote-quarantine-ttl.maru_remote_timeout_s; if any check fails, startup is refused. With the defaults (reservation 60 s, ticket 120 s, transfer timeout 30 s), the reservation TTL sits exactly at this bound, so raisingmaru_remote_timeout_srequires raising--remote-reservation-ttlas well.MaruConfig.timeout_msand cannot be changed through the vLLM connector configuration.maru_common/sizes.py:parse_size, used by the--remote-*size options and by the measurement tool.pip install 'maru[remote]'.Worker node
maru_remote/(new)client.py)transport.py)protocol.py)maru_handler/storage/remote.py,remote_buffers.py(new):RemoteStorageClient.maru_remote_load_reserve).maru_remote_retry_s) ends. The monitor thread's pings and eviction-log fetches have a 0.5 s response deadline. While calls are already stopped, a single missed deadline restarts the retry interval from the beginning; while they are not, two consecutive missed deadlines stop calls. Even when the server answers, the previous NIXL peer is not replaced until the end of in-flight transfers is confirmed. When a load or store encounters a new run, it reconnects within that call and tries once more, without waiting for the monitor thread.close(), the mapping is kept so that it can be retried.abandon) when the server answers again: the page could not be published because a reconnect happened during the WRITE or another thread stopped calls; the publish request timed out or did not reach the server (28e8516); or the transfer ended with an error or the publish was rejected, and the server did not answer the return request sent right afterward (104b177). The pool skips return requests for tickets that are already published. When a response from a new server run arrives, the earlier run's pages no longer exist, so the tickets are dropped; if the server rejects a return with an error, retrying would give the same result, so a warning is logged and the ticket is dropped.releaseresponse equals the number of keys that were found and pinned. The lease is held until the GPU copy finishes.maru_vllm/connector.py: addedremoteconfiguration and validation, batched loads (retrieve_capacity), and the wiring of async load and store for the remote backend.--no-async-schedulingis required only for synchronous load._cpu_mode→_lease_mode,_cpu_bypass_request(s)→_unkeyed_request(s),_cpu_engine_config→_bind_lease_namespace,_cpu_load_range→_external_load_range. The "CPU M1" in error messages becomes the name returned by_lease_label()("Remote storage" forremote, "CPU M1" forcpuandmixed). Three messages change forcpuandmixed: the setup rejection ("CPU cache setup rejected"→"CPU M1 setup rejected"), the chunk-size check, and a load-failure reason. These are renames only; behavior changes forcpuandmixedare collected under "Changes to the existing lease path" below.maru_handler/handler.py: also selectsRemoteStorageClientas a storage backend, and renames the internal_cputo_storage. Added the public methodsretrieve_capacity()andstorage_gpu_accessible();healthcheck()andget_stats()handle the remote backend.Examples, docs, and tools
examples/vllm/remote/examples/vllm/remote/remote_storage_roundtrip.py: round-trips stores and lookups through the handler API without vLLM, checking byte equality and measuring RDMA bandwidth.tools/is reserved for marutop-compatible scripts, so 138da66 moved it next to the example.Points for review
maru-remote-server); 73c1edb moved it into maru-server. There are also two sets of old fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96 commits: b101d3c, 166771e, and fe257f3, brought onto this branch before a753bc1; and 9d33a2f, ce95214, and aac1426 (handler liveness checks and region reclaim, reverted by 6be1e64), brought in by the a753bc1 merge. In the current fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96, 5a5a66d, 4c7e6a6, and 5e89c8a carry the same changes. Please review against the final diff. When merging, the squash message will be rewritten to match the final diff._commit_hashwere removed from the namespace hash of the lease-based backends (cpu,mixed,remote), so that nodes holding the same weights at different paths can share. Telling apart different weights with the same configuration is therefore left tomaru_cache_namespace.cpuandmixedcopies do not survive restarts, so no existing cache breaks. Loads on the same lease path now look up, copy, and return per request.remotefurther splits them into batches that fit in staging (retrieve_capacity).remotebackend.8c8b935, handlers using the remote backend (one on the scheduler side per engine, and one per worker) sent a ping once a second, and the pool node's remote endpoint, which handles requests one at a time, processed as many extra pings per second as there were handlers. Now, while calls succeed, the monitor thread sends three kinds of requests: an eviction-log fetch when a response signals new evictions (including one ping the first time it meets that run), a check ping when no response has arrived after a store was skipped for a remembered key, and a request returning a page whose quarantined WRITE has finished. The scheduler-side handler's monitor thread sends no requests during this time. In exchange, the first call after a disconnect incurs the 2 s stall above once: the engine thread waits at the first existence check on the scheduler side, and at the first load or store on the worker side when synchronous load and store (the default) are used. Before8c8b935, only calls that arrived before the monitor thread noticed the disconnect (two pings with a 0.5 s timeout) incurred it. Idle handlers also learn of server restarts and evictions at their next call.--max-model-len 40960gives 10 GiB of CUDA-pinned host memory per worker, and the model's default context (131,072 tokens) gives 32 GiB. It can be changed withmaru_remote_staging_size.✅ Tests
test_remote_access.py,test_remote_storage.py,test_remote_protocol.py,test_remote_transport.py,test_vllm_remote_config.pytest_vllm_cpu_load_range.pyand others909947f. Jenkins #12 also completed successfully after Draft removal; it did not schedule aCOMMON_TESTSdownstream job.909947f, 2026-10-14): the first CI run after the merge exposed same-port restart failures (EADDRINUSE) on Python 3.12 and 3.14. The endpoint now owns its ZMQ context and waits for its socket cleanup before returning, including when bind fails; other endpoints and clients keep their contexts. An immediate-rebind probe failed 50/50 before the fix and 0/50 afterward. The focused port-release, bind-failure, and pool-restart cases passed 150 repetitions with Python 3.14 and PyZMQ 27.2.0.909947f, the full non-integration suite gives 1,344 passed, 4 skipped, 160 deselected on Python 3.12 with native CUDA/CXL-flush extensions and 913 passed, 23 skipped, 160 deselected on Python 3.14 with the CI dependency set. Ruff 0.16.5 lint and format checks pass. CXL/RDMA serving and recovery were not rerun on this commit.5c6face, 2026-10-14): merged the current fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96 head (c86e7a5), which includes main atbdeb384. Resolved the packed-load conflict by keeping remote load failure reporting and partial-chunk handling together with main's per-slab pinned/pageable copy fallback.CUDA_VISIBLE_DEVICES=0 PYTHONPATH="$PWD" ~/.venv/bin/python -m pytest -m 'not integration' --ignore=tests/sglang -qgives 1,343 passed, 4 skipped, 160 deselected. All 12 packed-kernel GPU round-trip cases pass, including pinned, pageable, and mixed host buffers on an NVIDIA RTX PRO 6000 Blackwell Server Edition. These GPU tests use host-memory stand-ins; the CXL/RDMA serving and recovery checks below remain evidence for their explicitly listed earlier commits, and were not rerun on this merge commit.mypy .stops at the existing duplicaterun_benchmarkmodule names; the same diagnostic is reproduced on104b177before the merge.104b177): with Python 3.12, no GPU visible, andMARU_PLUGINS=none,pytest -q tests/unitgives 1,316 passed, 20 skipped (in an environment with themaru_kv_opsextension built);ruff checkandruff format --checkpass.maru_handler/handler.pygained errors relative to base (0dad3ca), 6 of them. They arise because the type of_storageis inferred asCpuStorageClient | Nonefrom its first assignment; fixing them would require widening the return types of the handler's public methods, so they are left in this PR.28e8516, 2 more in104b177): checks that a page the pool quarantined on reservation expiry is returned after reconnecting, when calls stopped after the WRITE finished, when the publish request timed out, and when the transfer ended with an error and the return request sent right afterward got no answer. The first two tests fail one71c172, and the third ondd75ab8. Also checks that when the publish was executed and only its response was lost, the return request does not delete that key, and that when the server rejects a return, that ticket is not sent again (the latter fails ondd75ab8).8c8b935, 5 remote backend tests added: 4 new tests, one of which runs under two conditions): confirmed that a monitor thread with its period reduced to 0.02 s sends no ping for 0.3 s (about 15 rounds) after a successful call, and that after the server was taken down and brought back up, it reconnected through a ping and then stopped pinging again. The absence of pings while healthy was not measured on the real device; reconnecting after a forced pool kill was confirmed in the355df0fre-check below.138da66): changed a test to check directly that the prefault function is not called when remote regions are mapped. If remote access prefaults, this test fails. The moved round-trip measurement tool was run against a test pool server and passed.9d498a5gave 1,311 passed, 30 skipped, with no increase over the original in strict docs build warnings (16) or in mypy errors in the changed modules (10). The 25 failures in the 10-11 re-check happened because NVML was not initialized on the measurement host that day, so vLLM could not infer the device type; in an environment with healthy NVML, the same tests pass.6be1e64gave 1,302 passed, 20 skipped, after removing the 19 reverted liveness-check and region-reclaim tests.355df0f, only the example round-trip tool, reconnecting after a forced pool kill during a real WRITE, and Qwen2.5-0.5B serving were rerun on the real device (the "355df0fre-check" item below). The commits after that are the merge of fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96's 0dad3ca (e71c172, which changes when entries are removed from the scheduler-side budget waiting list), reclaiming pages that could not be returned (28e8516,104b177), and the eviction-criterion wording (dd75ab8). 0dad3ca was re-checked on the real device in fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96, and28e8516and104b177were verified by unit tests only. The delayed-WRITE corruption cases, the six failure paths, Llama-3.1-8B serving, and the performance measurements are results from code earlier than355df0f. The 10-07 verification tree (9d498a5) contained the later-reverted region-reclaim commit (aac1426) and the 1 s periodic ping, and the performance and behavior measurements were taken on the even earlier server-integrated implementation.remotebackend and an engine on another node read. For eviction and the multi-turn load, a single engine on another node stored and read. Configurations in which two different worker nodes (neither of them the pool node) share KV, and in which writes and reads go through different NICs on the pool node, were not checked.355df0fre-check (2026-10-13): same devices and connection as the 10-07 verification below.examples/vllm/remote/remote_storage_roundtrip.pyfrom the GPU node against the pool node. In 3 rounds of 8 objects each of 4 MiB and 32 MiB, SHA-256 matched 8/8 in every round; the 4 objects stored again with different bytes all got an "already exists" answer; and afterward the pool node had 0 reservations, quarantined pages, and lookup pins. Against a pool with a 3 s reservation TTL (shorter than twice the tool's 30 s transfer time limit), the tool refused to start at the connect stage, as the connect checks above specify. The results above used a 256 MiB pool with 32 MiB pages (from the second round on, the previous round's keys are evicted), a 60 s read-ticket TTL (default 120 s), a 2 s quarantine warning time (--remote-quarantine-ttl, the time before a warning about an unconfirmed WRITE page; default 600 s), and a 60 s reservation TTL (the default).maru_log_timingon; the pool had 2 GiB capacity with 4 MiB pages, a 60 s read-ticket TTL (default 120 s), a 2 s quarantine warning time (default 600 s), and a 60 s reservation TTL (the default). For a request while the pool was down, the worker copied the 8 chunks to be stored into staging and then skipped the store (a copy that did not happen on 10-07). The likely cause: after an API exception was injected once just before a READ in the preceding scenario, there was a window in which the retry interval had passed but the worker had not yet reconnected, and the check that hands out staging slots does not look at the pending reconnect. Per the code, the same window also opens with the default settings after the pool comes back, until quarantined transfers are cleaned up. No data is stored incorrectly; while the window is open, each store wastes one GPU→staging copy.9d498a5(2026-10-07): the pool node's CXL-DRAM device and two RTX PRO 6000 GPUs on the GPU node were connected over 400GbE RoCE. NIXL 1.2.0, UCX 1.21.0, NIC port pinning,rc_mlx5, and real DAX mapping were confirmed.be40358, 6 cases in which a delayed WRITE on an expired reservation corrupted a new key were reproduced, and the hardening protected the same 6 cases. The two reservation roles were performed in sequence in one worker's test program, and this was checked 3 times each with and without a quarantine request.--enforce-eager, async load and store, async scheduling on), with 256-token chunks and 32 MiB remote pages. TTFT is the time from sending a request to receiving the first token.--max-model-len 32768, and vLLM's prefix cache on. Completing all 2,462 requests took 55.85 / 55.70 s over two runs, with 0 failures. Of the prompt tokens of turn-9 requests, 79% came from cache (the GPU prefix cache and the pool). Compared with the same procedure against the earlier stage with remote access in a separate process, completion time and cache ratio were both the same within 3%.examples/vllm/remoteexample (Qwen2.5-0.5B)🔗 Related Issues (optional)
🌿 Related PRs (optional)
fix/lease-path-store-copy-sync-scheduling), and once fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96 is merged into main, the base will be switched to main.maru_vllm/connector.py, which both PRs modify, was reconciled in the merge commit (a753bc1), and the re-stacked fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96 was merged in ef93c9a (two renamed spots inconnector.pyconflicted; fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96's change was already included, so this branch's side was kept, and no file changed). b6f20fb, added to fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96, was merged in 63e1894, and 0dad3ca in e71c172, both without conflicts. Latest fix: kernel-less chunk-slab store copy, test plugin isolation, async-load block budget #96 headc86e7a5, including mainbdeb384, is now merged in5c6face; the packed-load conflict preserves both remote failure handling and the unpinned-slab fallback from fix(vllm): fall back to copies for unpinned CXL slabs #102.🌿 Related Branches (optional)
📦 Release Note (for auto-generation / write in English)
NEW
CHANGED
FIXED
IMPORTANT NOTES