Repository navigation
mgr/cephadm: give nvmeof gateways a key they can use (squid aes256k) - #28
Open
dvanders wants to merge 1 commit into
Conversation
Since 19.2.6 cephadm creates daemon keys as aes256k (ROTATION_CIPHER). The nvmeof gateway images for squid (quay.io/ceph/nvmeof 1.2.x, the default 1.2.1, and the squid line 1.4.x up to 1.4.28) carry their own librados, built before aes256k (19.0.0 dev, 19.2.3), and cannot connect with such a key: the gateway crash-loops with "RADOS I/O error (error connecting to the cluster)" and "Unable to create OMAP, exiting!", and the service never comes up (orch:cephadm:smoke-roleless 2-services/nvmeof times out on "nvmeof.foo has 0/1"). Create and rotate nvmeof keys with the previous key type (aes): a per-daemon-type key type in utils (key_type_for), used by the nvmeof service's keyring and by key_rotate (ceph orch daemon rotate-key). A gateway already deployed with an aes256k key gets its key rotated to aes on its next deploy, so `ceph orch redeploy nvmeof.<svc>` heals a failed service. Reproduced on a cephadm cluster: with an aes256k key, python3-rados in quay.io/ceph/nvmeof:1.2.1, 1.2.5 and 1.4.28 fails to connect ([errno 5]); in 1.7 (librados 20.2.4) and 2.0.0 it connects. Fixes: https://tracker.ceph.com/issues/81778 Assisted-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>
dvanders
force-pushed
the
wip-dvanders-cobaltcore-nvmeof-legacy-key
branch
from
October 14, 2026 16:18
772974d to
46a0393
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since 19.2.6, cephadm creates daemon keys as
aes256k(ROTATION_CIPHER). The squid nvmeof gateway images carry their own librados, built beforeaes256k, and cannot connect with such a key. The gateway crash-loops and the nvmeof service never comes up.Found in our teuthology run of this branch,
orch:cephadm:smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final}(job 3763, 2026-10-10):auth get-or-create client.nvmeof.foo.vm06.qoywko ... "key_type": "aes256k"quay.io/ceph/nvmeof:1.2.5), on each of 5 starts:rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster), thenUnable to create OMAP, exiting!; no mon sees a connection attempttasks.cephadm: nvmeof.foo has 0/1, thenMaxWhileTries: reached maximum tries (300) after waiting for 300 secondsReproduced outside teuthology: a fresh cephx user with an
aes256kkey, thenrados.Rados(...).connect()with python3-rados inside each gateway image:[errno 5] RADOS I/O errorDEFAULT_NVMEOF_IMAGEin cephadm/module.py, still in 19.2.7)[errno 5] RADOS I/O error[errno 5] RADOS I/O errorNo squid gateway image can use the key, so this changes cephadm instead of the default image:
utils.key_type_for(): nvmeof gets the previous key type,aes; every other daemon stays onaes256k.aes256kkey gets its key rotated toaeson its next deploy, soceph orch redeploy nvmeof.<svc>heals a failed service.key_rotate(ceph orch daemon rotate-key) uses it too. The post-upgrade key rotation only covers mon/mgr/osd/mds, so it is unchanged.Tests: new
cephadm/tests/test_nvmeof_key_type.py(5 tests); the full cephadm suite passes (1236). flake8 is clean, and mypy reports no new errors. Not yet run live: the smoke-roleless nvmeof job on a build with this commit is the test.Upstream: https://tracker.ceph.com/issues/81778 (this bug, squid and tentacle; related: #79868, ceph/ceph-nvmeof#2203).
🤖 Generated with Claude Code