This is a utility for migrating files between CephFS data pools without changing directory paths or disrupting access. A given CephFS volume comprises at least one data pool, but additional pools may be added. Once attached to the MDS, new files can be directed to one pool or the other via file layouts.
A simple mv does not move data to a different CephFS data pool, as it is a rename operation that touches only metadata. The script works by writing a copy of each file to a staging directory, with the new layout, then atomically moving the new copy on top of the original.
Use-cases include:
- Tiering by assorting files to faster, more efficient, cost-effective storage media
- Reclaiming raw capacity by converting from replication to erasure coding
- Reclaiming raw capacity by converting EC pool files to a wider EC profile
Two scripts are provided:
-
vcephfs-transcode-setup.shsets up a CephFS volume for migration of existing files, creating and attaching a new RADOS pool, along with an EC profile and CRUSH rule as needed. Care is taken to re-use existing EC profiles and CRUSH rules for clarity, and to avoid hitting the hard limit of 256 CRUSH rules. The setup script has an option to execute migration once set up, but I recommend leaving this disabled so that a sanity check may be performed first, and because often enough the two scripts will need to be run on different systems.The new pool is set to be the _dir layout` for the specified subdirectory, which may or may not be the root of the CephFS mount. Migration can be specified for subtrees, potentially to different volumes.
-
vcephfs_transcoder.pyperforms the actual migration.
Note that both require a CephX client user with the p capability flag.
Some feel that what this utility accomplishes does not qualify as transcoding. Tough. Marcan42 called it that, and so it remains.
Usage:
Usage:
vcephfs-transcode-setup.sh [OPTIONS]
Options:
--mount PATH Mount point for the volume
(e.g. /shared/cephlab/howie)
Volume name is derived from the last path
component (e.g. "howie").
--ec-k INT EC data chunks
--ec-m INT EC parity chunks
--ec-plugin NAME EC plugin (default: isa)
--crush-device-class CLASS
Device class: ssd or hdd
--replicated Transcode to a replicated pool instead of EC.
Pool name: cephfs.<vol>.rdata
Incompatible with --ec-k and --ec-m.
EC profile, CRUSH rule, and pool name are derived automatically:
profile: ec<k>.<m><class> e.g. ec8.2hdd
rule: ec<k>.<m>-rule-<class> e.g. ec8.2-rule-hdd
pool: cephfs.<vol>.ec<k>.<m>.<class>.data
--pg-num INT PG count for new pool (default: 128)
--tmpdir PATH Temp dir for transcoder (default: mount/tmp)
--fastec Enable allow_ec_optimizations (FastEC)
--compression MODE Compression mode for new pool (default: aggressive)
(e.g. "aggressive", "force", "passive", "none")
--compression-algorithm ALG
Compression algorithm (default: snappy)
(e.g. "snappy", "zstd", "lz4", "zlib")
--min-age DAYS Min file age in days (default: 1)
--min-size SIZE Min file size (e.g. "1M", "300K")
--subdir PATH Subdirectory under mount to transcode
(e.g. "ethel/merman")
--log-file PATH Log file for transcoder
--skip-pool-setup Skip pool creation (already done)
--skip-layout Skip layout / setfattr steps
--transcode Run the transcoder (off by default)
--execute Actually run commands (default: dry-run)
--help Show this help
usage: vcephfs_transcoder.py [-h] [--version] [--tmpdir TMPDIR]
[--process-hardlinks] [--debug]
[--min-age MIN_AGE] [--min-size SIZE]
[--max-size SIZE] [--threads THREADS] [--dry-run]
[--config PATH] [--print-config-example [VOLUME]]
[--config-poll-seconds SEC] [--force-tmpdir-pool]
[--log-file LOG_FILE]
[--log-rotate-lines LOG_ROTATE_LINES]
[--log-rotate-time LOG_ROTATE_TIME]
[--log-rotate-size GIB] [--no-copy-file-range]
[--copy-file-range]
[--regulate-prometheus-url URL]
[--regulate-query PROMQL]
[--regulate-pause-ms MS] [--regulate-slo-ms MS]
[--regulate-period-s SEC]
[--regulate-floor-ms MS]
[--regulate-quiet-ticks N] [--source-pool POOL]
[--prune-small-subtrees]
[--prune-subtree-max-bytes BYTES]
[--prune-budget-bytes BYTES]
[--max-files MAX_FILES] [--file-delay MS]
[--prune-dir-regex REGEX]
[dirs ...]
Transcode cephfs files to their directory layout
positional arguments:
dirs Directories to scan
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
--tmpdir TMPDIR Temporary directory to which to copy files. Important:
This directory should have its layout set to the
*default* data pool for the FS, to avoid excess
backtrace objects.
--process-hardlinks Process files with nlink > 1, which is potentially
dangerous
--debug, -d
--min-age MIN_AGE Minimum age of file before transcoding, in days
(adjustable at runtime via SIGRTMIN+2/SIGRTMIN+3)
--min-size SIZE Skip files smaller than this size. Suffix B/K/M/G
(binary); plain number means bytes. 0 disables.
--max-size SIZE Skip files larger than this size (same format as
--min-size). Omit for no upper limit.
--threads THREADS Number of threads for data copying
--dry-run, -n Perform transcode but do not replace files
--config PATH key=value file of live tunables (file_delay_ms,
threads, min_age_days, min_size, prune_dir_regex), re-
read when its mtime changes. Keep it on local disk,
NOT in the CephFS volume: a stat there is an MDS round
trip.
--print-config-example [VOLUME]
print a fully-populated --config file for VOLUME and
exit
--config-poll-seconds SEC
how often to stat --config for changes (default: 10)
--force-tmpdir-pool proceed even if the tmpdir is in a target pool.
Disables the only check that makes a failed layout
application visible; do not use routinely.
--log-file LOG_FILE Also log to this file
--log-rotate-lines LOG_ROTATE_LINES
Rotate the log file after this many lines (requires
--log-file)
--log-rotate-time LOG_ROTATE_TIME
Rotate the log file after this duration, e.g. 30m, 2h,
1d (requires --log-file)
--log-rotate-size GIB
Rotate the log file when it reaches this size in GiB
(requires --log-file)
--no-copy-file-range Disable use of copy_file_range and always use
userspace copy. DEFAULT since 2026-09-05: a 15-thread
A/B on a production volume measured copy_file_range no
faster overall (154.0 vs 170.1 MiB/s) and 1.3-2.1x
SLOWER for files under 1 MiB, which is the size range
these jobs now work in.
--copy-file-range Opt back in to copy_file_range (server-side copy when
CephFS supports it). Off by default; see --no-copy-
file-range.
--regulate-prometheus-url URL
Prometheus /api/v1/query endpoint for self-regulation.
Unset (the default) disables regulation entirely and
the job simply runs at --file-delay and --threads.
--regulate-query PROMQL
A complete PromQL expression returning ONE value in
MILLISECONDS for the latency to protect. {volume} is
substituted with the CephFS name from the mount,
regex-escaped; write the query without it if that
cannot be derived. Single line, no '#'.
--regulate-pause-ms MS
Pause the job while the query exceeds this (default
150).
--regulate-slo-ms MS Soft target, used only to express readings as a
percentage in log messages (default 75). --regulate-
pause-ms is what actually gates.
--regulate-period-s SEC
Seconds between samples (default 30).
--regulate-floor-ms MS
Lower bound on --file-delay that regulation may ease
down to. Repeated pauses raise it; sustained quiet
releases it back to this baseline (default 0).
--regulate-quiet-ticks N
Consecutive clean samples before easing the delay
(default 10).
--source-pool POOL Only transcode files whose CURRENT data pool is POOL.
Without it, every file not already on the target pool
is eligible. Use this to drain one pool into another
without also sweeping the default pool.
--prune-small-subtrees
Skip whole subtrees whose recursive size makes them
not worth walking, using the MDS's own
ceph.dir.rbytes/rfiles (one getfattr per directory, no
walk). OFF by default: it changes what the pass
covers, which should always be deliberate. INERT
without --min-size, since the mean-size test compares
against min-size/8; a warning is logged at startup if
you pass this with --min-size 0.
--prune-subtree-max-bytes BYTES
With --prune-small-subtrees, only prune a subtree
whose TOTAL recursive bytes are below this (default 1
GiB). This is a bound on what pruning can cost you:
whatever the size distribution inside, skipping the
subtree forgoes at most this many bytes.
--prune-budget-bytes BYTES
With --prune-small-subtrees, stop pruning once this
many bytes have been skipped in total (default 100
GiB). Deliberately finite: the per-subtree bound above
says nothing about the aggregate, so without this a
pass could skip unbounded data a gigabyte at a time.
--max-files MAX_FILES
Stop after submitting this many files for transcoding
--file-delay MS Delay in milliseconds before statting each new file
(adjustable at runtime via SIGRTMIN/SIGRTMIN+1)
--prune-dir-regex REGEX
Regular expression (unanchored, via re.search) matched
against directory NAMES; any name containing a match
is pruned from the walk entirely (not descended into
or statted). Anchor with ^ / $ for exact names, e.g.
'\.runfiles$' to skip Bazel runfiles symlink farms.
Overrides the built-in default set (see
DEFAULT_PRUNE_DIRS); pass an empty string to disable
pruning entirely.
runtime signals:
SIGUSR1 (10) increase thread count by 1 (resumes from pause)
SIGUSR2 (12) decrease thread count by 1 (0 = pause)
SIGTSTP (20) throttle to 1 thread (Ctrl+Z)
SIGRTMIN (34) increase file delay x1.25 (min +1ms, cap 600000ms)
SIGRTMIN+1(35) decrease file delay /1.25 (min -1ms, floor 0ms)
SIGRTMIN+2(36) increase min-age by 3 days
SIGRTMIN+3(37) decrease min-age by 3 days (min 1)
SIGRTMIN+4(38) dump current state/tunables to the log
example --config file (every key at its default):
# vcephfs_transcoder runtime config -- VOLUME
# Conventional path: /home/USER/tc_VOLUME.conf
#
# Re-read when this file's mtime changes (--config-poll-seconds,
# default 10s). Only keys whose value CHANGED in the file are applied,
# so editing one line never re-asserts the others -- a delay set by
# signal survives an unrelated edit here.
#
# Keep this on LOCAL disk, never inside the CephFS volume: a stat there
# is an MDS round trip, which is the load being bounded.
#
# Every key is optional. Commenting one out means this file does not
# assert it at all, leaving it to the command line or to signals.
# --- pace -------------------------------------------------------------
# Sleep per file ENCOUNTERED in the walk, before the stat. Scan rate is
# 1000/file_delay_ms files/s REGARDLESS of threads -- the walker is
# single-threaded and is what gates a pass. 0 = unthrottled.
file_delay_ms = 0
# Concurrent copies. Bounds the copier, not the walker. 0 pauses the job
# (in-flight copies finish); set > 0 to resume.
threads = 1
# --- how the delay signals move ---------------------------------------
# SIGRTMIN multiplies the delay, SIGRTMIN+1 divides it. Separate rates
# let backing off be coarser than recovering: 2.0 up with 1.1 down
# retreats in doublings and probes back in 10% increments.
delay_step_up = 1.25
delay_step_down = 1.25
# Floor for the down direction. 0 permits a step to fully unthrottled;
# raise it to guarantee the signal path can never do that on a live
# filesystem.
delay_min_ms = 0
# --- what counts as eligible ------------------------------------------
# Skip files modified within this many days.
min_age_days = 1
# Skip files smaller than this many bytes. Pre-Tentacle, EC pads small
# objects to a whole stripe, which is why this is not lower.
min_size = 512000
# --- what to skip entirely --------------------------------------------
# Matched against directory NAMES during the walk; matches are never
# descended into or statted. Cost is one regex match per directory, not
# per file. Set empty to disable pruning entirely.
prune_dir_regex = ^(\.runfiles|site\-packages|node_modules|__pycache__|\.venv|venv|penv|renv|packrat|\.git|\.tox|\.mypy_cache|\.pytest_cache|\.cargo|\.gradle)$
redirect it to disk with: vcephfs_transcoder.py --print-config-example > /home/USER/tc_VOLUME.conf
Example invocations:
/usr/local/sbin/vcephfs-transcode-setup.sh --mount /shared/ceph/ethel --ec-k 6 --ec-m 3 --crush-device-class ssd --subdir /merman
/usr/local/sbin/vcephfs_transcoder.py --min-age 1 --min-size 500k /shared/ceph/ethel/merman --tmpdir /shared/ceph/ethel/merman/tmp --log-file ~/ethel/transcode.log --dry-run
This was presented at Ceph Day Seattle 2026, and based on a script posted
to Reddit by marcan42.
The default here is off, on measurements taken with clients well connected to the cluster. That is not everyone's situation, and the reasoning behind the default does not transfer to all of them.
copy_file_range on CephFS can become an OSD-to-OSD copy, in which case the data
never traverses the client's link at all. Where a client reaches the cluster over a
slower or higher-latency path than the cluster's own interconnect, that matters far
more than any per-call overhead: a userspace copy moves the file across that link
twice, once in and once out.
In this deployment that offload does not appear to happen. Measured on a 1 GiB cross-pool copy, counting the client's own interfaces:
method secs MiB/s client rx client tx wire/filesize
cfr 1.13 905.9 1034Mi 1033Mi 2.02x
userspace 1.17 874.9 1034Mi 1033Mi 2.02x
Both methods put the whole file across the client link in each direction. Whatever
copy_file_range is doing here, it is not keeping the data off the wire, which is
why it shows up in our numbers as pure overhead.
Two honest limits on that result. It is one configuration — the copies are cross-pool (replicated to erasure-coded), which is this tool's entire purpose and a plausible reason the fast path declines, but we did not isolate the cause. And counting bytes at the client says nothing about what the fabric beyond that hop carried, or what a differently-connected client would experience.
So: if your clients are remote, or you have reason to believe the OSD-side copy
engages in your setup, measure it before accepting the default. Compare the client's
interface counters across an identical copy done each way — if copy_file_range
moves markedly less data across the client link than a userspace copy does, the
offload is working for you and --copy-file-range is likely worth enabling
regardless of what the per-call overhead costs.
A second pass at a lower --min-size usually already knows its candidates. The
previous run logged every file it skipped and why, so the list of files that were
"below --min-size" is the candidate set for the next threshold.
Rediscovering them by walking is the expensive part. file_delay sleeps once per
file encountered, in the single-threaded producer loop, before the stat — so
a pass costs roughly rfiles × file_delay no matter how few files qualify. On one
production volume that meant visiting 195M paths to reach 8.7M candidates: about
45 days of walking to do 2 days of work.
# extract the candidates from the previous run's log
grep -a 'below --min-size' prev.log \
| sed -E 's/.*Skipping (.*): size [0-9]+ below --min-size [0-9]+$/\1/' \
> candidates.txt
vcephfs_transcoder.py --min-size 131072 /vol/data \
--tmpdir /vol/data/tmp-transcode --paths-from candidates.txt
The directories still have to be given. They are not scanned; they bound what the list is allowed to touch, and a listed path outside them is refused and counted. A list is easy to generate against the wrong volume and the damage would be quiet and large.
The file is NUL- or newline-delimited, detected from its content, so both
find -print0 and a line-oriented scrape work without a flag. It is read in
chunks rather than slurped — these lists run to tens of millions of lines.
Nothing here trusts the list. Every path is re-stated and its layout re-read,
exactly as the walker would, so a stale list is safe rather than dangerous. What
it costs you is coverage, and that can go fast. Measured on two volumes six days
after the logs were written:
| still exists | of those, still needing the move | |
|---|---|---|
| volume A | 36.7% | 60.6% |
| volume B | 100.0% | 98.2% |
On volume A only 22% of the list was still worth acting on. So regenerate the list
close to when you use it, and read vanished in the completion line as the measure
of how stale it was:
Complete: 0 transcoded, 0 failed, 20 already matched, 0 wrong source pool,
20 vanished, ...
A path that has disappeared is logged at INFO and counted. It is an expected outcome of a list that has aged, not an error, and it never aborts the run.
- It does not discover. A list cannot contain files created after the log was written, or files the earlier run never reached. Use it to re-process known candidates cheaply; keep a full walk for finding new work.
- It declines multiply-linked files. Every link has to be in hand before any
of them can be replaced, and a list cannot promise it holds them all.
Transcoding a partial set would break the link relationship, so this mode skips
them regardless of
--process-hardlinks. Walk for those. --prune-dir-regexand--prune-small-subtreeshave no effect, and say so at startup. They prune the walk; here the list has replaced the walk.
This mode checks the file's layout before its size; the walker does the
reverse. The walker is right for its case: most of what it encounters is tiny, a
stat alone rejects it, and reading a layout for every file would add an MDS
round trip to the common path.
A list is already filtered to plausible candidates, and its stalest entries are exactly the ones an earlier pass has since moved. Testing size first would log those as "below --min-size" and hide the fact that they are already done — which is not hypothetical: it is how one analysis came to overstate its remaining work, by counting already-converted files as outstanding candidates.
Long passes need retuning while they run. A pass over a large volume takes weeks, the filesystem underneath it does not hold still for that long, and restarting to change a setting throws away the walk position — there is no checkpoint, so a restart re-walks from the top.
So the tunables live in a key = value file, re-read whenever its mtime changes:
vcephfs_transcoder.py --print-config-example > ~/tc_myvolume.conf
vcephfs_transcoder.py --config ~/tc_myvolume.conf /mnt/cephfs/myvolume ...
Edit the file and the running job picks it up within --config-poll-seconds
(default 10). No restart, no signals, no lost position. --print-config-example
emits every key at its default with the reasoning for each inline, so the file is
its own documentation; the same example is reproduced in --help.
Currently settable at runtime: file_delay_ms, threads, min_age_days,
min_size, prune_dir_regex, prune_subtree_max_bytes, prune_budget_bytes,
and the delay-stepping controls delay_step_up / delay_step_down /
delay_min_ms.
An edit to the file re-applies all the keys it contains, not just the one you changed. So any key left present in the file is silently reasserted whenever you touch anything else.
This is not hypothetical. Lowering min_size on a running job reset its thread
count from 15 back to 1, because threads was still set in the file from an
earlier session, and the job crawled until somebody noticed.
If an external controller owns a tunable, comment that key out:
# threads and file_delay_ms are owned by the controller. Leaving them set here
# means any unrelated edit to this file resets them.
# threads = 1
# file_delay_ms = 100
min_age_days = 33
min_size = 131072
A key that is absent is not applied. A key that is present is applied on every reload, whether or not you meant to change it.
- One file per volume. The jobs are independent and their sensible settings differ by orders of magnitude — a volume of 4 MiB files and a volume of 30 KiB files want nothing in common.
- Comments are stripped at
#, and keys split on the first=, so values may contain=but not#. - Unknown keys are rejected with an error naming the line, rather than ignored — a typo in a tuning edit should not read as success.
Ceph has an in-flight alternative: transparent pool migration at the RADOS layer,
below the filesystem — see ceph/ceph
#65746 "Pool Migration" and
#69112 "Pool Migration 2" by jamiepryde.
It uses the redirect/manifest shape: the source pool keeps a small redirect object per
migrated object and clients are forwarded transparently.
The two approaches trade off in opposite directions, and which one you want depends almost entirely on whether you need the source pool to go away.
| This transcoder | RADOS-layer redirect | |
|---|---|---|
| Layer | File, MDS-mediated | Object, below the filesystem |
| Residue in the source pool | A zero-byte backtrace object, and only in the filesystem's first data pool | A redirect object per migrated object, in every source pool |
| Can the source pool be deleted afterwards? | Yes, for any pool that is not the filesystem's first data pool | No — the redirects are load-bearing |
| Client transparency | Each replacement is an atomic rename, but it is per-file work | Fully transparent; no walk at all |
| Cost model | One full tree walk plus per-file MDS operations | No walk, but a permanent indirection on every read |
| Reversible mid-flight? | Yes — stop the job; whatever moved has moved | Bounded by the redirect layer's own lifecycle |
The short version: this tool pays an enormous one-time walk and in exchange leaves a source pool that is genuinely empty and can be dropped. The redirect model skips the walk entirely, and in exchange you carry the source pool and an extra hop forever.
Verified on a production cluster rather than inferred:
- A transcode does not move a file. It creates a new inode, copies into it, then renames over the original — which unlinks the old inode completely. After a journal flush the old inode's objects are absent from every pool, so a non-first source pool really does reach zero and can be deleted.
- Every inode keeps a backtrace in the filesystem's first data pool
(
data_pools[0]), regardless of where its data lives. That pool can never be emptied, and it is not the same thing as a directory's default pool (ceph.dir.layout.pool), which is freely settable and is how a transcode target is chosen. Pointing the volume root at an EC pool changes where new files land; it does not change the first data pool. - A file whose data is outside the first pool therefore costs two objects: its data
object, plus a zero-byte object in the first pool holding only the backtrace —
measured at size 0, a 305-355 byte
parentxattr, zero omap entries.
That last point is the price of using a non-first pool, not a price of transcoding: a file created directly on the EC pool carries exactly the same two objects.
Across 980 OSDs holding 17.4 PiB, with the BlueStore DB colocated on the block device on every OSD:
raw size 23512.7 TiB
used total 17403.0 TiB
of which data 17332.3 TiB
of which META 63.4 TiB <- RocksDB/BlueFS
of which omap 7.4 TiB
META as % of used: 0.36%
Metadata is 0.36% of used space, so capacity is not the concern. Object count is: it drives scrub duration, recovery granularity, peering cost and onode cache pressure. Per-GB is the wrong denominator; per-OSD and per-PG are the right ones.
This is the mechanism behind "bytes leave, objects stay" — a fully transcoded volume keeps one zero-byte object per file in its first pool forever, so its object count does not fall even as its stored bytes approach zero. Size pools for the object count they will retain, not the bytes they will shed.
A corollary if you are widening an EC profile rather than migrating from replication: 4+2 to 8+2 raises shards per object from 6 to 10, so object count rises with it. On a cluster whose DB lives on a separate device rather than colocated, that is the case to size for before starting.