Repository navigation
Backport: RGW Retry-After header and LIST/DELETE rate limits - #25
Merged
Merged
Conversation
s0nea
changed the title
Add rate limiting specific to LIST ops, similar to the current rate-limiting (https://docs.ceph.com/en/latest/radosgw/admin/#rate-limit-management) Example usage: ``` ./bin/radosgw-admin ratelimit set --ratelimit-scope=user --uid=<UID> --max_list_ops=2 ./bin/radosgw-admin ratelimit set --ratelimit-scope=user --uid=<UID> --max_delete_ops=2 ./bin/radosgw-admin ratelimit enable --ratelimit-scope=user --uid=<UID> ./bin/radosgw-admin ratelimit get --ratelimit-scope=user --uid=<UID> { "user_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_list_ops": 2, "max_delete_ops": 2, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": true } } pkill -9 radosgw ./bin/radosgw -c ./ceph.conf ... aws --endpoint-url 'http://0:8000' s3 mb s3://bkt aws --endpoint-url 'http://0:8000' s3 cp ./ceph.conf s3://bkt aws --endpoint-url http://0:8000 s3api list-objects-v2 --bucket bkt --prefix 'ceph.conf' --delimiter '/' { "Contents": [ { "Key": "ceph.conf", "LastModified": "2025-07-30T13:59:38+00:00", "ETag": "\"13d11d431ae290134562c019d9e40c0e\"", "Size": 32346, "StorageClass": "STANDARD" } ], "RequestCharged": null } aws --endpoint-url http://0:8000 s3api list-objects-v2 --bucket bkt --prefix 'ceph.conf' --delimiter '/' { "Contents": [ { "Key": "ceph.conf", "LastModified": "2025-07-30T13:59:38+00:00", "ETag": "\"13d11d431ae290134562c019d9e40c0e\"", "Size": 32346, "StorageClass": "STANDARD" } ], "RequestCharged": null } aws --endpoint-url http://0:8000 s3api list-objects-v2 --bucket bkt --prefix 'ceph.conf' --delimiter '/' argument of type 'NoneType' is not iterable tail -F ./out/radosgw.8000.log | grep beast ... beast: 0x7fffbbe09780: [30/Jul/2025:15:44:50.359 +0000] " GET /bkt?list-type=2&delimiter=%2F&prefix=ceph.conf&encoding-type=url HTTP/1.1" 200 535 - "aws-cli/2.15.31 Python/3.9.21 Linux/5.14.0-570.28.1.el9_6.x86_64 source/x86_64.rhel.9 prompt/off command/s3api.list-objects-v2" - latency=0.000999995s beast: 0x7fffbbe09780: [30/Jul/2025:15:44:53.904 +0000] " GET /bkt?list-type=2&delimiter=%2F&prefix=ceph.conf&encoding-type=url HTTP/1.1" 200 535 - "aws-cli/2.15.31 Python/3.9.21 Linux/5.14.0-570.28.1.el9_6.x86_64 source/x86_64.rhel.9 prompt/off command/s3api.list-objects-v2" - latency=0.000999995s vvv beast: 0x7fffbbe09780: [30/Jul/2025:15:44:58.192 +0000] " GET /bkt?list-type=2&delimiter=%2F&prefix=ceph.conf&encoding-type=url HTTP/1.1" 503 228 - "aws-cli/2.15.31 Python/3.9.21 Linux/5.14.0-570.28.1.el9_6.x86_64 source/x86_64.rhel.9 prompt/off command/s3api.list-objects-v2" - latency=0.000000000s beast: 0x7fffbbe09780: [30/Jul/2025:15:44:58.798 +0000] " GET /bkt?list-type=2&delimiter=%2F&prefix=ceph.conf&encoding-type=url HTTP/1.1" 503 228 - "aws-cli/2.15.31 Python/3.9.21 Linux/5.14.0-570.28.1.el9_6.x86_64 source/x86_64.rhel.9 prompt/off command/s3api.list-objects-v2" - latency=0.000999994s beast: 0x7fffbbe09780: [30/Jul/2025:15:44:59.807 +0000] " GET /bkt?list-type=2&delimiter=%2F&prefix=ceph.conf&encoding-type=url HTTP/1.1" 503 228 - "aws-cli/2.15.31 Python/3.9.21 Linux/5.14.0-570.28.1.el9_6.x86_64 source/x86_64.rhel.9 prompt/off command/s3api.list-objects-v2" - latency=0.000000000s s3cmd put ./ceph.conf s3://bkt/1 s3cmd put ./ceph.conf s3://bkt/2 s3cmd put ./ceph.conf s3://bkt/3 s3cmd rm s3://bkt/1 s3cmd rm s3://bkt/2 s3cmd rm s3://bkt/3 delete: 's3://bkt/1' delete: 's3://bkt/2' WARNING: Retrying failed request: /3 (503 (SlowDown)) WARNING: Waiting 3 sec... WARNING: Retrying failed request: /3 (503 (SlowDown)) ^^^ ``` Fixes: https://tracker.ceph.com/issues/72894 Signed-off-by: Mark Kogan <mkogan@ibm.com> Update PendingReleaseNotes Co-authored-by: Yuval Lifshitz <yuvalif@yahoo.com> Signed-off-by: Mark Kogan <31659604+mkogan1@users.noreply.github.com> Update PendingReleaseNotes Co-authored-by: Yuval Lifshitz <yuvalif@yahoo.com> Signed-off-by: Mark Kogan <31659604+mkogan1@users.noreply.github.com> (cherry picked from commit dae572d) Conflicts: src/common/options/rgw.yaml.in - Remove “rgw_redis_connection_pool_size” because tentacle doesn’t have 1ac08d3 src/rgw/rgw_rest_ratelimit.cc - Removed "rgw_sal.h”, “rgw_sal_config.h” and imports because tentacle doesn’t have de90732 and 09383ab
…set to 0 Signed-off-by: morphes1995 <morphes1995@gmail.com> (cherry picked from commit 91e9f00)
When RGW rejects an S3 request with -ERR_RATE_LIMITED, return an HTTP Retry-After header that tells the client how many seconds to wait before retrying. The value is computed from the current token-bucket deficit and replenishment rate, so a polite client that honors the header will retry right around when its tokens recover. The delay is threaded out of the rate limiter through req_state and onto the response in abort_early(). The rate-limit check now returns int64_t seconds (0 = allowed) instead of bool, with a 1-second floor on the rejected path. The seconds-to-recover calculation is factored into a small static helper RateLimiterEntry::compute_delay(limit, needed, interval) so it can be unit-tested directly. Each dimension (ops, bytes) calls it independently and the caller takes the max, leaving room for additional limit dimensions in the future. The user-facing documentation is updated to describe the new header and to suggest proxy-layer remapping for operators who need 429 instead of the existing 503. Tracker: https://tracker.ceph.com/issues/64083 Tracker: https://tracker.ceph.com/issues/67655 Signed-off-by: Lumir Sliva <61183145+lumir-sliva@users.noreply.github.com> (cherry picked from commit 3dae903)
Signed-off-by: Tatjana Dehler <tatjana.dehler@clyso.com>
nodeenv 1.11.0 (released 2026-09-26) no longer leaves the node-env/src directory behind after installing a prebuilt node. The build step runs an unpinned `pip install nodeenv`, so as of today every build with WITH_MGR_DASHBOARD_FRONTEND fails at chown: cannot access '.../node-env/src': No such file or directory ninja: build stopped: subcommand failed. (e.g. ceph-pr-pipeline build 2504). The chown is a workaround for tarball uid mapping in containers and is only needed when the src directory exists, so skip it when there is nothing to fix up. Assisted-by: Claude Fable 5 Signed-off-by: David Galloway <david.galloway@ibm.com> (cherry picked from commit 5aff221) Conflicts: src/pybind/mgr/dashboard/frontend/CMakeLists.txt squid still creates the npm cache directory with "mkdir -p" and has no DEPENDS on setup-virtualenv.sh; kept those lines as they are and took only the guarded chown and its comment
uw-sc
approved these changes
Oct 12, 2026
uw-sc
merged commit Oct 12, 2026
116222c
into
tentacle-20.2.4-cobaltcore-storage
2 of 3 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes cobaltcore-dev/cloud-storage#588
Backport file
release-management/backports/<uuid>.mdadded (or updated) by this PRidin frontmatter matches the filename stemprovenance.upstream_prsset (orprovenance.type: otherwith another:block)Stage-B risk (filled per
release-management/backports/RISK-RUBRIC.md)blast— cosmetic / availability / data-lossconflict— clean / trivial / substantivecoverage— strong / partial / weakhighband)Merge
Land this via
just merge-backport PR=<N>— do not click the green merge button.The just recipe constructs the merge commit's
Backport-Idtrailer.