Conversation
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
🔍 Preview links for changed docs |
Elastic Docs Style Checker (Vale)Summary: 2 warnings, 16 suggestions found
|
| File | Line | Rule | Message |
|---|---|---|---|
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 169 | Elastic.Spelling | 'substeps' is a possible misspelling. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-self-managed-eck.md | 153 | Elastic.Spelling | 'substeps' is a possible misspelling. |
💡 Suggestions (16): Optional style improvements. Apply when helpful.
| File | Line | Rule | Message |
|---|---|---|---|
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 91 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 97 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 198 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 199 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 290 | Elastic.WordChoice | Consider using 'deactivates, deselects, hides, turns off, makes unavailable' instead of 'disables', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 301 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 313 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 320 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 331 | Elastic.WordChoice | Consider using 'deactivates, deselects, hides, turns off, makes unavailable' instead of 'disables', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 339 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 426 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 440 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 444 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 446 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 456 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 463 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
@stefnestor : I'm adding you here as a reviewer because you also collaborated when we created the ECH/ECE version of the procedure. Note that I didn't merge all into a single document because the resulting document would need a lot of applies switches and part of the narrative should be modified due to operational differences between Elastic Cloud + ECE and self-managed systems and ECK, but this could be evaluated in the future. Also note that I found a few logical and technical caveats in the original procedure and I've tried to improve them. Hoping you like the result Stef and @yetanothertw |
| ### Remove a tier with regular indices [non-searchable-snapshot-data-tier] | ||
|
|
||
| The frozen tier only stores [partially mounted](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) {{search-snaps}}. [Fully mounted](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) {{search-snaps}} can be allocated to the hot or cold tier depending on the {{ilm-init}} phase, while the cold tier can also hold regular indices. Use the checks in [Before you remove a data tier](#before-you-remove-a-data-tier) if you are unsure what is on the tier. | ||
| This section covers the removal of a tier that holds regular indices. The goal is to ensure all shard allocation rules allow the data to move to other tiers before you disable the tier. You also need to temporarily stop {{ilm-init}} to prevent new indices from being routed to the tier while you work. |
There was a problem hiding this comment.
We in Support do not disable ILM 👻. Especially where drain off time can be extensive & you might full disk hot nodes when you can't stop ingest.
Instead we normally recommend users exclude nodes, ex:
PUT _cluster/settings
{"transient": {"cluster": {"routing": {"allocation": {"exclude": {"_name": "instance-0000000043"} } } } } }
Note: this will incur ILM errors (and if you're hot>cold>frozen and removing frozen then agreed you'd still fill up hot, which is why we tell users to disable tiers the same as you'd rolling restart: from lowest-to-highest temp).
There was a problem hiding this comment.
This is interesting because in the original procedure we were also suggesting to stop ILM, so this isn't a change I've introduced.
But I get your point. Stopping ILM for long time can be critical and fill up the hot tier in very active clusters causing serious and unneeded damage.
That's a very good point, I'll find a way to cover it.
This could also be related with the ILM policies review (that I have at the end of the procedure atm). Maybe if the user first removes the tier from the ILM policies first there's no need to stop ILM, and would be the smoothest way to approach this.
So if a user is going to remove let's say the cold tier, I'd suggest to first update the ILM policies to not use cold anymore and then proceed with the procedure without stopping ILM.
| ``` | ||
| 1. The example represents an index in the `warm` tier. | ||
|
|
||
| Before disabling the tier, update `_tier_preference` so that the tier where you want the data to move is the first available tier in the list. This allows {{es}} to begin relocating the shards before the deployment plan removes the tier. |
There was a problem hiding this comment.
TBC shards will not allocate to a fallback list-item unless/until no nodes of preceding exist in cluster. So this sentence reads catch-22 IMO. You must update _tier_preference to migrate shards off current deprecating tier, it is not to "begin relocating" early, it is a required step.
ref: https://www.elastic.co/docs/manage-data/lifecycle/data-tiers#data-tier-allocation
"This setting also accepts multiple tiers in order of preference. This prevents indices from remaining unallocated if there are no nodes in the cluster for the preferred tier."
rel: #1903
There was a problem hiding this comment.
@stefnestor , when you say:
You must update _tier_preference... it is a required step.
Yep, this is a required* step, not optional, and we say:
Before disabling the tier, update
_tier_preferenceso that the tier where you want the data to move is the first available tier in the list.
We tell users to ensure the destination tier is the first in the list, and then we give an example of moving data from warm to cold before removing the warm data.
So, I don't get your comment, as I agree with it but I think the text already states it properly. If you want us to rephrase please share some options.
On the other hand:
shards will not allocate to a fallback list-item unless/until no nodes of preceding exist in cluster.
That's totally accurate, I was actually thinking on including that type of sentence in a note so people understands how tier_preference really works.
There was a problem hiding this comment.
Anyway I'm updating this section to be clearer and avoid the "this allows {{es}} begin relocating" statement for a clearer one.
| ::: | ||
|
|
||
| 2. Updating node attribute allocation requirement rules. | ||
| 2. Update node attribute allocation requirement rules. |
There was a problem hiding this comment.
Potential noop comment : This is considered user customization responsible and not default anymore. This is why https://www.elastic.co/docs/manage-data/lifecycle/data-tiers#data-tier-allocation-troubleshooting was simplified down to only ~"by now you should have migrated to data tiers & ensure you don't induce allocation conflicts". From Support's side, we usually say, "update tier preferences, let shards drain off, check allocation explain for anybody stubborn not moving off to see why".
This is the first common example: https://www.elastic.co/docs/troubleshoot/elasticsearch/cluster-allocation-api-examples#conflicting-settings
There was a problem hiding this comment.
Yes, I understand node attribute based allocation shouldn't be in place anymore by ILM, but it might be a user customization so it's worthy to mention it here.
That's why we treat this as possible "custom configuration"
I'll check if it's worthy merging this section with the next step, which is for "other custom allocation rules" (they are both related actually).
| Confirm that no shards are left on the nodes to be removed after the allocation completes: `GET /_cat/shards` (filter by `node` as needed) should show that the tier is empty. Updating settings starts the relocation process, but you must wait until [shard allocation and recovery](/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery.md) finish. If shards stay on the original tier, use the [cluster allocation explain](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-allocation-explain) API to determine the cause. Common reasons can be [disk watermarks](/troubleshoot/elasticsearch/fix-watermark-errors.md) or [`index.routing.allocation.total_shards_per_node`](elasticsearch://reference/elasticsearch/index-settings/total-shards-per-node.md#total-shards-per-node) on the destination nodes. | ||
| If your allocation setting changes start relocation, wait until [shard allocation and recovery](/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery.md) finish. Use `GET /_cat/allocation?v=true&s=node` to monitor the nodes that the plan will remove. Shards might remain if you only removed a `require` rule because that change does not force them to move. The deployment plan relocates them when it disables the tier. | ||
|
|
||
| If shards that you expect to move remain on the original tier, use the [cluster allocation explain]({{es-apis}}operation/operation-cluster-allocation-explain) API to determine the cause. Common causes include [disk watermarks](/troubleshoot/elasticsearch/fix-watermark-errors.md) or the [`index.routing.allocation.total_shards_per_node`](elasticsearch://reference/elasticsearch/index-settings/total-shards-per-node.md#total-shards-per-node) limit on the destination nodes. |
There was a problem hiding this comment.
These are the common examples: https://www.elastic.co/docs/troubleshoot/elasticsearch/cluster-allocation-api-examples
| 6. Once the plan change completes, confirm that there are no remaining nodes associated with the disabled tier and that `GET _cluster/health` reports `green`. If this is the case, re-enable {{ilm-init}}. | ||
| 6. Once the plan change completes, confirm that `GET /_cat/nodes?v` shows no nodes associated with the disabled tier and that `GET /_cluster/health` reports `green`. | ||
|
|
||
| 7. Review your {{ilm-init}} policies and consider removing references to the disabled tier to keep them consistent with the deployment topology. This is especially important in older deployments where {{ilm-init}} uses node-attribute-based allocation, as those policies cannot run phases that target nodes that no longer exist. |
There was a problem hiding this comment.
FYI: Should probably do this before you start.
👻 Also need to review/modify index+legacy+component templates
There was a problem hiding this comment.
yes, moved to the beginning. I think the approach is much more robust this way, first the user should ensure no future data is going to reach the tier to remove, then they can vacate existing data in the tier and remove the nodes / tier.
| GET /_all/_ilm/explain?human=true&expand_wildcards=all&only_errors=true | ||
| ``` | ||
|
|
||
| Confirm that `operation_mode` is `RUNNING`. Investigate any reported errors and verify that no policy still attempts to allocate data to the disabled tier. |
There was a problem hiding this comment.
If you just ran a plan, you likely just start by doing a global Retry API
| * **[Partially mounted {{search-snaps}}](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) on the frozen tier:** The only way to keep the data available as indices when removing the frozen tier is to restore all partially mounted indices as regular indices on another tier. Follow the steps in this section to restore the indices and remove the original {{search-snap}} indices. | ||
| * **[Fully mounted {{search-snaps}}](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) on the cold tier:** To keep the indices as {{search-snaps}}, move them to another tier by following [Remove a tier with regular indices](#non-searchable-snapshot-data-tier). This works because fully mounted indices follow the same shard placement rules as regular indices. Alternatively, follow the steps in this section to restore them as regular indices on another tier. | ||
|
|
||
| If you do not need to preserve the data, delete the {{search-snap}} indices and continue from the shard verification before disabling the tier. |
There was a problem hiding this comment.
You'd also delete the index-backing searchable snapshot. If you only delete the index, you'll leave stray data in your repository (which adds to your bill/disk).
| ::: | ||
|
|
||
| 5. Restore indices from the {{search-snaps}}. | ||
| 5. Restore each index in the inventory from its source snapshot. |
There was a problem hiding this comment.
I'm a bit lost by this; it doesn't seem updated. This is the way old steps Support had raised concern on. Can we cross-compare https://support.elastic.dev/knowledge/view/39606517 instead. (Ex: you don't add rollover_alias you set indexing_complete otherwise you'll ILM error 😕)
There was a problem hiding this comment.
@stefnestor , that article doesn't look aligned with this topic. Maybe wrong paste?
There was a problem hiding this comment.
I think I found it: https://support.elastic.dev/knowledge/view/68395ecc
From that article I like the call to _ilm/remove (we didn't have that) together with the fact that in the article the ILM policy association + rollover_alias are removed from the index during the restore (what we are already doing). I'll be taking that from the article.
But the KB seems to mix two different approaches. I'll try to explain why but let me know if you feel I'm wrong.
If we restore the index without index.lifecycle.name and index.lifecycle.rollover_alias (both set to null as we are already doing), and then call _ilm/remove to clear its ILM execution state, indexing_complete is not needed at all. And that's the most efficient procedure from my point of view.
indexing_complete would be relevant when retaining or reapplying a rollover policy and intentionally marking the restored index as no longer requiring rollover, but it's not the case here. Even then, the rollover alias must be configured or indexing_complete without a rollover alias would end up in an error state (per this doc).
Anyway, the proposal I'll share for this rehydrate would be:
- Restore the index with no ILM association, and clean (
_ilm/remove) the ILM state on the restored index (almost the same as in your KB). - Then tell the user to apply the policy they want, recommending to use a policy without rollover as the nature of searchable snapshots data is data that was already rolled over, so there's no need to roll it over again.
I'll let you know when I have the final version so you can review. Thanks!
|
👋 @eedugon, thanks for this! It looks great. Added notes about technical truths of certain lines and/or more links to share. Housekeeping: Apologies. As of the top of this month, my GitHub notification flow has gotten broken. So please feel free to ping me if you don't hear back as soon as you're expecting. Or TBH, if I probably missed anywhere else you or your team pinged me. Sorry! 🙈 |
|
Making this a draft again until we merge #8327 |
2 participants
Summary
Main procedure improvements
_tier_preference, legacyrequirerules, and custom allocation filters affect relocation, including the difference between relocation triggered by index settings and by the deployment or orchestration plan.Closes #6186
Validation
git diff --check