redis-migrate: read-only validator for a legacy Azure Cache for Redis

README.md

@@ -34,6 +34,7 @@ Extras: `labagent[rag]` (Databricks RAG), `[rag-local]` (laptop RAG), `[rag-lab]
 | Pack | What it builds | Where |
 |---|---|---|
 | `azure-labs` | Copy Azure VMs, disks and network from one resource group to another with snapshots, or snapshot a dead VM or disk. Source is read-only. An Azure Lab Services alternative. [Guide](docs/azure-labs.md) | Azure (needs `az login`) |
+| `redis-migrate` | Check whether a legacy Azure Cache for Redis (retiring) can be exported to Azure Managed Redis. Read only, creates nothing. Export and import come next. [Guide](docs/redis-migrate.md) | Azure (needs `az login`) |
 | `rag-sec-tutorial` | Nine notebooks: RAG on SEC 10-K filings with embeddings, pgvector, hybrid search, citations, an evaluation gate, a deployed app | Databricks Free Edition |
 | `db-agent` | [db-agent](https://github.com/db-agent/db-agent) text-to-SQL app on Databricks Apps, pinned commit. [Which model, warehouse and schema it picks](docs/packs.md) | Databricks Free Edition |
 | `dbx-lakehouse` | Medallion lakehouse (bronze, silver, gold) from real open data, with masks, a row filter, notebooks and exercises. [Guide](docs/dbx-lakehouse.md) | Databricks Free Edition, AWS, Azure |

docs/redis-migrate.md

@@ -0,0 +1,25 @@
+# redis-migrate: check a legacy Azure Cache for Redis before moving it
+
+Azure Cache for Redis is retiring: Basic, Standard and Premium on **2028-09-30**, Enterprise and Enterprise Flash on **2027-03-31**. Microsoft no longer lets new customers create one. This pack tells you, before you touch anything, whether a cache can be exported to Azure Managed Redis. Not Microsoft's tool.
+
+```bash
+az login
+labagent launch redis-migrate --param source_cache=my-cache --param source_rg=my-rg
+labagent launch redis-migrate --param source_cache=my-cache --param storage_account=mystorage --param target_region=eastus
+```
+
+Read only: it creates, changes and deletes nothing, so it needs no allow-list. It checks the cache, and the storage account if you give one. `--dry-run` shows the steps. Exit code 0 means no blockers (warnings may remain). Exit code 2 means blocked, and the error lists the blocker codes. Every finding is printed on stderr and saved in `~/.labagent/packs/redis-migrate/<name>.json`.
+
+| Level | Meaning |
+|---|---|
+| blocker | export/import cannot work as asked |
+| warning | it works, but something does not carry over |
+| info | context |
+
+**Blockers.** Basic or Standard tier (no export). Cache not in Succeeded state. VNet injection. Active geo-replication. Redis newer than 7.x. Storage account that is blob-only, Data Lake Gen2, or in another region than the cache.
+
+**Warnings.** Only database 0 is migrated. Clustered cache, and cross-slot commands (Azure Managed Redis is clustered by default). Keyspace notifications, RDB/AOF persistence, private endpoints, the non-TLS port 6379. A storage firewall, or a storage account in another subscription (needs a managed identity). A target region where Azure Managed Redis is not listed.
+
+**Not covered yet.** Enterprise and Enterprise Flash sources, the export, the import, and the checks after the import. Keys in databases above 0 cannot be counted without a Redis connection, so that is always a warning. Progress is tracked in [#36](https://github.com/becloudready/labagent/issues/36).
+
+**How far to trust it.** The rules come from Microsoft's documentation and the field names from Azure's own SDK models. They have not been run against a real legacy cache. The import into Azure Managed Redis was tested with a Redis 6.2 snapshot (3,400 keys, every data type and TTL matched).

labagent/packs/redis_migrate/facts.py

@@ -0,0 +1,44 @@
+"""Read-only Azure calls that collect the facts `rules.assess` needs. Anything optional fails soft: a missing fact becomes an info line."""
+from __future__ import annotations
+
+from typing import Any, Dict, Optional
+
+from ..azure_labs.azcli import AzCli
+from ..base import PackError
+from .rules import norm_region
+
+
+def _try(fn, default=None):
+    try:
+        return fn()
+    except PackError:
+        return default
+
+
+def find_cache(az: AzCli, name: str, rg: Optional[str]) -> Dict[str, Any]:
+    if rg:
+        return az.json("redis", "show", "-g", rg, "-n", name)
+    hits = [c for c in az.json("redis", "list") if c.get("name") == name]
+    if not hits:
+        raise PackError(f"no Azure Cache for Redis named {name} in this subscription")
+    if len(hits) > 1:
+        raise PackError(f"{len(hits)} caches are named {name}: add --param source_rg=<resource group>")
+    return hits[0]
+
+
+def gather(az: AzCli, p: Dict[str, Any]) -> Dict[str, Any]:
+    cache = find_cache(az, p["source_cache"], p.get("source_rg"))
+    rg = cache.get("resourceGroup") or p.get("source_rg")
+    links = _try(lambda: az.json("redis", "server-link", "list", "-g", rg, "-n", cache["name"]), []) or []
+    pes = _try(lambda: len(az.json("network", "private-endpoint-connection", "list", "--id", cache["id"])))
+    storage = None
+    if p.get("storage_account"):
+        rows = az.json("storage", "account", "list", "--query", f"[?name=='{p['storage_account']}']")
+        if not rows:
+            raise PackError(f"storage account {p['storage_account']} was not found in this subscription")
+        storage = rows[0]
+    regions = _try(lambda: az.json("provider", "show", "-n", "Microsoft.Cache", "--query",
+                                   "resourceTypes[?resourceType=='redisEnterprise'].locations[]"))
+    return {"cache": cache, "links": links, "private_endpoints": pes, "storage": storage,
+            "target_region": p.get("target_region") or cache.get("location"),
+            "amr_regions": {norm_region(r) for r in regions} if regions else None}

labagent/packs/redis_migrate/pack.py

@@ -0,0 +1,60 @@
+"""redis-migrate: find out whether a legacy Azure Cache for Redis can be moved to Azure Managed Redis.
+
+Azure Cache for Redis is retiring and Microsoft no longer lets new customers create one. This pack only reads: `launch` checks the
+cache (and, if given, the storage account that would hold the export) and reports blockers and warnings. It creates nothing.
+Export, import and post-import checks come in later steps. Not Microsoft's tool.
+"""
+from __future__ import annotations
+
+from typing import Any, Dict, List
+
+from ..base import Pack, PackContext, PackError, Param, Step
+from ..azure_labs.pack import _LazyAz
+from . import facts, rules
+
+
+class RedisMigrate(Pack):
+    id = "redis-migrate"
+    title = "Redis Migrate: check a legacy Azure Cache for Redis before moving it to Azure Managed Redis"
+    summary = "Check a legacy Azure Cache for Redis before moving it (read only)"
+    aliases = ("redis", "azure-redis")
+    platforms = ("azure",)
+    cost_tier = "free"
+    default_name = "check"
+    params = [
+        Param("source_cache", None, "Name of the Azure Cache for Redis to check. It is only read", required=True),
+        Param("source_rg", None, "Its resource group (default: found by name)"),
+        Param("storage_account", None, "Storage account that would hold the export. Checked if given"),
+        Param("target_region", None, "Region for the new Azure Managed Redis (default: the cache's region)"),
+        Param("subscription", None, "Subscription ID or name (default: the one `az login` selected)"),
+    ]
+    output_keys = ("verdict", "findings", "cache")
+
+    def services(self, profile):
+        return {"az": _LazyAz()}
+
+    def _validate(self, ctx: PackContext) -> Dict[str, Any]:
+        az = ctx.services["az"]
+        if getattr(az, "subscription", None) is None and ctx.params.get("subscription"):
+            az.subscription = ctx.params["subscription"]
+        az.account()
+        got = facts.gather(az, ctx.params)
+        findings = rules.assess(got)
+        c = got["cache"]
+        verdict = "blocked" if any(x["level"] == "blocker" for x in findings) else "warnings" if any(x["level"] == "warning" for x in findings) else "ok"
+        ctx.log(f"  {c.get('name')}: {(c.get('sku') or {}).get('name')} {(c.get('sku') or {}).get('family', '')}{(c.get('sku') or {}).get('capacity', '')}, "
+                f"Redis {c.get('redisVersion')}, {c.get('location')}")
+        for x in sorted(findings, key=lambda x: ("blocker", "warning", "info").index(x["level"])):
+            ctx.log(f"  {x['level'].upper():8} {x['code']}: {x['message']}")
+        ctx.state.update(verdict=verdict, findings=findings,
+                         cache={"name": c.get("name"), "tier": (c.get("sku") or {}).get("name"), "version": c.get("redisVersion"), "location": c.get("location")})
+        if verdict == "blocked":
+            blockers = [f"{x['code']}: {x['message']}" for x in findings if x["level"] == "blocker"]
+            raise PackError(f"blocked by {len(blockers)} check(s): " + "; ".join(blockers))
+        return {"verdict": verdict, "findings": findings, "cache": ctx.state["cache"]}
+
+    def deploy_steps(self, ctx: PackContext) -> List[Step]:
+        return [Step("validate", "Read the cache and check that it can be exported to Azure Managed Redis", self._validate)]
+
+    def destroy_steps(self, ctx: PackContext) -> List[Step]:
+        return [Step("destroy", "Nothing to remove: this pack only reads", lambda c: {})]

labagent/packs/redis_migrate/rules.py

@@ -0,0 +1,113 @@
+"""redis-migrate checks: facts about a legacy Azure Cache for Redis in, findings out. Pure functions: no Azure calls, easy to test.
+
+Every rule comes from Microsoft's docs (import/export, retirement FAQ, the Azure Managed Redis migration skill), not from running a
+real legacy cache: Microsoft no longer lets new customers create one. `blocker` means export/import cannot work as asked,
+`warning` means it works but something does not carry over, `info` is context.
+"""
+from __future__ import annotations
+
+import re
+from typing import Any, Dict, List
+
+RETIRES = "2028-09-30"          # Basic, Standard and Premium. Enterprise tiers retire 2027-03-31 (not checked here yet)
+CAN_EXPORT = ("premium",)       # import/export is Premium and Enterprise only
+IN_PLACE_HINT = "Microsoft's in-place migration (preview) or an empty Azure Managed Redis cache are the options for Basic and Standard"
+
+
+def norm_region(region: Any) -> str:
+    """'Canada Central' and 'canadacentral' compare equal."""
+    return re.sub(r"[^a-z0-9]", "", str(region or "").lower())
+
+
+def _on(value: Any) -> bool:
+    return str(value).strip().lower() == "true"
+
+
+def assess(f: Dict[str, Any]) -> List[Dict[str, str]]:
+    """f: cache (az redis show), links, private_endpoints (int or None), storage (az storage account show or None),
+    target_region, amr_regions (set of normalized names or None)."""
+    out: List[Dict[str, str]] = []
+
+    def add(level: str, code: str, message: str) -> None:
+        out.append({"level": level, "code": code, "message": message})
+
+    c = f["cache"]
+    conf = c.get("redisConfiguration") or {}
+    tier = str((c.get("sku") or {}).get("name", "")).lower()
+    sub = str(c.get("id", "")).split("/")[2] if str(c.get("id", "")).count("/") > 2 else ""
+
+    add("info", "retirement", f"Azure Cache for Redis {tier or 'cache'} retires {RETIRES} and is disabled the day after")
+
+    if c.get("provisioningState") != "Succeeded":
+        add("blocker", "not-running", f"cache state is {c.get('provisioningState')}, not Succeeded: wait for it to finish first")
+    if tier in ("basic", "standard"):
+        add("blocker", "no-export", f"{tier.title()} caches cannot export data (Premium and Enterprise only). {IN_PLACE_HINT}")
+    elif tier not in CAN_EXPORT:
+        add("warning", "unknown-tier", f"tier {tier or '(unknown)'} was not recognised: this check covers Basic, Standard and Premium")
+
+    if c.get("subnetId"):
+        add("blocker", "vnet-injection", "the cache is injected into a virtual network: Azure Managed Redis has no VNet injection, only Private Link")
+    if c.get("linkedServers") or f.get("links"):
+        add("blocker", "geo-replication", "geo-replication is active: unlink the caches before migrating")
+
+    pe = f.get("private_endpoints")
+    if pe:
+        add("warning", "private-endpoints", f"{pe} private endpoint(s): private endpoint settings are not copied, create them again on the new cache")
+    elif pe is None:
+        add("info", "private-endpoints", "could not read private endpoints")
+
+    if int(c.get("shardCount") or 0) > 0:
+        add("warning", "clustered", f"clustered cache ({c.get('shardCount')} shards): keys are redistributed on import. "
+                                      "Azure Managed Redis is clustered by default, so check your client for cross-slot commands")
+    else:
+        add("warning", "cross-slot", "Azure Managed Redis is clustered by default: commands that touch keys in different slots (some MGET, MULTI, Lua) can fail. "
+                                       "The cache cannot tell us whether your app uses them")
+    add("warning", "other-databases", "only database 0 is migrated: keys in databases 1 and above are lost. Run INFO keyspace on the cache to see if you have any")
+
+    if str(conf.get("notify-keyspace-events") or "").strip():
+        add("warning", "keyspace-notifications", "keyspace notifications are on: Azure Managed Redis does not have them")
+    if _on(conf.get("rdb-backup-enabled")) or _on(conf.get("aof-backup-enabled")):
+        add("warning", "persistence", "RDB/AOF persistence settings are not copied: set them again on the new cache")
+    if _on(c.get("enableNonSslPort")):
+        add("warning", "non-tls-port", "the non-TLS port 6379 is on: Azure Managed Redis allows one mode only (TLS on port 10000, set at creation)")
+    add("info", "new-endpoint", "the app must switch to the new hostname and port 10000, and use a new access key or Microsoft Entra ID")
+    add("info", "writes-lost", "writes after the snapshot are not copied, and import clears the target and makes it unreachable while it runs")
+
+    version = str(c.get("redisVersion") or "")
+    major = re.match(r"(\d+)", version)
+    if major and int(major.group(1)) <= 7:
+        add("info", "rdb-version", f"Redis {version} writes an RDB that loads into Azure Managed Redis (7.x): older into newer is supported")
+    elif major:
+        add("blocker", "rdb-version", f"Redis {version} is newer than Azure Managed Redis (7.x): its RDB may not load")
+
+    region, target = norm_region(c.get("location")), norm_region(f.get("target_region") or c.get("location"))
+    amr = f.get("amr_regions")
+    if amr is None:
+        add("info", "amr-region", "could not check which regions offer Azure Managed Redis")
+    elif target not in amr:
+        add("warning", "amr-region", f"Azure Managed Redis is not listed for {f.get('target_region') or c.get('location')}: pick another region (an RDB can cross regions)")
+    if target != region:
+        add("info", "cross-region", "target region differs from the cache: the storage account for the import must be in the target region")
+
+    _storage(f.get("storage"), c, sub, add)
+    return out
+
+
+def _storage(s: Any, cache: Dict[str, Any], sub: str, add) -> None:
+    if not s:
+        add("info", "storage", "no storage_account given: storage checks skipped")
+        return
+    name = s.get("name", "storage account")
+    if str(s.get("kind", "")).lower() == "blobstorage":
+        add("blocker", "storage-kind", f"{name} is a blob-only account: export needs a general-purpose account (page blobs)")
+    if s.get("isHnsEnabled"):
+        add("blocker", "storage-adls", f"{name} has a hierarchical namespace (Data Lake Gen2): export to it is not supported")
+    if norm_region(s.get("location")) != norm_region(cache.get("location")):
+        add("blocker", "storage-region", f"{name} is in {s.get('location')}, the cache is in {cache.get('location')}: export needs the same region")
+    s_sub = str(s.get("id", "")).split("/")[2] if str(s.get("id", "")).count("/") > 2 else ""
+    if s_sub and sub and s_sub.lower() != sub.lower():
+        add("warning", "storage-subscription", f"{name} is in a different subscription: export then needs a managed identity, not a SAS token")
+    rules = s.get("networkRuleSet") or {}
+    if str(rules.get("defaultAction", "")).lower() == "deny":
+        add("warning", "storage-firewall", f"{name} has a firewall: a Premium cache needs 'trusted services' allowed plus a managed identity with "
+                                          "Storage Blob Data Contributor. Enterprise tiers do not support a firewall at all")

labagent/packs/registry.py

@@ -14,7 +14,8 @@ def builtin() -> Dict[str, Pack]:
     from .dbx_workspace.pack import DatabricksWorkspace
     from .dbx_lakehouse.pack import DatabricksLakehouse
     from .azure_labs.pack import AzureLabs
-    return {p.id: p for p in (RagSecTutorial(), DbAgent(), RagIde(), AmlLab(), DatabricksWorkspace(), DatabricksLakehouse(), AzureLabs())}
+    from .redis_migrate.pack import RedisMigrate
+    return {p.id: p for p in (RagSecTutorial(), DbAgent(), RagIde(), AmlLab(), DatabricksWorkspace(), DatabricksLakehouse(), AzureLabs(), RedisMigrate())}
 
 
 def all_packs() -> Dict[str, Pack]:

tests/test_packs.py

@@ -75,7 +75,7 @@ def go(c):
 
 
 def test_registry_lists_the_current_packs_and_the_cli_refuses_unknown_ones(capsys, home):
-    assert sorted(registry.all_packs()) == ["aml", "azure-labs", "db-agent", "dbx-lakehouse", "dbx-workspace", "rag-ide", "rag-sec-tutorial"]
+    assert sorted(registry.all_packs()) == ["aml", "azure-labs", "db-agent", "dbx-lakehouse", "dbx-workspace", "rag-ide", "rag-sec-tutorial", "redis-migrate"]
     assert main(["packs"]) == 0 and "rag-sec-tutorial" in capsys.readouterr().out
     assert main(["launch", "nope", "--name", "x"]) == 2
 
@@ -149,7 +149,7 @@ def test_an_explicit_profile_is_never_overridden(home):
 def test_packs_json_describes_every_pack_with_platforms_and_params(capsys, home):
     assert main(["packs", "--json"]) == 0
     cat = {p["id"]: p for p in json.loads(capsys.readouterr().out)["packs"]}
-    assert sorted(cat) == ["aml", "azure-labs", "db-agent", "dbx-lakehouse", "dbx-workspace", "rag-ide", "rag-sec-tutorial"]
+    assert sorted(cat) == ["aml", "azure-labs", "db-agent", "dbx-lakehouse", "dbx-workspace", "rag-ide", "rag-sec-tutorial", "redis-migrate"]
     assert cat["rag-ide"]["platforms"] == ["local", "ec2", "modal"] and cat["rag-ide"]["cost_tier"] == "metered"
     assert {"name": "hours", "default": 4, "help": "Hard cap per session, 1 to 24", "required": False} in cat["rag-ide"]["params"]
     assert all(p["title"] and p["summary"] and p["platforms"] and p["cost_tier"] for p in cat.values())