[next gen on clould] connection count can not balance after tidb memory increased

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run batch insert

2. What did you expect to see? (Required)

connection count can balance

3. What did you see instead (Required)

connection count can not balanced after tidb memory increased

4. What is your version? (Required)

mysql> select distinct concat('[',type,']') component_type, concat('version:',version,',') version,concat('git_hash:',git_hash) git_hash from information_schema.cluster_info;
+-------------------+----------------------------------------------+---------------------------------------------------+
| component_type | version | git_hash |
+-------------------+----------------------------------------------+---------------------------------------------------+
| [tiflash_compute] | version:9.0.0-beta.2.pre-79-gc68b24e1a, | git_hash:c68b24e1a901797f84d01bf9b950fb776597d418 |
| [tidb] | version:8.5.4-nextgen.202510.0-7-gc634954, | git_hash:c634954c6ea64c0d4547b5273e92421cca2cb12e |
| [pd] | version:8.5.4-nextgen.202510.1-1-gf912cdd, | git_hash:f912cdd1e9cdae8d4927b888545e6f003bea375a |
| [tso] | version:8.5.4-nextgen.202510.1-1-gf912cdd, | git_hash:f912cdd1e9cdae8d4927b888545e6f003bea375a |
| [scheduling] | version:8.5.4-nextgen.202510.1-1-gf912cdd, | git_hash:f912cdd1e9cdae8d4927b888545e6f003bea375a |
| [tikv] | version:8.5.0, | git_hash:15e3ce20089cc5ec97e2f07a5880cf6f732c0780 |
| [tiflash_write] | version:8.5.4-nextgen.202510.0-3-g492b32cd2, | git_hash:492b32cd2f8576cc3cfd4bc1bf936060c9c7cfe7 |
+-------------------+----------------------------------------------+---------------------------------------------------+
7 rows in set (0.09 sec)

mysql>

[Lily2025]

/assign YangKeao
/severity critical

[djshow832]

I've seen a similar issue. TiProxy keeps reporting OOM risk, but the risk level is 1, and the root cause is actually CPU-related.
Most of the CPU usage is caused by only one connection, when the connection is migrated, the CPU changes sharply. Then the connections are migrated back slowly. Once the high-load connection is migrated, the CPU changes back.

[djshow832]

The next-gen case is different. It's caused by memory.

Here's the connection count and monitored memory usage from one TiProxy to one TiDB:

All the 20 connections are on one TiDB. When the memory increases sharply (from 19% to 45% in 30 seconds), TiProxy thinks TiDB has an OOM risk and migrates all connections very quickly.

From this TiProxy to another TiDB, once all the connections are migrated in, its memory increases sharply and all the connections are migrated away again.

The result is that all the connections are migrated back and forth, even if the batch insert ends.

The dilemma is, if the migration is too slow, the connections can't be preserved when OOM. If the migration is too fast, it leads to the OOM risk on another TiDB.