[mohabsafey]

Summary

This PR speeds up the computation of Groebner bases over QQ (msolve -g 2) and reduces its memory use. Most of the gain comes from needing fewer primes and from cheaper rational reconstruction. Peak memory is lower because the modular images are stored more compactly.

On 38 benchmark systems over QQ (5 random seeds each, single thread), compared with v0.11.0:

• Time: on average 1.76 times faster (geometric mean over all runs), and the total time of the benchmark suite drops from 39,082 s to 21,971 s (1.78 times faster). Per system, the speedup ranges from 1.00 to 3.00 on systems taking more than 2 s. It reaches 5.48 on random2_6_6, which takes about 1 s.
• Number of primes: up to 1.96 times fewer (cp_d_4_n_4_p_3: 2113–2129 down to 1081–1090), and it barely varies with the random seed any more (minrank_d_3_p_3_q_6_r_3: 2460–3282 down to 1633–1643).
• Peak memory: lower on every system, by 1.76 times on average and up to 4.00 times (eco14: 4190 MB down to 1048 MB). The sum of the peaks over the suite drops from 32,184 MB to 15,111 MB.
• Outputs: identical to those of v0.11.0 on every run.

The largest speedups are on systems that need many primes: MinRank, critical points (cp_d, crit) and dense random systems (random2, random3), where the new version is 1.95 to 3.00 times faster. Systems that need few primes (eco, kat, noon, cyclic) gain 1.00 to 1.43 in time, but use 1.4 to 4 times less memory.

Within a family, the absolute time saved grows with the problem size, and so does the memory gain on most families (random2: 1.82×, 2.03× and 2.18× less memory from random2_7_7 to random2_9_9). The speedup itself grows with the size on the eco (1.13× to 1.27× from eco12 to eco14), chandra and random3 families, stays roughly constant on cp_d, and slightly decreases on MinRank and random2 (3.00× on random2_7_7, 2.61× on random2_9_9), whose smaller systems benefit most from the cheaper reconstruction.

Changes

The PR contains four commits, all on the multi-modular path (src/msolve/lifting-gb.c and src/msolve/duplicate.c, plus F4 fixes in src/neogb/f4.c and src/neogb/hash.c).

1. Reduce memory used by multi-modular Groebner bases over QQ

• Modular images: the images of each polynomial are stored in one contiguous prime-major block, instead of one array per term. The block grows by max(4, nprimes/16) primes instead of 16*nthrds, since the multi-modular loop is single-threaded and most of the images allocated for -t N were never used.
• CRT images: the per-term array of CRT images is gone. Each coefficient is lifted into a scratch integer and reconstructed immediately.
• Lifted coefficients: only integer numerators are stored, with the common denominator kept once per polynomial.

This commit also fixes several bugs in the restart paths of core_groebner_qq (an uninitialized size, a double free, leaks, a stale tracer), a NULL dereference when F4 detects a bad prime while applying the tracer, and a leak in duplicate_tracer.

2. Fix per-prime leaks in F4 and avoid touching unused hash table memory

• Final reduction: the reducer rows built by symbolic preprocessing were leaked once per prime. They are now freed.
• Bad primes: when the tracer detects a bad prime, the local data and the matrix are now released.
• Hash table copy: copy_hash_table, called once per prime, now copies only the used part of the table.

3. Speed up the lifting of Groebner bases over QQ

• FLINT reconstruction: rational reconstruction goes through FLINT's half-gcd based fmpq_reconstruct_fmpz_2 instead of the quadratic Euclidean algorithm, with a fallback whenever the bounds do not meet FLINT's requirement 2ND < m.
• Three witness coefficients: three coefficients per polynomial are reconstructed and checked instead of one: the last nonzero one, and the nonzero ones found scanning downwards from 2*len/3 and from len/2. A spurious reconstruction is thus detected before an expensive full lift is attempted.
• Denominator guesses: the guessed denominator of a polynomial is taken from the previous polynomial (its lifted denominator, or the common denominator of its verified witnesses). It is combined with the first witness's denominator for the next witnesses, and it is no longer reset to 1 after a failure. A spurious reconstruction therefore never becomes the guess.
• Full lift: the full lift of a polynomial uses the denominator accumulated so far as a guess, so only the missing factor is reconstructed for each coefficient.
• Safety margin: reconstruction bounds keep 16 bits of slack, which makes spurious reconstructions unlikely.
• Deterministic period: witnesses are reconstructed every rr primes, rr being the largest power of 2 <= nprimes/128. Previously the period depended on timings, so the number of primes depended on the machine load. It is now reproducible.

4. Choose rational reconstruction bounds from the guessed denominator

When a denominator g is guessed, the value reconstructed is u * g. Its numerator is about g times the denominator still missing, and the missing denominator is small when the guess is good.

So far the bounds were either balanced (N = D = sqrt(M), with M = mod_p / 2^(1+RR_MARGIN)) or fixed to D = M^(1/3) and N = M^(2/3), whatever the size of g. They are now set to D = sqrt(M / g) and N = M / D, and recomputed whenever the guess changes. Since N * D stays the same, the bound on the numerator grows with g instead of being wasted on the denominator. Successful reconstructions therefore happen with fewer primes, by about a third on the systems that need many primes.

Benchmarks

Setup:

• Command: msolve -g 2 -t 1 --random-seed s with seeds 1 to 5.
• Peak memory: the maximum resident set size, measured by /usr/bin/time.
• Reported values: times are means over the seeds, and speedups are geometric means of per-seed ratios.
• Concurrency: at most 4 runs at a time. KLYZ22 was rerun two runs at a time because it was sensitive to interference with large neighbouring runs.

System selection: systems over QQ taking between 1 s (with v0.11.0) and 20 min (with the new version), screened family by family in increasing size.

v0.11.0 baseline: the only other changes on master since v0.11.0 are in the term parser and the choice of sorting routine, so they don't affect these measurements.

Overall results:

	v0.11.0	this PR	gain
total time of the suite	39,082 s	21,971 s	1.78×
speedup per run (geometric mean)			1.76×
speedup per system (systems > 2 s)			1.00× – 3.00×
sum of peak memory	32,184 MB	15,111 MB	2.13×
peak memory per system (geometric mean)			1.76×
peak memory per system			1.12× – 4.00×
primes, cp_d_4_n_4_p_3	2113–2129	1081–1090	1.96×
primes, minrank_d_3_p_3_q_6_r_3 (spread over seeds)	2460–3282	1633–1643	1.60×

Results system per system

system	primes v0.11.0	primes new	time v0.11.0 (s)	time new (s)	speedup	peak mem v0.11.0 (MB)	peak mem new (MB)	mem ratio
random2_6_6	729	417–419	1.2	0.2	5.48	14	12	1.12
minrank_d_3_p_3_q_5_r_3	397–401	252–254	1.0	0.5	2.24	25	18	1.37
chandra-10	89	80	1.0	0.8	1.21	42	28	1.51
eco11	11–13	10	1.3	1.1	1.20	49	25	1.92
minrank_d_2_p_3_q_6_r_3	586–601	409–413	2.6	1.2	2.18	45	29	1.51
minrank_d_2_p_2_q_6_r_2	689–697	445–447	2.7	1.2	2.30	35	23	1.51
noon8	5	5	1.5	1.4	1.07	123	67	1.82
random3_5_5	757–953	489–491	4.0	1.8	2.23	36	26	1.37
random2_7_7	1441–2145	985–989	5.8	1.9	3.00	42	23	1.82
chandra-11	101–102	80	4.4	3.2	1.39	138	89	1.55
cp_d_3_n_5_p_2	929–945	633–637	7.5	3.3	2.26	67	44	1.53
minrank_d_3_p_2_q_5_r_2	809–1081	537–541	9.2	3.6	2.55	90	53	1.70
kat11	35–44	28	7.1	5.0	1.43	139	62	2.26
eco12	13–15	13	10.1	9.0	1.13	156	76	2.05
cyclic8	46–50	46	10.0	9.6	1.05	67	49	1.36
vor1	130–135	124–125	11.6	10.9	1.07	102	61	1.67
cp_d_4_n_4_p_3	2113–2129	1081–1090	28.9	11.4	2.54	163	99	1.66
noon9	5–6	6	11.4	11.4	1.00	638	334	1.91
random2_8_8	3905–5057	2593–2609	50.2	20.8	2.40	251	124	2.03
minrank_d_2_p_2_q_7_r_2	1745–2353	1321–1329	57.0	24.7	2.31	354	191	1.85
kat12	51–56	43	61.0	50.1	1.22	499	254	1.96
minrank_d_2_p_3_q_7_r_3	2529–2545	1633–1641	116	58.4	1.99	865	444	1.95
eco13	24	18	90.7	74.1	1.22	425	236	1.80
minrank_d_2_p_3_q_4_r_2	3521–4545	2289–2305	184	81.3	2.27	817	446	1.83
crit_deg_3_n_6_p_6	3554–4321	2209	182	81.4	2.22	858	391	2.19
bruno	1087–1093	1089–1097	105	82.3	1.27	132	101	1.30
random3_6_6	4306–4337	2209–2225	200	83.1	2.41	712	390	1.82
cp_d_3_n_6_p_2	3393–3458	2225–2241	205	92.4	2.21	1074	561	1.91
minrank_d_3_p_3_q_6_r_3	2460–3282	1633–1643	231	102	2.24	1648	845	1.95
henrion7	227–243	202–203	130	111	1.17	643	532	1.21
KLYZ22	726–729	725–729	165	159	1.04	211	147	1.43
random2_9_9	9857–12546	6241–6273	568	217	2.61	1831	841	2.18
zimmermann	1515–1525	1489–1497	279	228	1.22	83	63	1.33
crit_d_4_n6	14338–17412	8834–8898	1014	401	2.52	3595	1527	2.35
minrank_d_3_p_2_q_6_r_2	6188–6244	3201–3222	1153	494	2.33	4879	2297	2.12
kat13	70–89	57–59	712	540	1.32	2427	1162	2.09
minrank_d_2_p_2_q_8_r_2	5473–7300	3713–3731	1110	567	1.95	4721	2391	1.97
eco14	35–36	26–27	1082	849	1.27	4190	1048	4.00

The table is sorted by the time of the new version. Times are in seconds and peak memory in MB; "speedup" and "mem ratio" are v0.11.0 divided by new.

The changes in this PR were made with the assistance of generative AI (Claude Code).

🤖 Generated with Claude Code