Docs/new generate scripts

.agents/skills/generate-scripts-2/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: generate-scripts-2
+description: Generate self-contained, modern libEnsemble 2.x workflows for sampling, optimization, external applications, and HPC resources
+---
+
+Generate runnable libEnsemble 2.x scripts from the user's requirements. Resolve every
+`references/...` path relative to this loaded `SKILL.md`; never assume the skill lives at a
+specific `.claude`, `.agents`, or repository path. Use only the standardized gest-api/VOCS
+interfaces described here. Do not depend on access to the
+libEnsemble repository and do not emit legacy `libE()`, `gen_f`, `sim_f`, bare-spec
+dictionary, or explicit allocation-function patterns.
+
+## Workflow
+
+1. Read `references/intake-and-validation.md`. Extract the problem definition,
+   termination budget, parallelism, generator intent, simulator interface, dependencies,
+   files, and resources. Ask only questions whose answers materially change the script.
+   Never invent bounds, objective direction, executable paths, output parsing, or HPC
+   resource requirements. Missing values block execution, not necessarily generation:
+   produce a clearly marked scaffold when its structure is still useful and safe.
+
+2. Read `references/generator-selection.md`. Preserve a user-specified gest-api generator
+   and its VOCS. Otherwise choose the least-complex suitable generator, preferring
+   libEnsemble's built-in sampling, APOSMM, or preloaded-sample classes before optional
+   ecosystems. Read `references/aposmm.md` or `references/external-generators.md` when
+   applicable. Check that every non-core package is installed before relying on it.
+
+3. Read `references/canonical-patterns.md`. For a basic Python simulator, adapt
+   `examples/local_sampling.py`. For an existing set of points, adapt
+   `examples/preloaded_points.py`. Keep the complete script structure: `Ensemble`, VOCS,
+   typed `SimSpecs`/`GenSpecs`/`LibeSpecs`, `generator=` and `simulator=`, a main guard,
+   explicit `ensemble.run(...)` stopping criteria, and manager-only result reporting/saving.
+
+4. For an executable, read `references/executors-and-files.md`. Use `Executor` for a
+   normal subprocess and `MPIExecutor` only when the application itself needs an MPI or
+   resource-aware launcher. Keep application launch separate from manager/worker launch.
+   Use isolated simulation directories for file-producing applications. Never modify an
+   original user input file; copy or render it into each simulation directory.
+
+5. For clusters, GPUs, variable task sizes, or scheduler scripts, also read
+   `references/hpc-resources.md`. Do not guess machine topology, launcher, scheduler
+   directives, process counts, GPU counts, or platform settings.
+
+6. Validate the generated files against `references/intake-and-validation.md`. In
+   particular, verify exact names across VOCS, simulator inputs/returns, generator
+   mappings, executable registration/submission, parser output, and result analysis.
+   Run a syntax/import check when tools are available. For built-in adapter generators,
+   also reject ambiguous field mappings such as a scalar VOCS variable named `x` in a
+   multi-variable problem unless an explicit tested mapping resolves the collision.
+
+7. Summarize generated files, generator choice, variables/bounds, objectives and
+   directions, batch size, workers, stopping criteria, dependencies, and application
+   resources. Identify every placeholder the user must replace. For production artifacts,
+   include a minimal dependency manifest with tested versions when the user wants one.
+
+8. Ask before executing a generated workflow. For a fixed local script run
+   `python script.py`; do not append `-n`, `--comms`, or use `mpirun`/`srun` unless the
+   script intentionally uses `Ensemble(parse_args=True)` and the user requested that
+   launch mode. `MPIExecutor` may launch simulation applications with MPI even when the
+   calling script itself runs with plain Python.
+
+9. If execution is approved, read `references/running-and-results.md`, run the smallest
+   useful validation first, fix actionable failures, and report only completed, finite
+   results. Do not claim a `.npy` result exists unless `save_output()` ran successfully.
+   Treat an exception raised from `ensemble.run()` separately: manager-only code after the
+   call will not execute, though libEnsemble may write its own abort checkpoint.
+
+## Non-negotiable defaults
+
+- Generate modern standardized interfaces only. If the request requires a legacy-only
+  feature, explain that this skill does not generate it and ask whether a modern design
+  is acceptable.
+- Prefer programmatic `LibeSpecs(nworkers=...)`; use `parse_args=True` only when requested.
+- Standardized generators run on manager Worker 0 by default, so all `nworkers` are
+  normally available for simulations. Do not subtract a generator worker unless setting
+  `gen_on_worker=True` intentionally.
+- Omit `AllocSpecs` unless a documented modern requirement cannot be represented through
+  `GenSpecs`/`LibeSpecs`.
+- Do not set `async_return=True` globally. Choose it only when the generator supports
+  one-at-a-time feedback after initialization.
+- Use `safe_mode=True` for generated production workflows unless a documented requirement
+  needs protected-field writes. Never return protected History metadata from a simulator.
+- Use a reproducible seed where the selected generator supports one.
+- Keep examples' numbers and paths out of user scripts unless they match the request.
+
+## References
+
+Read only what the request needs; all paths are relative to this skill directory.
+
+- `references/intake-and-validation.md` — requirements and final checks
+- `examples/local_sampling.py` — runnable built-in sampling workflow
+- `examples/preloaded_points.py` — runnable workflow for evaluating supplied points
+- `references/canonical-patterns.md` — adaptation guidance and generator/VOCS pitfalls
+- `references/generator-selection.md` — generator decision table and batch behavior
+- `references/aposmm.md` — standardized APOSMM configuration
+- `references/external-generators.md` — Xopt, Optimas, and gpCAM
+- `references/executors-and-files.md` — subprocesses, MPI applications, files, failures
+- `references/hpc-resources.md` — schedulers, resource sets, GPUs, platform settings
+- `references/running-and-results.md` — validation, execution, saving, interpretation
+
+## User request
+
+$ARGUMENTS

.agents/skills/generate-scripts-2/examples/local_sampling.py

@@ -0,0 +1,56 @@
+"""Run a small, reproducible Latin-hypercube sampling workflow locally.
+
+Run with ``python local_sampling.py`` from a fresh working directory. The run
+writes ``local_sampling.npy`` and ``local_sampling.pickle`` in that directory.
+"""
+
+import numpy as np
+from gest_api.vocs import VOCS
+
+from libensemble import Ensemble
+from libensemble.gen_classes.sampling import LatinHypercubeSample
+from libensemble.specs import GenSpecs, LibeSpecs, SimSpecs
+
+
+def simulate(inputs: dict, **kwargs) -> dict:
+    """Evaluate a simple objective using the two sampled variables."""
+    x0 = inputs["x0"]
+    x1 = inputs["x1"]
+    return {"f": x0**2 + x1**2}
+
+
+if __name__ == "__main__":
+    vocs = VOCS(
+        variables={"x0": [-3.0, 3.0], "x1": [-2.0, 2.0]},
+        objectives={"f": "MINIMIZE"},
+    )
+
+    # Keep the complete LHS in one initial batch. Splitting it across requests
+    # would create multiple smaller designs instead of one 12-point design.
+    generator = LatinHypercubeSample(vocs, random_seed=1)
+    ensemble = Ensemble(
+        sim_specs=SimSpecs(simulator=simulate, vocs=vocs),
+        gen_specs=GenSpecs(
+            generator=generator,
+            vocs=vocs,
+            initial_batch_size=12,
+            batch_size=12,
+        ),
+        # Using local workers lets this example run with plain Python. The default
+        # ensemble directory is created in the fresh working directory.
+        libE_specs=LibeSpecs(comms="local", nworkers=4, safe_mode=True),
+    )
+
+    history, _, exit_flag = ensemble.run(sim_max=12)
+
+    if ensemble.is_manager:
+        completed = history[history["sim_ended"]]
+        finite = completed[np.isfinite(completed["f"])]
+        history_path = ensemble.save_output("local_sampling", append_attrs=False)
+        print(f"Saved History to {history_path}")
+        print(f"Completed {len(completed)} of 12 evaluations; exit flag: {exit_flag}")
+        if len(finite):
+            best = finite[np.argmin(finite["f"])]
+            print(f"Best point: x0={best['x0']:.4f}, x1={best['x1']:.4f}, f={best['f']:.6f}")
+        if exit_flag != 0:
+            raise RuntimeError(f"libEnsemble exited with flag {exit_flag}")

.agents/skills/generate-scripts-2/examples/preloaded_points.py

@@ -0,0 +1,54 @@
+"""Evaluate a fixed list of points with the standardized preloaded generator.
+
+Run with ``python preloaded_points.py`` from a fresh working directory. The
+run writes ``preloaded_points.npy`` and ``preloaded_points.pickle`` there.
+"""
+
+import numpy as np
+from gest_api.vocs import VOCS
+
+from libensemble import Ensemble
+from libensemble.gen_classes.preloaded import PreloadedSampleGenerator
+from libensemble.specs import GenSpecs, LibeSpecs, SimSpecs
+
+
+def simulate(inputs: dict, **kwargs) -> dict:
+    """Return the objective for one point supplied by the preloaded generator."""
+    x0 = inputs["x0"]
+    x1 = inputs["x1"]
+    return {"f": x0**2 + x1**2}
+
+
+if __name__ == "__main__":
+    # Every point must provide the variable fields the simulator reads. Constants,
+    # if present in VOCS, should also be included in each point or added explicitly.
+    points = [
+        {"x0": -1.0, "x1": 0.5},
+        {"x0": 0.0, "x1": 0.0},
+        {"x0": 1.0, "x1": 0.5},
+    ]
+    vocs = VOCS(
+        variables={"x0": [-3.0, 3.0], "x1": [-2.0, 2.0]},
+        objectives={"f": "MINIMIZE"},
+    )
+
+    generator = PreloadedSampleGenerator(points, vocs=vocs, batch_size=2)
+    ensemble = Ensemble(
+        sim_specs=SimSpecs(simulator=simulate, vocs=vocs),
+        gen_specs=GenSpecs(generator=generator, vocs=vocs),
+        libE_specs=LibeSpecs(comms="local", nworkers=2, safe_mode=True),
+    )
+
+    history, _, exit_flag = ensemble.run(sim_max=len(points))
+
+    if ensemble.is_manager:
+        completed = history[history["sim_ended"]]
+        finite = completed[np.isfinite(completed["f"])]
+        history_path = ensemble.save_output("preloaded_points", append_attrs=False)
+        print(f"Saved History to {history_path}")
+        print(f"Evaluated {len(completed)} of {len(points)} supplied points")
+        if len(finite):
+            best = finite[np.argmin(finite["f"])]
+            print(f"Lowest objective: x0={best['x0']:.4f}, x1={best['x1']:.4f}, f={best['f']:.6f}")
+        if exit_flag != 0:
+            raise RuntimeError(f"libEnsemble exited with flag {exit_flag}")

.agents/skills/generate-scripts-2/references/aposmm.md

@@ -0,0 +1,106 @@
+# Standardized APOSMM
+
+APOSMM runs multiple local optimizations concurrently to discover multiple minima. It is
+not the default choice for a single global optimum, constrained optimization, categorical
+variables, or multi-objective optimization.
+
+## SciPy template
+
+```python
+import numpy as np
+
+from gest_api.vocs import VOCS
+from libensemble import Ensemble
+from libensemble.specs import GenSpecs, LibeSpecs, SimSpecs
+
+
+def six_hump(inputs: dict, **kwargs) -> dict:
+    x0, x1 = inputs["x0"], inputs["x1"]
+    return {
+        "f": (4 - 2.1 * x0**2 + x0**4 / 3) * x0**2
+        + x0 * x1
+        + (-4 + 4 * x1**2) * x1**2
+    }
+
+
+if __name__ == "__main__":
+    import libensemble.gen_funcs
+
+    libensemble.gen_funcs.rc.aposmm_optimizers = "scipy"
+    from libensemble.gen_classes import APOSMM
+
+    vocs = VOCS(
+        variables={"x0": [-3.0, 3.0], "x1": [-2.0, 2.0]},
+        objectives={"f": "MINIMIZE"},
+    )
+    generator = APOSMM(
+        vocs,
+        max_active_runs=4,
+        initial_sample_size=40,
+        localopt_method="scipy_Nelder-Mead",
+        opt_return_codes=[0],
+        variables_mapping={"x": ["x0", "x1"], "f": ["f"]},
+        random_seed=1,
+    )
+    ensemble = Ensemble(
+        sim_specs=SimSpecs(simulator=six_hump, vocs=vocs),
+        gen_specs=GenSpecs(
+            generator=generator,
+            vocs=vocs,
+            initial_batch_size=40,
+            batch_size=4,
+        ),
+        libE_specs=LibeSpecs(comms="local", nworkers=4, safe_mode=True),
+    )
+    H, _, _ = ensemble.run(sim_max=500)
+
+    if ensemble.is_manager:
+        completed_minima = H[
+            H["sim_ended"] & H["local_min"] & np.isfinite(H["f"])
+        ]
+        ensemble.save_output("aposmm_results", append_attrs=False)
+        print(completed_minima[["x0", "x1", "f"]])
+```
+
+The standardized APOSMM class currently requires one compatibility setup through the
+legacy backend registry: set `libensemble.gen_funcs.rc.aposmm_optimizers` before importing
+`APOSMM`. Keep this isolated in the main block; the generated workflow still uses the
+standardized generator API. Use `"nlopt"` for an NLopt method and ensure the selected
+backend package is installed.
+
+## Required design choices
+
+- `max_active_runs`: concurrent local optimizer runs; size it to useful simulation
+  concurrency rather than blindly copying worker count.
+- `initial_sample_size`: evaluated points APOSMM waits for before local optimization.
+- `localopt_method`: must match the configured backend and objective information.
+- `variables_mapping`: map APOSMM's vector `x` to ordered VOCS variable names and scalar
+  `f` to the objective. Add mappings required by gradient/residual methods.
+- `initial_batch_size`: normally equal to `initial_sample_size` when APOSMM generates the
+  initial design itself.
+
+APOSMM can obtain initial data from its own sample, `sample_points`, warm-start History,
+or pre-ingested records. Do not feed arbitrary new sample points after local optimization
+has begun.
+
+## Constraints and constants
+
+Current standardized APOSMM ignores VOCS constraints and constants and warns about them.
+Do not present it as a constrained optimizer. Ask the user to select a compatible method
+or transform the problem only with explicit approval.
+
+## Local optimizer guidance
+
+- `scipy_Nelder-Mead`: derivative-free baseline; SciPy required.
+- SciPy gradient methods require the corresponding derivative output/mapping.
+- NLopt methods require `nlopt` and method-specific stopping tolerances/return codes.
+- PETSc/TAO, DFO-LS, IBCDFO, and external local optimizers require their own packages and
+  output contracts.
+
+Do not guess return codes, gradient fields, residual shapes, or tolerances. Confirm them
+for the chosen backend/version.
+
+## Results
+
+`local_min` marks minima identified by APOSMM. Filter by both `sim_ended` and `local_min`,
+then require finite objectives. Generated-but-unevaluated rows are not minima to report.