[Discussion/Proposal] Re-organizing Unit Tests

[allenanie]

Right now, the tests are organized under two folders: unit tests and LLM-dependent tests (llm_optimizers_tests).

Some unit tests are organized by functionality:

• test_batch_run.py
• test_copy.py
• test_export.py

Some unit tests are organized by modules:

• test_dataloader.py
• test_nodes.py

Solution proposal:

Create three folders: llm_optimizer_tests, module_unit_tests, functional_tests

@doxav, can you give some advice on this as well, given your engineering background?

[allenanie]

This is not urgent, but just documenting it here so that we don't lose track. The unit test folder is getting a bit messy...

[doxav]

Hello @allenanie ,

I look into tests and I think there are 2 different issues:

1. tests should be classified by intent/use cases and easy to classify by any contributor (not always easy).
   Separating tests between functionality and module is not easy, because a functionality could be part of a module.
   The separation is usually more between stand-alone/unit vs multi/integration.
   We could change to:
   . unit: stand-alone functions/classes; deterministic; fast
   . integration/dummy: multiple components, but no real network; dummy/mocked LLMs allowed
   . integration/real_llm: multiple components with live LLM calls (gated)
   . performance: perf/memory/speed benchmarks (gated)
   And inside each, when there are many functions tested in a module, we could make a sub-folder. We could already make sub-folders mapping opto sub-folders modules (optimizers, trainers, trace, utils) ?

2. some unit tests are integration (we wanted to start simple) and some even use real llm, and some tests in llm_optimizers_tests are not using real llm. We can re-organize this.

Therefore I propose to apply it like this below:

# create the new tree
mkdir -p tests/{unit,integration/{dummy,real_llm}}/{trace,optimizers,trainer,utils} tests/perf

# A) from tests/unit_tests
git mv tests/unit_tests/test_apply_op.py                 tests/unit/trace/test_apply_op.py
git mv tests/unit_tests/test_async.py                    tests/integration/dummy/trace/test_async.py
git mv tests/unit_tests/test_async_run.py                tests/integration/dummy/trainer/test_async_run.py
git mv tests/unit_tests/test_asyncio.py                  tests/integration/dummy/trace/test_asyncio.py
git mv tests/unit_tests/test_backward.py                 tests/unit/trace/test_backward.py
git mv tests/unit_tests/test_basic_containers.py         tests/unit/trace/test_basic_containers.py
git mv tests/unit_tests/test_basic_operators.py          tests/unit/trace/test_basic_operators.py
git mv tests/unit_tests/test_batch_run.py                tests/integration/dummy/trainer/test_batch_run.py
git mv tests/unit_tests/test_bool.py                     tests/unit/trace/test_bool.py
git mv tests/unit_tests/test_bundle.py                   tests/unit/trace/test_bundle.py
git mv tests/unit_tests/test_class_method.py             tests/unit/trace/test_class_method.py
git mv tests/unit_tests/test_containers.py               tests/unit/trace/test_containers.py
git mv tests/unit_tests/test_context_around_code.py      tests/unit/trace/test_context_around_code.py
git mv tests/unit_tests/test_copy.py                     tests/unit/trace/test_copy.py
git mv tests/unit_tests/test_dataloader.py               tests/unit/trainer/test_dataloader.py
git mv tests/unit_tests/test_dependencies.py             tests/unit/trace/test_dependencies.py
git mv tests/unit_tests/test_error_handling.py           tests/integration/dummy/optimizers/test_error_handling.py
git mv tests/unit_tests/test_llm.py                      tests/integration/real_llm/trace/test_llm_smoke.py
git mv tests/unit_tests/test_modules.py                  tests/unit/trace/test_modules.py
git mv tests/unit_tests/test_multi_decorators.py         tests/unit/trace/test_multi_decorators.py
git mv tests/unit_tests/test_nodes.py                    tests/unit/trace/test_nodes.py
git mv tests/unit_tests/test_not_covered_usage_cases.py  tests/integration/dummy/trace/test_usage_cases.py
git mv tests/unit_tests/test_optimizer_xml_parsing.py    tests/unit/optimizers/test_optimizer_xml_parsing.py
git mv tests/unit_tests/test_optoprime_update.py         tests/integration/dummy/optimizers/test_optoprime_update.py
git mv tests/unit_tests/test_priority_search.py          tests/integration/dummy/optimizers/test_priority_search.py
git mv tests/unit_tests/test_projection.py               tests/unit/utils/test_projection.py
git mv tests/unit_tests/test_python_funcs.py             tests/unit/trace/test_python_funcs.py
git mv tests/unit_tests/test_randomness.py               tests/unit/trace/test_randomness.py
git mv tests/unit_tests/test_re_parsing.py               tests/unit/trace/test_re_parsing.py
git mv tests/unit_tests/test_sampler.py                  tests/integration/dummy/trainer/test_sampler.py
git mv tests/unit_tests/test_saving_loading.py           tests/integration/dummy/optimizers/test_saving_loading.py
git mv tests/unit_tests/test_to_data.py                  tests/unit/trace/test_to_data.py
git rm tests/unit_tests/run.py

# B) from tests/llm_optimizers_tests
git mv tests/llm_optimizers_tests/test_guides.py                tests/unit/optimizers/test_guides.py
git mv tests/llm_optimizers_tests/test_opro_v2.py               tests/integration/dummy/optimizers/test_opro_v2.py
git mv tests/llm_optimizers_tests/test_optimizer.py             tests/integration/real_llm/optimizers/test_optimizer.py
git mv tests/llm_optimizers_tests/test_optimizer_optoprimemulti.py tests/integration/dummy/optimizers/test_optoprimemulti.py
git mv tests/llm_optimizers_tests/test_optoprime_v2.py          tests/integration/dummy/optimizers/test_optoprime_v2.py

# C) perf
git mv tests/test_memory_leak.py tests/perf/test_memory_leak.py
git mv tests/llm_optimizers_tests/test_bbh_subset.py tests/perf/test_bbh_subset.py

Later we could put in perf some other trainer related perf bench.

We should also change the CI file: .github/workflows/ci.yml

*** File: .github/workflows/ci.yml
@@
-on:
-  push:
-    branches: [ main ]
-  pull_request:
-    branches: [ main ]
+on:
+  push:
+    branches: [ main, experimental ]
+  pull_request:
+    branches: [ main, experimental ]
@@
-    # 8) Run all Trace unit tests
-    - name: Run unit tests
-      run: pytest tests/unit_tests/
+    # 8) Run all Trace unit tests (migration: run legacy + new locations)
+    - name: Run unit tests (legacy + new)
+      # Run both old and new paths so CI is migration-safe while you move files.
+      run: pytest tests/unit_tests tests/unit tests/integration/dummy -q
@@
-      - name: Run optimizer test (legacy, silenced)
-        run: pytest tests/llm_optimizers_tests/test_optimizer.py || true
+      - name: Run tests (migration: old + new)
+      run: pytest tests/llm_optimizers_tests/test_optimizer.py tests/integration/real_llm/optimizers -q || true
*** End Patch

[allenanie]

We should discuss whether to create a PR and merge this for the next developer meeting :D