Update directory structure and rename from Cppyy->CppJIT by aaronj0 · Pull Request #14 · compiler-research/cppjit

aaronj0 · 2026-08-06T16:37:20Z

From the agent:

This PR carries the pre-squash structural work: it turns the fork-derived tree into cppjit proper, layout updates, renames, no legacy so the churn is buried into a single commit before we begin new developments on top.

Layout

src/CPyCppyy/{include/CPyCppyy,src}/ flatten into a single flat src/cpyrt/; the backend wrapper is src/interop/ and clingwrapper.cxx is renamed interop_wrapper.cxx.
The C++ sides of the tests — the 15 dictionary .h/.cxx pairs — move under test/cpp/ and build into *Dict.so next to their sources; pytest files and helpers stay at the test/ root.
The valgrind suppressions live under .github/valgrind/ (they are CI assets, read by the ci-workflows composite); etc/ is gone.

Rename

Everything is spelled cppjit, by one ordered, case-respecting token map over file contents and file names:

Python: import cppjit is the frontend; cppjit_backend and the merged libcppjit extension stay top-level, upstream's own package shape. tp_names, reprs and the cppjit.gbl/cppjit.gbl.std module strings are consistent on the C++ and Python sides.
C++: the namespace structure is cppjit::{cpyrt, interop} — cpyrt replaces CPyCppyy (the CPython runtime layer), interop replaces the backend Cppyy:: API (reflection/JIT over CppInterOp). Implementation files read the short cpyrt::/interop:: under an injected using namespace cppjit;; headers, namespace
declarations, and identifiers emitted into JIT-compiled code keep the full spelling, since the interpreter resolves only the real namespace.
Tests carry the same map and import cppjit directly: the fixture classes are CppjitTestData/CppjitTestPod, and the conftest.py cppyy-alias shim is gone (conftest is back to upstream's single line).
Breaking for fork-era users: the pythonization hooks are cppjit_pythonize/cppjit_explicit_pythonize, env vars are CPPJIT_*, and the exported symbols follow the new namespaces — JIT-side namespace Cppyy prototypes must be respelled.
Deliberately kept: CppyyLegacy (ROOT's real legacy namespace) and references to cppyy-the-upstream-project (docs and issue URLs).

Dead code and metadata

Everything without a reachable path in the merged package is dropped:

PyPy loader (cppjit requires CPython and now says so)
PyInstaller hook and cling-config CLI (their entry points were never registered)
backend loader.py (importing libcppjit is loading the backend)
win32 dictionary-build chain
genreflex .xml selection files and Makefile rules (a dictionary build is a plain $(CXX) -shared)
cling-PCH bootstrap.

The package version is single-sourced from _version.py (previously reported cppyy's 3.0.0 from the frontend and ROOT's 6.28.0 from the backend) and requires-python is now 3.12+, matching what CI tests.

Validation

pip source install (LLVM 22, fresh venv) + the full upstream test suite: 539 passed / 40 skipped / 19 xfailed / 1 xpassed, 0 failed — identical to the pre-rename reference run. CI needs the matching ci-workflows composite change (smoke import + suppression paths) on main.

aaronj0 · 2026-08-06T16:37:58Z

As discussed/planned, following this merge we can squash the history and begin active development.

vgvassilev · 2026-08-06T17:21:13Z



-namespace CPyCppyy {
+namespace PyCppJIT {


Is that the middle layer? I am wondering if we have better options for a namespace name to signal that.

aaronj0 · 2026-08-06T22:36:13Z

@vgvassilev I'm continuing our discussions here, so I can propose the design we seem to be converging upon:

cppjit namespace design: cppjit::{cpyrt, interop}

We use a single C++ namespace tree,
cppjit::, with two members today: cppjit::cpyrt is the replacement
for CPyCppyy and holds the CPython runtime
layer, and cppjit::interop is reserved for what is currently
clingwrapper and all interactions with CppInterOp. clingwrapper is renamed
to the interop wrapper (interop_wrapper.cxx)

Namespace	What it is	Sources
`cppjit::cpyrt`	the CPython runtime: proxies, converters, executors, pythonizations — everything that includes `Python.h`	`src/cpyrt/`
`cppjit::interop`	reflection and JIT over CppInterOp — what the interop wrapper implements	`src/interop/`

What one writes and reads as a cppjit developer

The two layers are siblings under cppjit::, so code in cpyrt reaches
the reflection API as plain interop:: without full qualification:

// src/cpyrt/Converters.cxx
namespace cppjit::cpyrt {

bool VectorConverter::SetArg(PyObject* pyobject, Parameter& para, CallContext* ctxt)
{
    // sibling namespace, no full qualification needed
    interop::TCppScope_t scope = interop::GetScope("std::vector<double>");
    ...
}

} // namespace cppjit::cpyrt

Implementation files that are not namespace-wrapped carry
using namespace cppjit; after the main bindings include, so definitions
and references read cpyrt::X and interop::X there as well. Headers
keep the full qualification, since they are installed or included into
arbitrary translation units. The same holds for anything we emit into
JIT-compiled code: the interpreter resolves only the real namespace, so
generated code spells cppjit::cpyrt:: in full.

Someone extending cppjit from outside includes the installed headers
(they keep the layer prefix, cppjit_backend/include/cpyrt/) and writes:

#include "cpyrt/API.h"

class MyConverter : public cppjit::cpyrt::Converter {
    bool SetArg(PyObject*, cppjit::cpyrt::Parameter&,
                cppjit::cpyrt::CallContext* = nullptr) override;
};
cppjit::cpyrt::RegisterConverter("MyType", ...);

The cpyrt/ include prefix is an interim spelling. When we converge
on a public API surface we would prefer #include "cppjit/API.h", or a
single cppjit.h, and the layer prefix then stays internal.

From Python and from JIT-side C++ the same tree shows up as nested
namespace proxies:

import cppjit

cppjit.cppdef("int f(int x) { return x + 1; }")
cppjit.gbl.f(41)

# the reflection API is a normal namespace under gbl
scope = cppjit.gbl.cppjit.interop.GetScope("std::vector<int>")

The naming conventions in one place:

Surface	Convention	Example
C++ namespaces	`cppjit::<layer>`	`cppjit::cpyrt`, `cppjit::interop`
Layer macros/guards	`CPYRT_`, `CPPJIT_`	`CPYRT_API_H`, `CPPJIT_IMPORT`
Python package	`cppjit` (+ top-level `cppjit_backend`, `libcppjit`)	`import cppjit`
tp_names / reprs	`cppjit.<Type>`	`cppjit.CPPInstance`
Module strings	`cppjit.gbl`, `cppjit.gbl.std`	pickling, `__module__`
Hook protocol	`__cppjit_*`	`__cppjit_pythonize__`
Env vars	`CPPJIT_*`	`CPPJIT_API_PATH`

How python-fortran interop would fit

The split between the two namespaces is also the extension seam.
cppjit::interop has no Python.h dependency (that separation is what
lets the merged library exist in the first place), and cppjit::cpyrt
does not care what source language an entity came from — converters and
executors are keyed on reflected types. So adding fortran support is
additive on both sides:

On the compiler side, interop gains a Fortran provider (Flang-based),
either as new entry points in cppjit::interop or as a sibling
implementing the same reflection contract. The existing C++ paths and
CppInterOp stay as they are.
On the runtime side, fortran-backed entities surface to Python through
the cpyrt machinery that already exists: new converter and executor
registrations via the same public API third parties use, not changes
to existing ones. If fortran needs runtime state of its own it gets a
sibling namespace and directory:

src/
├── cpyrt/       cppjit::cpyrt       (CPython runtime)
├── interop/     cppjit::interop     (reflection/JIT)
└── fortran_rt/  cppjit::fortran_rt  (hypothetical fortran runtime — additive)

On the Python side a cppjit.fortran submodule exposes the fortran
namespace the way gbl exposes C++, and the package absorbs it
without renames.

The invariant that makes this "unintrusive": new languages add siblings
(namespaces, directories, registrations) and don't rename or restructure
existing members. That is what motivated the flat src/ layout plus the
cppjit:: umbrella, and no api//include/ tree is created until we
converge on an agreed-upon public API surface.

cc @guitargeek

guitargeek · 2026-08-07T08:40:18Z

Sounds good! cppjit::runtime would read less clunky to me compared to cppjit::cpy_rt, and it's obvious that this is a CPython extension anyway, since there is no pypy version anymore.

Anyway I have no strong preference at all because there is no public API, but if there would be one at some point, I'd definitely prefer #include "cppjit/API.h" over #include "cpy_rt/API.h". Or even cppjit.h. I mean the name API and the fact that it's a public header file is also kind of redundant.

vgvassilev · 2026-08-07T08:52:42Z

# the reflection API is a normal namespace under gbl
scope = cppjit.gbl.cppjit.interop.GetScope("std::vector<int>")

This is a bit clunky but I guess we can solve that later not now...

vgvassilev · 2026-08-07T08:54:40Z

Sounds good! cppjit::runtime would read less clunky to me compared to cppjit::cpy_rt, and it's obvious that this is a CPython extension anyway, since there is no pypy version anymore.

Anyway I have no strong preference at all because there is no public API, but if there would be one at some point, I'd definitely prefer #include "cppjit/API.h" over #include "cpy_rt/API.h". Or even cppjit.h. I mean the name API and the fact that it's a public header file is also kind of redundant.

Structurally, interop and cpy_rt are both performed at runtime. So having a generic name runtime can be confusing if we want to distinguish where the api lives.

guitargeek · 2026-08-07T08:55:49Z

# the reflection API is a normal namespace under gbl
scope = cppjit.gbl.cppjit.interop.GetScope("std::vector<int>")
This is a bit clunky but I guess we can solve that later not now...

Not an actual problem for now fortunately, because we won't expose the public C++ API

aaronj0 · 2026-08-07T09:00:06Z

Structurally, interop and cpy_rt are both performed at runtime. So having a generic name runtime can be confusing if we want to distinguish where the api lives.

sI would prefer cpyruntime over cpy_rt as that is more readable for the same syllables

vgvassilev · 2026-08-07T09:10:34Z

Structurally, interop and cpy_rt are both performed at runtime. So having a generic name runtime can be confusing if we want to distinguish where the api lives.

sI would prefer cpyruntime over cpy_rt as that is more readable for the same syllables

If we are discussing folder name then that's fine, for a namespace I think it is a bit long. In that sense (assuming we follow the 80 col llvm rule) people will probably do using namespace cpyruntime which will hide again which belongs where when reading it.

EDIT: cpyrt is fine for me.

aaronj0 · 2026-08-07T09:16:45Z

EDIT: cpyrt is fine for me.

Sounds good. @guitargeek?

guitargeek · 2026-08-07T09:18:13Z

Sure!

github-actions · 2026-08-07T09:29:19Z

Test Results

Configuration	Result
macos-26-intel-llvm21-py3.14-cxx20	`= 433 passed, 52 skipped, 82 xfailed, 32 xpassed, 100 warnings in 61.25s (0:01:01) =`
macos-26-llvm21-py3.14-cxx20	`==== 419 passed, 60 skipped, 90 xfailed, 30 xpassed, 100 warnings in 32.88s ====`
ubuntu-24.04-llvm20-py3.14-cxx20-cling	`no summary (build or setup failed)`
ubuntu-24.04-llvm21-py3.14-cxx20-vg	`=========== 531 passed, 48 skipped, 18 xfailed, 2 xpassed in 43.47s ============`
ubuntu-24.04-llvm22-py3.14-cxx20	`=========== 531 passed, 48 skipped, 19 xfailed, 1 xpassed in 29.29s ============`

vgvassilev · 2026-08-07T09:38:59Z

Test Results

Configuration Result
macos-26-intel-llvm21-py3.14-cxx20 no summary (build or setup failed)
macos-26-llvm21-py3.14-cxx20 no summary (build or setup failed)
ubuntu-24.04-llvm20-py3.14-cxx20-cling no summary (build or setup failed)
ubuntu-24.04-llvm21-py3.14-cxx20-vg no summary (build or setup failed)
ubuntu-24.04-llvm22-py3.14-cxx20 no summary (build or setup failed)

For maybe a future improvement, we should have this either to the bottom of the PR description which has the last run and in <details> tag all the previous runs ordered; or as the first comment including the relevant commit hash it was built against. This way for multirun setups we won't have to scroll up and down and click on "load" to see the full integration history of a given change.

aaronj0 · 2026-08-07T09:43:24Z

For maybe a future improvement, we should have this either to the bottom of the PR description which has the last run and in <details> tag all the previous runs ordered; or as the first comment including the relevant commit hash it was built against. This way for multirun setups we won't have to scroll up and down and click on "load" to see the full integration history of a given change.

This is currently posts the last run results, ensuring it is always at the end of the PR (so it follows a force push for e.g), and the intermediate runs are discarded: at any given point in time there is only one "Test Results" displayed by github actions. What do you mean by multirun setups

aaronj0 · 2026-08-07T10:44:50Z

Pending work-items for this PR:

improve test directory structure
drop dead files
clang-format after approval
switch to using namespaces in source files

This should in the end give us a relatively "minimal" starting point for development after squashing the history. Some post squash items have been recorded in #15

[build] Flatten the CPyCppyy include and src trees into src/CPyCppyy

a2cec93

aaronj0 requested review from guitargeek and vgvassilev August 6, 2026 16:37

vgvassilev reviewed Aug 6, 2026

View reviewed changes

aaronj0 force-pushed the pre-squash-layout branch from fe26006 to 9702f93 Compare August 7, 2026 09:25

aaronj0 added 4 commits August 10, 2026 20:17

Rename the cppyy namespaces and packages to cppjit

420b542

Drop dead code and genreflex leftovers inherited from the forks

a4c94cb

[test] Move the C++ dictionary sources under test/cpp

17119f3

[ci] Move the valgrind suppressions under .github/valgrind

c114dee

aaronj0 force-pushed the pre-squash-layout branch from 9702f93 to c114dee Compare August 10, 2026 18:42

aaronj0 mentioned this pull request Aug 10, 2026

[cppjit] Handle package migration and fork-rewrite compiler-research/ci-workflows#94

Merged

aaronj0 requested a review from vgvassilev August 10, 2026 19:19

Uh oh!

Conversation

aaronj0 commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Layout

Rename

Dead code and metadata

Validation

Uh oh!

aaronj0 commented Aug 6, 2026

Uh oh!

vgvassilev Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

aaronj0 commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cppjit namespace design: cppjit::{cpyrt, interop}

What one writes and reads as a cppjit developer

How python-fortran interop would fit

Uh oh!

guitargeek commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vgvassilev commented Aug 7, 2026

Uh oh!

vgvassilev commented Aug 7, 2026

Uh oh!

guitargeek commented Aug 7, 2026

Uh oh!

aaronj0 commented Aug 7, 2026

Uh oh!

vgvassilev commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaronj0 commented Aug 7, 2026

Uh oh!

guitargeek commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

vgvassilev commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

aaronj0 commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaronj0 commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aaronj0 commented Aug 6, 2026 •

edited

Loading

aaronj0 commented Aug 6, 2026 •

edited

Loading

guitargeek commented Aug 7, 2026 •

edited

Loading

vgvassilev commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

vgvassilev commented Aug 7, 2026 •

edited

Loading

aaronj0 commented Aug 7, 2026 •

edited

Loading

aaronj0 commented Aug 7, 2026 •

edited

Loading