Fixissue/#415 - #416
Open
aaadelmann wants to merge 21 commits into
Open
Conversation
Member
Author
|
closes #415 |
srikrrish
requested changes
Oct 28, 2025
srikrrish
left a comment
Member
There was a problem hiding this comment.
Looks good to me. But currently there is one conflict in the CMakeLists which must be resolved and CI needs to be invoked before merging
aaadelmann
marked this pull request as draft
July 28, 2026 14:32
Resolve the ParticleAttrib scatter conflict by keeping master's captured local view form while preserving the mixed attribute/field type behavior covered by the branch tests. Preserve the cosmology mc-4 initializer after master moved demos under demos/cosmology by moving the initializer sources there and wiring them into the StructureFormation target.
Split ParticleAttrib::scatter into a host-side hash/no-hash dispatcher and a templated scatterImpl so the non-hashed kernel does not instantiate hash-array indexing. Route the mapped-index selection through a small KOKKOS_INLINE_FUNCTION helper. This avoids the CUDA extended-lambda restriction where a variable is first captured inside an if constexpr context, while still keeping the hash access compiled out for the non-hashed instantiation. Also normalize the hash extent to the range-policy end type before checking the policy range, removing the signed/unsigned comparison warning.
Move the hash-aware ParticleAttrib scatter kernel body out of the ParticleAttrib private member scope and into a namespace-scope detail helper. CUDA extended host-device lambdas cannot be enclosed by private or protected class member functions. The previous scatterImpl helper fixed the hash-array constexpr dispatch but still placed the KOKKOS_LAMBDA inside a private member, which NVCC rejects for CUDA builds. Keep ParticleAttrib::scatter as the public host-side dispatcher. It validates the optional hash view and selects either the hashed or non-hashed compile-time implementation: particleAttribScatterImpl<true> particleAttribScatterImpl<false> The mapped-index selection remains in a KOKKOS_INLINE_FUNCTION helper so the hash access is compiled only for the hashed instantiation, avoiding CUDA's first-capture-in-constexpr-if extended-lambda restriction. Tested locally with a SERIAL debug build: cmake --build build-fixissue-415-merge -j 8 ctest --test-dir build-fixissue-415-merge --output-on-failure -R '^(TestScatter|GatherScatterTest)$' Both GatherScatterTest and TestScatter passed.
Add explicit GatherScatterTest coverage for scattering a lower-precision particle attribute into a higher-precision field: ParticleAttrib<float> -> Field<double> This exercises the scatter API change that allows the particle attribute value type to differ from the field value type. The new coverage includes both the plain scatter path and the custom hash-array scatter path, so the host-side hash/no-hash dispatch and the CUDA-safe scatter implementation are both covered for mixed value types. Extend the test bunch with an additional float charge attribute and add a generic hash-reduction helper so expected charges can be accumulated in double while reading either float or double particle attributes. Add a scatterConservationTolerance helper that chooses the tolerance from the weaker precision of the scatter attribute and field value type. This avoids testing float -> double scatter with an unrealistically strict double tolerance while still scaling the allowed error with the conserved charge norm. Tested locally with: cmake --build build-fixissue-415-merge --target GatherScatterTest -j 8 ctest --test-dir build-fixissue-415-merge --output-on-failure -R '^GatherScatterTest$' mpirun -np 2 build-fixissue-415-merge/unit_tests/Particle/GatherScatterTest --gtest_filter='*MixedValueType*' mpirun -np 4 build-fixissue-415-merge/unit_tests/Particle/GatherScatterTest --gtest_filter='*MixedValueType*' All tests passed.
aaadelmann
marked this pull request as ready for review
August 6, 2026 17:36
Member
Author
|
You already approved in the past, here I needed to fix some CUDA related issues and added an other test |
Member
Author
|
cscs-ci run cscs-ci-gh200, cscs-ci-mi300, cscs-ci-openmp |
srikrrish
approved these changes
Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The actual Kokkos kernel body was moved out of the private
ParticleAttribmember scope into a namespace-scope helper:ippl::detail::particleAttribScatterImpl(...)This avoids NVCC errors for extended
__host__ __device__lambdas whose enclosing parent function has private/protected class access.The mapped-index selection is also handled through a small
KOKKOS_INLINE_FUNCTIONhelper:This avoids CUDA’s restriction around first-capturing a variable inside an
if constexprcontext.Mixed value type scatter
The scatter implementation now supports different particle attribute and field value types, for example:
The scattered particle value is passed to the field scatter operation using the particle attribute’s value type, while the field accumulation happens in the field’s value type.
Regression tests
unit_tests/Particle/GatherScatterTest.cppnow includes explicit coverage for:ParticleAttrib<float> -> Field<double>plain scatterParticleAttrib<float> -> Field<double>hashed scatterThe test uses a conservation tolerance based on the weaker precision of the scatter attribute and field value type. This avoids applying double-precision tolerances to a float-valued particle source while still scaling the tolerance with the conserved charge norm.
Motivation
The original hashed scatter refactor triggered CUDA compilation failures on GH200/NVCC:
and then:
These were CUDA-specific restrictions around extended lambdas. The final implementation keeps the same host-side dispatch semantics, but places the kernel-bearing function in namespace scope and avoids first-capture inside
if constexpr.Validation
Tested locally with a serial debug build:
All tests passed.
The CUDA build was validated externally through the project CI/CD matrix after the scatter implementation changes.