Example for using RAPIDS with Dagster-Slurm - #718
Open
HPicatto wants to merge 28 commits into
Open
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Contributor
|
We're gonna trigger CI to run additional tests, and test the whole thing on a SLURM cluster. Like we discussed before, It would be ideal to replace |
* Update docs to use NVIDIA Sphinx theme * fix: artifacting in rendered page * fix: run precommit * rebase fix conflcits * small edit html title * clean up comment in version switcher * clean up comment in conf.py --------- Co-authored-by: Jaya Venkatesh <jjayabaskar@nvidia.com> Co-authored-by: ncclementi <natyclementi@gmail.com>
|
… example Adds a workflow example showing RAPIDS cuML (UMAP + HDBSCAN) running on a Slurm HPC cluster orchestrated with dagster-slurm: a partitioned CPU fan-out of gensim LDA jobs followed by GPU reduction/clustering stages in a separate packed pixi environment, with a CPU fallback path for GPU-less development. Includes UI screenshots from a real cluster run and a refined variant with metaxy sample-level incremental tracking. Refs rapidsai#715 Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
- add a quickstart and an explicit 'by hand vs. here' contrast up front - call out the local iteration loop (identical code, laptop to cluster) - state which assets run in local mode plus expected size/runtime - mention run-scoped Slurm allocations for the queue-wait case - mention multi-cluster targeting via SLURM_EDGE_NODE_* configuration - add an 'HPC in the middle, Dagster end to end' section - soften the topic_map claim (plot not among the screenshots) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
docs: address tutorial review feedback (quickstart, personas, E2E story) Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
- Quickstart block right after the intro; 'what you would normally do' contrast paragraph for first-time HPC users - Explicit note that local mode materializes 3 of 6 assets - Download-size / wall-clock expectations for the docker chain - Queue-wait answer (experimental session/het-job modes) and multi-cluster deployment note for experienced Slurm users - Note that all-sbatch is a choice; ingest/publish can run off-Slurm in the same graph (and must, on sites without compute-node internet) - Iteration-loop paragraph in the conclusion - Add the topic_map artifact from the cluster run as the closing image - Stable cell ids (nbformat 5) Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
From a full end-to-end backfill on the real cluster (19m28s): the topic_map run view with cluster-count/path metadata, the topic_map preview rendering the scatter inline in the Dagster UI (closing the review gap about the promised 'streams a labeled meta-topic map' claim), and the all-green backfill overview with observed wall-clock numbers in the surrounding text. Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
HPicatto
force-pushed
the
feat/dagster-slurm-topic-modeling-example
branch
from
July 19, 2026 16:24
6661597 to
7ee9795
Compare
* change conf to PEP440 standards Signed-off-by: Jaya Venkatesh <jjayabaskar@nvidia.com> * Centralize CUDA/Python config in conf.py; default docs to CUDA 13 * Use standard conda packages for SageMaker (issue rapidsai#520 resolved) * Pin AzureML to CUDA 12 with inline conda spec and Docker Hub image * Use centralized CUDA tokens for Modal base image and wheel * Template Coiled CUDA/container references; shield YAML jinja from prettier * Refresh Vertex AI note to reflect CUDA 13 environment * Inline Modal base image and drop cuda_docker_version * Templatize custom-docker wheel references * Move Databricks to CUDA 13 and replace preinstalled cupy * added warning on AzureML --------- Signed-off-by: Jaya Venkatesh <jjayabaskar@nvidia.com>
* Fix nightly pip version specifier to allow prereleases * Use pip version reference for Modal wheel install * change nightly versioning * Drop explicit cupy handling from Databricks init script * Update Databricks runtime guidance to 18 LTS ML * Reflect new Databricks Machine Learning runtime toggle * update runtime wording * Pass RAPIDS pip index to Modal wheel install
* X-ORG-722: Publish API docs to docs.nvidia.com * Lint
* update links to guide better the user to console * add note with workaround due to broken vmi tile * Apply suggestion --------- Co-authored-by: Jaya Venkatesh <jjayabaskar@nvidia.com>
update readme, better layout description
add kaggle page in platforms
* add Databricks Serverless * added intro links * remove dask-cuda from pip installation
Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
…icatto:HPicatto/deployment into feat/dagster-slurm-topic-modeling-example Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
Signed-off-by: Hernan Picatto <h.picatto@gmail.com>
Author
|
Hi @jayavenkatesh19 and @ncclementi I think that the draft is ready could you please review, thanks! |
Contributor
|
@HPicatto we are finishing with our release responsibilities and work for 26.08. We have this PR assigned for reviewed and prioritized for our next release. We will have time to start looking into this starting next week. We haven't forgot we just had to wrapped what was planned for 26.08. In the meantime, we need all commits to be signed so tehy showed as verify see contributing guidelines https://github.com/rapidsai/deployment/blob/main/CONTRIBUTING.md#developer-certificate-of-origin |
HPicatto
force-pushed
the
feat/dagster-slurm-topic-modeling-example
branch
2 times, most recently
from
August 10, 2026 09:25
e7028b4 to
927612a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… example
Adds a workflow example showing RAPIDS cuML (UMAP + HDBSCAN) running on a Slurm HPC cluster orchestrated with dagster-slurm: a partitioned CPU fan-out of gensim LDA jobs followed by GPU reduction/clustering stages in a separate packed pixi environment, with a CPU fallback path for GPU-less development. Includes UI screenshots from a real cluster run and a refined variant with metaxy sample-level incremental tracking.
Refs #715