condorctl

Drive a remote HTCondor pool from your laptop over SSH: check CPU/GPU availability across the pool, then submit, monitor, and retrieve jobs -- all from the command line, with no HTCondor install or Python bindings needed locally.

condorctl talks to any pool it can reach over SSH. Pool-specific conventions (a custom submit attribute, an enforced CPU architecture, a hostname domain for --platform shortcuts) live in config, not in the code, so the same tool works against a plain pool or one with local policy.

Reference deployment: it was built and hardened against NMRbox, a shared NMR-software pool of ~60 machines, and ships a ready-to-use profiles/nmrbox.yaml. If NMRbox is your pool, jump to Using it with NMRbox.

Not affiliated with NMRbox. condorctl is an independent, unofficial client that talks to a pool using standard HTCondor tools over your own authorized SSH access. It is not endorsed by, sponsored by, or connected to NMRbox or its maintainers; "NMRbox" is named only to describe compatibility.

Setup

1. Install Python and the package

Requires Python 3.10+.

python -m venv .venv
.venv\Scripts\pip install -e .

2. SSH key

condorctl authenticates with an SSH key only -- it never reads or stores a password. If you don't already have a key registered on the pool's submit host:

ssh-keygen -t ed25519

Register the printed public key with the pool (append it to ~/.ssh/authorized_keys on the submit host, or use whatever key-management the pool provides).

3. Configure

copy config.example.yaml config.yaml

Edit config.yaml with your submit host, username, and key path. The file is gitignored -- it never gets committed. See config.example.yaml for every field, including the optional pool-policy settings.

4. Verify

condorctl test-connection

Prints your remote user, host, HTCondor version, and whether the htcondor Python bindings are importable there (condorctl works either way -- it drives HTCondor via the CLI tools, parsed directly, so the bindings aren't required).

Commands

All commands accept --config PATH to point at a config file other than config.yaml in the project root (or set $CONDORCTL_CONFIG).

check -- pool availability

condorctl check --gpu

Reports idle CPU/GPU capacity across every machine in the pool (one condor_status query, no per-host SSH) and recommends the best idle GPU target, ranked by a rough capability tier (H100 > A100 > L40S > A10 > V100 > T4) and then by how many GPUs are actually free right now. Override the tier list per-pool with the gpu_tiers config key.

If your pool enforces a CPU architecture (required_arch in config), machines that can never match a default job are listed but marked unreachable and never recommended.

submit -- submit a job

condorctl submit script.sh --gpus 1 --memory 8GB --input data.txt -- --some-arg value

Runs script.sh (transferred if it's a local file, otherwise treated as already present on the remote) with the given input files, and prints the returned HTCondor cluster ID. Everything after -- is passed to the command unchanged -- it's never parsed as a condorctl flag, no matter what it looks like.

Any extra_submit_attributes from your config are attached to every job (see the NMRbox profile, which sets +Production = True).

Flags: --input FILE (repeatable), --output NAME (repeatable; if omitted, any new/modified file in the job's sandbox comes back), --cpus, --memory (e.g. 4GB or 4096MB), --gpus, --platform HOST (target a specific machine), --no-auto-target, --dry-run, --job-name.

If --gpus is set without an explicit --platform, the balancer auto-picks the best currently-idle GPU machine (see check above) and prints which one it chose. Pass --no-auto-target to disable this and let HTCondor's own matchmaker decide instead. Auto-targeting is skipped in --dry-run mode since it requires a live pool query.

With a host_domain set in config, --platform grace expands to grace.<host_domain>; otherwise --platform is used verbatim.

submit-existing -- submit a pre-written .sub file

condorctl submit-existing /path/on/remote/job.sub

For jobs that already have their own correct submit description on the remote host -- e.g. a project's own HTCondor tooling -- submits it unmodified rather than rebuilding it. No files are transferred and no config policy is injected; the job runs exactly as the .sub file specifies.

status -- poll until done

condorctl status 1995469 --pull-retries 40 --pull-interval 15

Polls a job's own HTCondor event log until it terminates, fails, or is held (or until retries run out). Reads the per-job log file rather than querying condor_q/condor_history pool-wide, which can be slow to search by cluster ID once a job has left the queue in a heavily-shared pool.

Exit codes: 0 succeeded, 1 terminated with a non-zero exit code or was held, 2 gave up waiting (job still running/idle -- run again with a higher --pull-retries or --pull-interval).

pull -- retrieve outputs

condorctl pull 1995469 --local-out ./results

Copies a completed job's files back. By default, pulls everything in the remote job directory except the files you originally uploaded (tracked automatically from submit), so you get logs and genuine outputs without redundant copies of your own inputs. Use --output NAME (repeatable) to pull only specific files.

run -- submit, wait, and pull in one step

condorctl run script.sh --gpus 1 --memory 8GB -- --some-arg value

Combines submit + status + pull over a single SSH connection. Accepts the union of their flags. --local-out defaults to ./runs/<job-name>-<cluster-id>/ if not given. --no-pull submits and waits without pulling outputs back automatically.

Configuration

A config file describes one pool: the connection (required) and its submission policy (optional). The optional fields all default to a plain, unrestricted HTCondor pool:

Key	Meaning
host, username, key_file	SSH connection to the submit host (required)
port, remote_workdir	SSH port; where remote job dirs are created
host_domain	Domain appended to bare --platform NAME values
required_arch	Architecture the pool enforces on every job (e.g. X86_64)
extra_submit_attributes	Custom ClassAd attributes added to every submit file
gpu_tiers	Override the GPU capability ranking used by check --gpu

See config.example.yaml for the fully-commented template.

Using it with NMRbox

NMRbox is the pool this tool was built against. Its conventions are captured in profiles/nmrbox.yaml:

• host_domain: nmrbox.org -- so --platform grace targets grace.nmrbox.org.
• required_arch: X86_64 -- NMRbox ANDs Arch == X86_64 into every job, so grace.nmrbox.org (its one aarch64 GH200 box) is listed but never recommended as a target.
• extra_submit_attributes: {Production: true} -- NMRbox only serves its full NMR software stack to jobs that declare +Production = True.

To use it, copy the profile and set your username:

cp profiles/nmrbox.yaml config.yaml
# edit `username`
condorctl test-connection

Register your SSH key with NMRbox via the user dashboard's SSH keys section (or by appending it to ~/.ssh/authorized_keys on any NMRbox VM); dashboard changes can take a few minutes to propagate across the pool. Any production VM works as host -- a single condor_status query from any one reports the whole pool.

How jobs are tracked locally

Each submit/run writes a small JSON record to jobs/<cluster_id>.json (gitignored) with the job's remote directory and which files were uploaded. status and pull use this to find a job by cluster ID alone -- no need to remember or re-type the remote path.

Project layout

condorctl/
  config.py         config.yaml loading and validation
  connection.py     SSH session (paramiko): run commands, transfer files
  availability.py   pool-wide condor_status query, GPU ranking, balancer
  submission.py     generic job template -> HTCondor submit description
  monitor.py        parses a job's event log into a status
  pull.py           copies a completed job's outputs back
  jobs.py           local JSON registry, keyed by cluster ID
  cli.py            argparse entry point tying it all together
profiles/
  nmrbox.yaml       ready-made config for the NMRbox pool
tests/              pytest suite (pure-logic tests, no live connection needed)

Testing

.venv\Scripts\pip install -e ".[test]"
.venv\Scripts\python -m pytest

The test suite covers the parsers and pure decision logic (submit description generation, job log parsing, pull file selection, config validation, GPU ranking) without needing a live connection. It does not cover the SSH/SFTP code paths (connection.py, the actual condor_submit/file-transfer calls) -- those were verified by hand against a live pool during development. If you change that code, retest against a real pool directly.

Known gaps

• No CI configured -- run pytest manually before relying on a change.
• --platform targets a single named machine; there's no way to target "any machine matching GPU model X" other than letting the auto-target balancer pick, or writing a custom HTCondor requirements expression by hand.
• The default GPU capability tiers are a rough manual ranking, not derived from benchmarks -- adjust with the gpu_tiers config key if it doesn't match your priorities.

License

MIT © Omar Soliman.

condorctl is independent software and is not affiliated with or endorsed by NMRbox. Use it only with pools you are authorized to access, and in line with that pool's acceptable-use policy.