Skip to content

feat(slurm): AUT-1454 persist multiplexed ssh connections - #583

Open
svcnemo-autobot wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:feat/ssh-multiplex
Open

feat(slurm): AUT-1454 persist multiplexed ssh connections#583
svcnemo-autobot wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:feat/ssh-multiplex

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Background

SlurmExecutor's Paramiko transport cannot be reused after its Python process exits. Repeated NeMo Run invocations therefore establish new SSH connections.

What changed

  • Add opt-in OpenSSH multiplexing through control_persist and an optional control_path.
  • Reuse a stable control master across NeMo Run processes for commands, SCP, rsync, and port forwarding.
  • Preserve existing Fabric/Paramiko behavior when multiplexing is not configured.
  • Add no dependencies.

Details

flowchart LR
  N1[NeMo Run process] --> M[OpenSSH control master]
  N2[Later NeMo Run process] --> M
  M --> S[Slurm host]
Loading

The master intentionally remains after Python cleanup and exits after the configured OpenSSH ControlPersist idle duration.

Tested

  • Whole-tree Ruff format and lint
  • 59 focused tunnel/rsync tests
  • 1,352 non-Docker tests with coverage on the implementation head
  • Tunnel configuration reconstruction smoke test

The Docker-scheduler module requires an unavailable Docker daemon. End-to-end Slurm validation was not performed, per task constraints.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot svcnemo-autobot changed the title feat(slurm): AUT-1454 keep ssh connections alive feat(slurm): AUT-1454 persist multiplexed ssh connections Aug 13, 2026
Comment thread nemo_run/core/tunnel/client.py Fixed
Comment thread nemo_run/core/tunnel/client.py Fixed
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants