Skip to content

Instance UNHEALTHY + locked out: SSH key mismatch with no recovery path (reset/stop unsupported) #448

Description

@makaylatang

Bug Report

Summary

Instance is UNHEALTHY and completely locked out due to SSH public key mismatch. None of the self-serve recovery options work for this instance type.

Instance Details

  • Name: srivatsanlab-kin
  • ID: 6jda575vz
  • Org: Genome-Scale Diffusion Models for Multimodal Genetic Design
  • Machine: gcp.h100x8.sxm.academic-grant
  • Status: UNHEALTHY (BUILD: COMPLETED, SHELL: READY)

Error

nvidia@srivatsanlab-kin: Permission denied (publickey).

This occurs on both nvidia@srivatsanlab-kin and nvidia@srivatsanlab-kin-host.

Root Cause

The instance has an old SSH public key in authorized_keys. After running brev login, a new brev.pem is generated locally but the new public key cannot be pushed to the instance because it is already UNHEALTHY — a deadlock.

Steps to Reproduce

  1. Instance becomes UNHEALTHY (for any reason)
  2. brev login regenerates ~/.brev/brev.pem with a new key pair
  3. New public key cannot be pushed to the UNHEALTHY instance
  4. All SSH access is permanently lost

Recovery Attempts

Command Result
brev login --token <token> Succeeds locally, but new key not pushed to instance
brev refresh Refreshes local SSH config only
brev reset srivatsanlab-kin ❌ "This instance does not support reset"
brev stop srivatsanlab-kin ❌ "instance does not support stop"
brev exec srivatsanlab-kin ... ❌ Permission denied (publickey)
brev exec srivatsanlab-kin --host ... ❌ Permission denied (publickey)

My Current Public Key

Please add this to /home/nvidia/.ssh/authorized_keys on the instance (ID: 6jda575vz):

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCzkiTZ/W11YyQGr2COM+JNegIibTP3OugU9eecJ6aD7SRdJqOT0emZuDgo9xKNwztrQCRqOPH+xbrCmYlZ82NwBKFs/mQSLVEsXvsNCGNY6le+d7gHycAHf7KFozQIltmlcUxDpv6WaGKM2yGoJoQt+HH+be2O3FdUmVciNfLFxpXs/KBsrgLmKtxSyMelNQ5Sl8sz0rQoTWv34sydAToAq6QmLqFqjkTqOa53WzF71fwwmw+iB8OJDDGFxKHrzYAR3qX0wUrzToQugfFNKc6VA9pIN97yz635dIVUnNQZcOS9lj6EdbZn/XhxF7uJIQ6kuUId+KegJ4rIZa8U6xpgKxQDKqvple70cYwRV0CJGU8/m7XIXlV4dxZSSNKIhWqmgWaJDcV6MCuktfTULnrR/uu70nhjgtvu11ta1uxReGfLMlnb5lxH2NzDOE5rWoXwF1/DmO6DSjEU+g4fqTnLQwgRVICQPFDHyRF2XY5Okm+RpIwNcOnKKJxlFzWdeucsSi32ouUFwLg6oPG6+IR1UGIiR3FD5Aodpq/xUloxtNJKkkj/3QUfRZMzt6ArABi1F4x6McnxP27fhqNP8mAYKUrNOhv1CNPzVZa++oJhWZLwifl0it9cJhXDrRre4v7Dse3MsjxAzcju4qd1/nMvE1Cu5yoI/OkpvuWfogHopw== makaylatang23@gmail.com

Feature Request

Even if immediate help isn't possible, this exposes a gap: instances that don't support reset or stop have no self-serve recovery path when SSH keys get out of sync. A brev repair-ssh <instance> command that re-pushes the public key via the backend (bypassing SSH) would prevent this deadlock entirely.

Environment

  • OS: macOS Darwin 25.5.0 (arm64)
  • brev CLI version: v0.6.331

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions