Out of the box, a Cosmos SDK validator keeps its consensus private key in a plaintext file: priv_validator_key.json, sitting on the same disk as the node, readable by anyone who compromises the host. For a hobbyist testnet node, fine. For a validator securing real delegated stake, this is the single largest risk in the entire system, bigger than DDoS, bigger than bugs. Whoever holds that file is the validator.
The fix is architectural, not procedural: make it so the key cannot be on the node, rather than promising it won't be. That's what remote signing does.
The remote signing model
CometBFT supports delegating all signing to an external process over a socket, via the priv validator interface. The node never sees the key; it sends a signing request ("sign this vote at height H, round R") and receives a signature back:
┌────────────┐ sign(vote H/R/S) ┌──────────────┐ PKCS#11 ┌─────────┐
│ validator │ ────────────────────→ │ tmkms signer │ ───────────→ │ HSM │
│ node │ ←──────────────────── │ (state DB) │ ←─────────── │ (YubiHSM│
└────────────┘ signature └──────────────┘ signature │ /Nitro)│
└─────────┘
holds no key holds no key, key generated inside,
enforces monotonic state can never be exportedThree properties fall out of this design, and each one closes a real incident class:
- Host compromise no longer steals the key. An attacker with root on the validator node can disrupt it, but can't exfiltrate what isn't there. The HSM generates the key internally and is physically incapable of exporting it.
- Double-signing is blocked below software. The signer persists the high-water mark (last signed height, round and step) and refuses to sign anything at or below it. This check happens regardless of which node is asking, which is what makes safe failover possible at all.
- Key ceremony becomes auditable. There is exactly one place the key exists, with tamper-evident hardware and an access log. "Who could have signed this?" has a one-line answer.
The double-sign protection detail everyone glosses over
The signer's state file is the crown jewel of the whole design, and also its most fragile part. Consider what happens during signer failover:
If you run two signers against copies of the same key "for redundancy," you have rebuilt the exact double-sign scenario the signer exists to prevent, just one level up. The high-water mark only works if it is globally authoritative.
Our production answer: one active signer per network, with the standby signer's activation gated behind a distributed lock that requires the primary's lease to expire. The failover cost is a few seconds of missed blocks, which, as we covered in the slashing post, costs precisely nothing until it lasts thirteen hours. Trading milliseconds of liveness for a hard safety guarantee is the easiest trade in infrastructure.
What runs where
| Component | Host | Holds key? | Public IP? |
|---|---|---|---|
| Validator node | Bare-metal, private VLAN | No | No |
| Sentry nodes ×3+ | Mixed cloud/bare-metal | No | Yes (rotated) |
| tmkms signer | Hardened host, separate failure domain | No (proxies to HSM) | No |
| HSM | Physically attached to signer host | Yes, non-exportable | No |
Note the separate failure domain for the signer. If signer and validator share a rack, a single power event takes out both the ability to sign and the ability to fail over. Ours live in different facilities with independent uplinks.
Practical notes if you're setting this up
- Latency budget: the signer round-trip adds single-digit milliseconds if it's in-region. Cross-continent signing (validator in Asia, signer in Europe) will make you miss precommits on fast chains. Keep them within ~20ms RTT.
- Monitor the signer as a first-class service: signing latency p99, state-file write errors, HSM session health. A wedged signer looks identical to a dead validator from the outside.
- Test tombstone protection before mainnet: on a testnet, deliberately try to make your setup double-sign. Restore a snapshot, race two nodes. If you can't force an equivocation in a lab, you've earned some confidence. If you can, better to learn it there.
- Back up the key ceremony, not the key: what you archive is the ability to regenerate your setup (mnemonics in cold storage, sharded), never a live copy of the consensus key.
Every slashing story starts the same way: "we had a copy of the key, just in case." The entire point of this architecture is that there is no copy, and there is no case.
This is layer 2 of our five-layer security model. Layer 3, the network topology that keeps the validator unreachable, is covered in Sentry Node Architecture: Keeping Validators Invisible.