SSH-Infrastructure / KEYS_AND_ACCESS.md
Newer Older
327 lines | 10.034kb
Bogdan Timofte authored 2 weeks ago
1
# SSH Keys and Access Guide
2

            
3
Centralized reference for SSH key management and access paths.
4

            
Bogdan Timofte authored 2 weeks ago
5
## Key Storage
Bogdan Timofte authored 2 weeks ago
6

            
Bogdan Timofte authored 2 weeks ago
7
### Local Keys (on your machine)
Bogdan Timofte authored 2 weeks ago
8

            
9
```
10
~/.ssh/keys/
11
  id_ed25519           → ~/.ssh/id_ed25519 (modern, preferred)
12
  id_ed25519_2026-05
13
  id_ed25519_2026-05.pub
14
  id_rsa               (RSA legacy, 2048-bit)
15
  id_rsa_old           (deprecated, from 2015)
16
  id_rsa_old.pub
17
  is-jumper_ed25519    (specific to is-jumper entry point)
18
  is-jumper_ed25519.pub
19
```
20

            
Bogdan Timofte authored 2 weeks ago
21
### Physical Hardware Key (on is-jumper)
22

            
23
**⚠️ IMPORTANT**: The primary authentication for the **company network (J1/J2)** is a **physical hardware security key (card/smartcard)** mounted **only on is-jumper** (192.168.2.100).
24

            
25
- **Location**: is-jumper (192.168.2.100) only
26
- **Type**: Hardware security module / smartcard (RSA 4096-bit)
27
- **Exposed via**: GPG agent at `/run/user/0/gnupg/S.gpg-agent.ssh` (on is-jumper)
28
- **Access method**: SSH agent forwarding through is-jumper
29
- **Users served**: Multiple users from the 192.168.2.0/24 network
30

            
31
**You cannot access J1/J2 directly from your local machine.** The connection must go through is-jumper, which holds the only valid physical key for the company network.
32

            
Bogdan Timofte authored 2 weeks ago
33
## Key Usage Matrix
34

            
Bogdan Timofte authored 2 weeks ago
35
| Key | Location | Used By | Purpose | Auth Method |
36
| --- | --- | --- | --- | --- |
37
| **Physical card (RSA 4096)** | Hardware on is-jumper only | J1, J2, company network | Primary auth for nextgen network | Hardware smartcard via GPG agent |
38
| **is-jumper ED25519** | `~/.ssh/keys/is-jumper_ed25519` | Your machine → is-jumper | Access the entry point | pubkey + IdentitiesOnly |
39
| **Modern ED25519** | `~/.ssh/id_ed25519` | Local lab, final hosts | Secondary/fallback auth | pubkey |
40
| **Legacy RSA** | `~/.ssh/id_rsa` | Legacy systems | Transitional | pubkey |
41
| **Deprecated RSA** | `~/.ssh/keys/id_rsa_old` | Very old hosts | Final fallback | pubkey |
42

            
43
**Key insight**: You authenticate TO is-jumper with your local ED25519 key, but once on is-jumper, the system uses the **physical hardware key** to authenticate to the company network (J1/J2).
Bogdan Timofte authored 2 weeks ago
44

            
45
## Access Paths and Routing
46

            
47
### 1. Local Lab Setup (192.168.2.0/24)
48

            
49
**Direct access** — no jump needed:
50

            
51
```
52
is-jumper (192.168.2.100)
53
├─ User: root
54
├─ Port: 22
55
├─ Key: ~/.ssh/keys/is-jumper_ed25519
56
├─ IdentitiesOnly: yes
57
└─ Routing: SSH_ROUTE=local
58

            
59
Local dev machines (192.168.2.110-122)
60
├─ User: bogdan
61
├─ Port: 22
62
└─ Key: ~/.ssh/id_ed25519
63
```
64

            
65
Access to is-jumper:
66

            
67
```bash
68
# Direct connection (SSH will use is-jumper_ed25519 via IdentitiesOnly)
69
ssh is-jumper
70

            
71
# Verify key in use:
72
ssh -G is-jumper | grep identityfile
73
```
74

            
75
### 2. Production: Via is-jumper → J1/J2 → Final Hosts
76

            
77
**Access chain for internal company network (10.253.51.0/24)**:
78

            
79
```
80
Your machine (local)
Bogdan Timofte authored 2 weeks ago
81
  ↓ (ED25519 pubkey)
82
is-jumper (192.168.2.100) [VPN client + physical card key guardian]
83
  ↓ (SSH agent forwarding + physical card RSA 4096)
Bogdan Timofte authored 2 weeks ago
84
J1 (10.253.51.50:25904) or J2 (10.253.51.52:25904) [jump hosts]
85
  ↓ (SSH forward)
86
Final host (voip, porta, radius, etc.)
87
```
88

            
Bogdan Timofte authored 2 weeks ago
89
**Critical**: You cannot connect to J1/J2 directly from your local machine because the physical hardware key is **only on is-jumper**. All company network access must go through is-jumper first.
90

            
Bogdan Timofte authored 2 weeks ago
91
#### Step 1: is-jumper (entry point)
92

            
Bogdan Timofte authored 2 weeks ago
93
Authenticate to is-jumper with your local ED25519 key:
94

            
Bogdan Timofte authored 2 weeks ago
95
- **User**: root
96
- **Host**: 192.168.2.100
97
- **Port**: 22
98
- **Key**: `~/.ssh/keys/is-jumper_ed25519`
99
- **IdentitiesOnly**: yes (forces use of specified key only)
100

            
101
```bash
102
ssh is-jumper
103
```
104

            
Bogdan Timofte authored 2 weeks ago
105
Once logged in, is-jumper has access to the physical card key via GPG agent at `/run/user/0/gnupg/S.gpg-agent.ssh`.
Bogdan Timofte authored 2 weeks ago
106

            
Bogdan Timofte authored 2 weeks ago
107
#### Step 2: J1/J2 (jump hosts) — via is-jumper's physical key
Bogdan Timofte authored 2 weeks ago
108

            
Bogdan Timofte authored 2 weeks ago
109
**SSH agent forwarding** connects your SSH client to is-jumper's physical card:
Bogdan Timofte authored 2 weeks ago
110

            
Bogdan Timofte authored 2 weeks ago
111
- **Routing**: ProxyJump is-jumper
112
- **Authentication on J1/J2**: Physical card (RSA 4096) via `SSH_AUTH_SOCK=/run/user/0/gnupg/S.gpg-agent.ssh`
113
- **User on J1/J2**: `bogdan.timofte` (company network default)
114
- **Port**: 25904 (VPN jump port)
Bogdan Timofte authored 2 weeks ago
115

            
Bogdan Timofte authored 2 weeks ago
116
```bash
117
ssh j1        # Routes: local → is-jumper → J1 (using is-jumper's physical key)
118
ssh j2        # Routes: local → is-jumper → J2 (using is-jumper's physical key)
Bogdan Timofte authored 2 weeks ago
119
```
120

            
Bogdan Timofte authored 2 weeks ago
121
**The SSH config handles this automatically** — you don't need to manually set up agent forwarding. The wrapper or ProxyJump chain uses is-jumper's physical key to authenticate.
122

            
Bogdan Timofte authored 2 weeks ago
123
**Verify J1 config**:
124

            
125
```bash
Bogdan Timofte authored 2 weeks ago
126
ssh -G j1 | grep -E '^(hostname|port|user|proxyjump)'
Bogdan Timofte authored 2 weeks ago
127
```
128

            
129
#### Step 3: Final Hosts
130

            
131
**From J1/J2 to actual hosts** (configured in J1's/J2's local SSH config):
132

            
133
- **User**: varies by host group (bogdan, bogdan.timofte, root, etc.)
134
- **Port**: 22 (standard) or 24 (jump hosts default) or custom
135
- **Key**: `~/.ssh/id_ed25519`
136
- **ProxyJump**: j1 or j2
137

            
138
```bash
139
# Example: PortaOne database server
140
ssh porta-db        # → is-jumper → J1 → 193.16.148.11
141

            
142
# Example: VoIP PBX
143
ssh voip-prov       # → is-jumper → J1 → 10.253.51.139
144

            
145
# Example: Radius database
146
ssh falticeni.radius-db  # → is-jumper → J1 → falticeni.radius-db:24
147
```
148

            
149
### 3. Emergency Public Routes (j1.next-gen.ro / j2.next-gen.ro)
150

            
151
If internal VPN is down, use public DNS names:
152

            
153
```bash
154
# Standard (internal VPN)
155
ssh j1              # 10.253.51.50:25904
156

            
157
# Emergency (public DNS)
158
ssh j1.next-gen.ro  # j1.next-gen.ro:25904
159
ssh j2.next-gen.ro  # j2.next-gen.ro:25904
160
```
161

            
162
Both routes go through is-jumper first (no direct connection).
163

            
164
## Key Migration Status
165

            
166
### Check which hosts have modern keys
167

            
168
```bash
169
# Test all local lab hosts
170
for h in is-baobab is-ebony is-tapia is-jumper is-mazeri is-toltec is-andrafiabe is-anjohibe is-nasturel is-mat; do
171
  timeout 2 ssh -o BatchMode=yes "$h" true 2>/dev/null && echo "$h: ✓" || echo "$h: ⚠"
172
done
173
```
174

            
175
### Migrate a host to modern ED25519 key
176

            
177
```bash
178
# Automatic migration (uses legacy key to install modern key)
179
tools/migrate-modern-key.sh is-baobab
180

            
181
# Or migrate all
182
tools/migrate-modern-key.sh
183
```
184

            
185
See [docs/KEY_MIGRATION.md](docs/KEY_MIGRATION.md) for manual procedures.
186

            
187
## SSH Config Generation
188

            
189
The `~/.ssh/config` is **auto-generated** from inventory — do not edit manually.
190

            
191
```bash
192
# Regenerate after inventory changes
193
python3 tools/generate-configs.py
194

            
195
# Deploy to ~/.ssh/config
196
cp generated/client.conf ~/.ssh/config
197
# or use the deploy script:
198
tools/deploy-local.sh
199
```
200

            
201
**Config files generated**:
202

            
203
| File | Target | Purpose |
204
|------|--------|---------|
205
| `generated/client.conf` | local client (~/.ssh/config) | local access config |
206
| `generated/is-jumper.conf` | is-jumper (via deploy) | is-jumper alias config |
207
| `generated/j1.conf` | J1 (server-side) | final host access on J1 |
208
| `generated/j2.conf` | J2 (server-side) | final host access on J2 |
209

            
210
## Troubleshooting
211

            
Bogdan Timofte authored 2 weeks ago
212
### "Permission denied (publickey)" on J1/J2
213

            
214
**⚠️ First check**: Do you have **access to is-jumper**?
215

            
216
```bash
217
ssh is-jumper "echo access ok"
218
```
219

            
220
If this fails, the issue is your local ED25519 key or is-jumper authentication. Fix that first.
221

            
222
**If is-jumper works but J1 fails**:
223

            
224
The issue is likely the **physical card key is not properly exposed** or **SSH agent forwarding isn't working**.
Bogdan Timofte authored 2 weeks ago
225

            
226
**Diagnosis**:
227

            
228
```bash
Bogdan Timofte authored 2 weeks ago
229
# 1. Check if you can reach is-jumper
230
ssh is-jumper
231

            
232
# 2. From is-jumper, verify the physical card is available
233
ssh is-jumper "ls -la /run/user/0/gnupg/S.gpg-agent.ssh"
234
# Should exist and be readable
Bogdan Timofte authored 2 weeks ago
235

            
Bogdan Timofte authored 2 weeks ago
236
# 3. Check J1 connectivity with verbose output
237
ssh -vvv j1 2>&1 | grep -E "ProxyJump|Offering|Authentications|Trying"
Bogdan Timofte authored 2 weeks ago
238
```
239

            
240
**Solutions**:
241

            
Bogdan Timofte authored 2 weeks ago
242
1. **is-jumper's SSH agent not running**:
Bogdan Timofte authored 2 weeks ago
243
   ```bash
Bogdan Timofte authored 2 weeks ago
244
   ssh is-jumper "ps aux | grep gpg-agent"
245
   # If not running, restart it or contact the system administrator
Bogdan Timofte authored 2 weeks ago
246
   ```
247

            
Bogdan Timofte authored 2 weeks ago
248
2. **SSH config not set up for agent forwarding**:
249
   - Verify ProxyJump is configured: `ssh -G j1 | grep proxyjump`
250
   - Verify `ForwardAgent yes` is set (check generated config)
251

            
252
3. **Physical card key not accessible**:
253
   - This is a system issue on is-jumper — contact the key administrator
254
   - The card may need to be re-mounted or re-authenticated
Bogdan Timofte authored 2 weeks ago
255

            
Bogdan Timofte authored 2 weeks ago
256
4. **SSH on your machine not supporting agent forwarding**:
Bogdan Timofte authored 2 weeks ago
257
   ```bash
Bogdan Timofte authored 2 weeks ago
258
   # Ensure SSH_AUTH_SOCK is set when you SSH to is-jumper
259
   ssh -A is-jumper
Bogdan Timofte authored 2 weeks ago
260
   ```
261

            
Bogdan Timofte authored 2 weeks ago
262
### "Permission denied (publickey)" on is-jumper
263

            
264
**The issue**: Your local ED25519 key isn't authorized on is-jumper.
265

            
266
**Verify your key is present**:
267

            
268
```bash
269
ls -la ~/.ssh/keys/is-jumper_ed25519
270
cat ~/.ssh/keys/is-jumper_ed25519.pub
271
```
272

            
273
**Check that is-jumper has your key**:
274

            
275
```bash
276
ssh is-jumper "cat ~/.ssh/authorized_keys | grep -i ed25519"
277
```
278

            
279
If your key isn't there, it needs to be added by the is-jumper administrator.
280

            
Bogdan Timofte authored 2 weeks ago
281
### "No route to host" (10.253.51.x)
282

            
Bogdan Timofte authored 2 weeks ago
283
**Cause**: Trying to reach J1/J2 directly without going through is-jumper.
Bogdan Timofte authored 2 weeks ago
284

            
Bogdan Timofte authored 2 weeks ago
285
**Check ProxyJump**:
Bogdan Timofte authored 2 weeks ago
286

            
287
```bash
Bogdan Timofte authored 2 weeks ago
288
ssh -G j1 | grep proxyjump
289
# Must show: proxyjump is-jumper
Bogdan Timofte authored 2 weeks ago
290
```
291

            
Bogdan Timofte authored 2 weeks ago
292
If not, regenerate config: `python3 tools/generate-configs.py && cp generated/client.conf ~/.ssh/config`
293

            
Bogdan Timofte authored 2 weeks ago
294
### SSH hangs or timeouts
295

            
296
**For jump hosts** (reduce to 5-10 seconds):
297

            
298
```bash
299
ssh -o ConnectTimeout=5 is-jumper
300
ssh -o ConnectTimeout=5 j1
301
```
302

            
303
**Check network**:
304

            
305
```bash
306
# is-jumper reachable?
307
ping 192.168.2.100
308

            
309
# If behind firewall, may need port 22 open to is-jumper
310
```
311

            
312
## Maintenance Checklist
313

            
314
- [ ] Keep both `id_ed25519` and `id_rsa_old` until all hosts migrated
315
- [ ] After adding new hosts: run `tools/migrate-modern-key.sh <host>`
316
- [ ] After inventory changes: regenerate with `python3 tools/generate-configs.py`
317
- [ ] Periodically check: `ssh -G <alias>` to verify config without connecting
318
- [ ] Never manually edit `~/.ssh/config` — it's auto-generated
319
- [ ] Keep keys in `~/.ssh/keys/` with mode `600`
320

            
321
## References
322

            
323
- [Ed25519 vs RSA Security](https://ianix.com/pub/ed25519-deployment.html)
324
- [OpenSSH Manual](https://man.openbsd.org/ssh)
325
- [Project inventory structure](inventory/hosts.yaml)
326
- [Key migration procedures](docs/KEY_MIGRATION.md)
327
- [Architecture details](ARCHITECTURE.md) (legacy Romanian docs: `.doc/ssh-jump-architecture.md`)