-Subproject commit 5bf8614cfafe29b3ec048bdcbf89ce09a65990dc

-- `tb-enlist@.service` now stays active until `network.target` stops, so NFS storages routed over `thunderbridge` can unmount cleanly before Thunderbolt ports are detached [ISSUE-2026-002]

-  timeouts.

-# Issue ISSUE-2026-002: Planned reboot stalls on NFS storages over thunderbridge before network shutdown

-

-## Issue ID: ISSUE-2026-002

-

-**Status:** investigating  

-**Priority:** high  

-**Created:** 2026-03-07  

-**Updated:** 2026-03-07  

-**Assigned to:** unassigned

-

-

-## Summary

-

-Planned node reboot on `baobab` spent ~106 seconds in shutdown because Proxmox NFS storages were still mounted after Thunderbolt transport had already been detached from `thunderbridge`.

-

-

-## Description

-

-During a controlled reboot validation on `baobab`, guest suspend worked correctly, but the host remained reachable over ICMP for almost two minutes after `systemctl reboot`. Journal analysis showed that the Thunderbolt bridge ports were detached early in shutdown, while Proxmox only attempted to unmount NFS storages later. Because `AutoNAS-1` and `AutoNAS-2` are mounted over `192.168.10.x` through `thunderbridge`, the NFS unmount path lost transport and waited for timeout.

-

-The same investigation exposed a second maintenance risk in `pgs`: preflight cleanup could block in kernel I/O wait when it touched remote NFS-backed storages that were stale or temporarily unavailable. That does not create the slow reboot itself, but it can block the maintenance preparation step.

-

-Follow-up validation on `ebony` showed a different but related cluster behavior: `AutoNAS-1` is currently exported by `ebony` itself. During reboot, `autonas.service` stops early, which makes the node's own Proxmox NFS client mount for `AutoNAS-1` stale and it then waits for timeout during unmount. In the same window, VM `301 is-anjohibe` (PBS `anjothibe`) is intentionally suspended by `pgs`, so PBS availability loss is expected during the maintenance window.

-

-Validation on `tapia` initially showed the same class of topology problem for `AutoNAS-2`, which is locally exported there and mounted back as a Proxmox NFS storage. The first AutoNAS shutdown-ordering patch remained active, but reboot timing still stayed near the pre-fix range because `mnt-pve-AutoNAS-2.mount` waited for timeout during shutdown while PBS `andrafiabe-AutoNAS` had already become unreachable.

-

-Follow-up work in the `autoNAS` project added an explicit `nfs-server.service` drop-in for self-hosted Proxmox NFS mounts discovered from `storage.cfg`. After that second patch, `tapia` reboot timing dropped into the same range as `ebony`, confirming that the remaining blocker was provider ordering on `nfs-server.service`, not on `autonas.service`.

-

-

-## Environment

-

-- **Affected nodes:** `baobab` confirmed, likely all nodes using Proxmox NFS storages over `thunderbridge`

-- **Component:** network + storage + maintenance workflow

-- **Version/software:** Proxmox VE 9.1 / kernel `6.17.13-1-pve`, `tb-enlist@.service`, `pgs`

-

-

-## Steps to Reproduce

-

-1. On a node with Proxmox NFS storages routed over `thunderbridge`, run `/usr/local/sbin/pgs suspend -v`.

-2. Trigger `systemctl reboot`.

-3. Measure ICMP availability during shutdown and boot.

-4. Inspect `journalctl -b -1` around the reboot window.

-

-

-## Expected Behavior

-

-- NFS storages should unmount while Thunderbolt transport is still available.

-- Host should stop replying to ICMP shortly after reboot is requested.

-- `pgs suspend` should not hang because a remote NFS mount is stale.

-

-

-## Actual Behavior

-

-- First validation on `baobab`:

-  - `TIME_TO_STOP_SECONDS 105.852`

-  - `TIME_TO_FIRST_REPLY_SECONDS 130.230`

-  - `DOWNTIME_SECONDS 24.377`

-- Follow-up validation on `ebony`:

-  - `TIME_TO_STOP_SECONDS 120.275`

-  - `TIME_TO_FIRST_REPLY_SECONDS 145.840`

-  - `DOWNTIME_SECONDS 25.565`

-- Follow-up validation on `tapia` after cluster-wide AutoNAS rollout:

-  - `TIME_TO_STOP_SECONDS 123.285`

-  - `TIME_TO_FIRST_REPLY_SECONDS 149.420`

-  - `DOWNTIME_SECONDS 26.135`

-- Revalidation on `tapia` after explicit `nfs-server.service` self-hosted ordering fix:

-  - `TIME_TO_STOP_SECONDS 28.305`

-  - `TIME_TO_FIRST_REPLY_SECONDS 53.588`

-  - `DOWNTIME_SECONDS 25.283`

-- `journalctl -b -1` showed:

-  - Thunderbolt bridge ports detached at `08:48:17.989`

-  - NFS unmount only started at `08:48:30.540`

-  - `mnt-pve-AutoNAS-1.mount` and `mnt-pve-AutoNAS-2.mount` timed out at `08:50:00.604/0.605`

-- `journalctl -b -1` on `ebony` showed:

-  - `autonas.service` stopped at `11:04:22.326`

-  - `mnt-pve-AutoNAS-2.mount` unmounted successfully by `11:04:38.693`

-  - `mnt-pve-AutoNAS-1.mount` timed out at `11:06:08.679`

-  - only after that did `network.target` stop and `tb-enlist@thunderbolt0.service` detach from `thunderbridge`

-- A later maintenance attempt also showed `pgs suspend` blocked in `nfs4_proc_getattr` while scanning storage paths.

-

-

-## Logs/Evidence

-

-```text

-Mar 07 08:48:17.989246 baobab NetworkManager[1096]: device (thunderbridge): bridge port thunderbolt0 was detached

-Mar 07 08:48:17.993120 baobab NetworkManager[1096]: device (thunderbridge): bridge port thunderbolt1 was detached

-Mar 07 08:48:30.540186 baobab systemd[1]: Unmounting mnt-pve-AutoNAS-1.mount - /mnt/pve/AutoNAS-1...

-Mar 07 08:48:30.541335 baobab systemd[1]: Unmounting mnt-pve-AutoNAS-2.mount - /mnt/pve/AutoNAS-2...

-Mar 07 08:50:00.604036 baobab systemd[1]: mnt-pve-AutoNAS-2.mount: Unmounting timed out. Terminating.

-Mar 07 08:50:00.605215 baobab systemd[1]: mnt-pve-AutoNAS-1.mount: Unmounting timed out. Terminating.

-```

-

-Blocked `pgs` stack during stale-NFS preflight:

-

-```text

-[<0>] rpc_wait_bit_killable+0x11/0x80 [sunrpc]

-[<0>] nfs4_do_call_sync+0x6a/0xc0 [nfsv4]

-[<0>] __nfs_revalidate_inode+0xd4/0x320 [nfs]

-[<0>] __do_sys_newfstatat+0x43/0x90

-```

-

-Validated timing after fixes on `baobab`:

-

-```text

-TIME_TO_STOP_SECONDS 14.599

-TIME_TO_FIRST_REPLY_SECONDS 35.651

-DOWNTIME_SECONDS 21.053

-```

-

-

-## Investigation Notes

-

-- 2026-03-07: Confirmed `AutoNAS-1` and `AutoNAS-2` on `baobab` are Proxmox NFS storages mounted from `192.168.10.21` and `192.168.10.22` over `thunderbridge`.

-- 2026-03-07: First reboot validation on `baobab` showed shutdown delay dominated by NFS unmount timeout, not by boot.

-- 2026-03-07: `tb-enlist@.service` had no ordering against `network.target`; systemd stopped Thunderbolt bridge membership before Proxmox unmounted remote storages.

-- 2026-03-07: Patched shared `tb-enlist@.service` with `Before=network.target` and deployed to `baobab`, then cluster-wide.

-- 2026-03-07: Separate maintenance attempt showed `pgs suspend` can block in `nfs4_proc_getattr` while scanning storage paths on stale remote NFS mounts.

-- 2026-03-07: Patched `pgs` cleanup to scan only local `dir` storages; remote storages such as NFS are skipped intentionally.

-- 2026-03-07: Revalidated on `baobab` after both fixes:

-  - NFS unmount started at `10:48:12.354/10:48:12.356`

-  - both NFS mounts unmounted successfully by `10:48:12.460`

-  - `network.target` stopped later at `10:48:16.152`

-  - ICMP loss dropped from ~106s to ~15s after reboot command

-- 2026-03-07: `pgs resume` completed successfully after reboot on `baobab`; state file survived boot and all 4 VMs + 1 CT were restored.

-- 2026-03-07: Validated `ebony` with current `pgs` and cluster-wide `thunderbolts` rollout. `pgs suspend` / `resume` succeeded for VMs `101`, `102`, `301`; state file survived reboot and restore completed.

-- 2026-03-07: `ebony` still showed long shutdown because `AutoNAS-1` is currently provided by `ebony` itself through `autonas`. Stopping `autonas.service` made the node's own NFS client mount stale and `mnt-pve-AutoNAS-1.mount` waited for timeout.

-- 2026-03-07: On `ebony`, PBS `anjothibe` availability loss during maintenance is expected because VM `301 is-anjohibe` is intentionally suspended by `pgs`, and its datastore dependency is also on `AutoNAS-1`.

-- 2026-03-07: Implemented AutoNAS shutdown-ordering experiment on `ebony`: `autonas.service` and `autonas-boot-scan.service` now declare `Before=remote-fs.target` and `Before=umount.target`.

-- 2026-03-07: Revalidated `ebony` after AutoNAS patch:

-  - previous timing: `TIME_TO_STOP_SECONDS 120.275`, `TIME_TO_FIRST_REPLY_SECONDS 145.840`

-  - new timing: `TIME_TO_STOP_SECONDS 27.573`, `TIME_TO_FIRST_REPLY_SECONDS 53.288`

-  - `mnt-pve-AutoNAS-2.mount` still unmounted cleanly

-  - `AutoNAS-1` no longer waited for the old 90s timeout, though a brief `Stale file handle` was still observed before the provider side stopped

-- 2026-03-07: Residual issue on `ebony`: even with later provider shutdown, `pvestatd` briefly logged `storage 'AutoNAS-1' is not online` / `Stale file handle` during the maintenance window, so the self-hosted NFS topology remains fragile but no longer dominates shutdown time.

-- 2026-03-07: Deployed the same AutoNAS ordering patch cluster-wide and revalidated `tapia`.

-- 2026-03-07: `pgs suspend` / reboot / `pgs resume` succeeded on `tapia` for VMs `104`, `107`, `113`, `302`; state file survived reboot and all four guests were restored.

-- 2026-03-07: `tapia` still showed slow shutdown after the AutoNAS patch:

-  - `TIME_TO_STOP_SECONDS 123.285`, `TIME_TO_FIRST_REPLY_SECONDS 149.420`

-  - `mnt-pve-AutoNAS-1.mount` unmounted immediately at `11:45:01.827`

-  - `autonas.service` and `nfs-server.service` stopped around `11:45:01.689/11:45:01.900`

-  - `mnt-pve-AutoNAS-2.mount` then waited until timeout at `11:46:31.778`

-  - `network.target` stopped only after that, at `11:46:31.781`

-- 2026-03-07: On `tapia`, the remaining delay is concentrated on self-hosted `AutoNAS-2` (`server 192.168.10.22`) plus expected maintenance-window loss of PBS `andrafiabe-AutoNAS` (`192.168.10.96`).

-- 2026-03-07: Implemented a second-generation AutoNAS fix that generates `/etc/systemd/system/nfs-server.service.d/50-autonas-self-hosted-proxmox.conf` from `storage.cfg`, adding `Before=` ordering from `nfs-server.service` to the matching self-hosted Proxmox mount units.

-- 2026-03-07: Revalidated `tapia` after the `nfs-server.service` ordering fix:

-  - previous timing after first AutoNAS patch: `TIME_TO_STOP_SECONDS 123.285`, `TIME_TO_FIRST_REPLY_SECONDS 149.420`

-  - new timing: `TIME_TO_STOP_SECONDS 28.305`, `TIME_TO_FIRST_REPLY_SECONDS 53.588`

-  - `nfs-server.service` stopped at `12:07:42.157`, `network.target` stopped later at `12:07:47.230`

-  - the old ~90s timeout on `mnt-pve-AutoNAS-2.mount` no longer dominated shutdown

-  - `pgs suspend` / reboot / `pgs resume` completed successfully for VMs `104`, `107`, `113`, `302`

-

-

-## Proposed Solution

-

-1. Keep Thunderbolt enlist units ordered before `network.target` so storage traffic over `thunderbridge` remains alive until remote filesystems are unmounted.

-2. Keep `pgs` cleanup path limited to local directory-backed storages; do not let remote NFS availability gate planned maintenance.

-3. Do not mount a node's own AutoNAS export back onto the same node as a Proxmox NFS storage; on `ebony`, exclude `AutoNAS-1` from local use or replace that local dependency with a direct/local storage path.

-4. Review colocated service dependencies before planned reboot, especially when the node provides the storage it also consumes (for example `autonas` and PBS on `ebony`).

-5. Keep the generated `nfs-server.service` self-hosted ordering drop-in as the cluster fix for nodes that export AutoNAS locally and also consume the same export back through Proxmox NFS.

-6. Validate the same shutdown path on the remaining nodes after storage-role cleanup and after the `nfs-server.service` ordering fix is deployed.

-

-

-## Related Issues

-

-- ISSUE-2026-001

-

-

-## Changelog References

-

-List CHANGELOG.md entries that reference this issue:

-- `projects/thunderbolts/CHANGELOG.md`: [Unreleased] - `tb-enlist@.service` now stays active until `network.target` stops... [ISSUE-2026-002]

-- `projects/pve-guests-state/CHANGELOG.md`: [1.5] - Suspend-artifact cleanup now scans only local `dir` storages... [ISSUE-2026-002]