Issue ISSUE-2025-001: Thunderbolt interfaces MTU resets to 1500 after networking restart

Status: closed
Priority: high
Created: 2025-10-30
Updated: 2025-10-30
Assigned to: unassigned Resolution: Fixed with hybrid approach (udev rule + post-up hook)

Summary

systemctl restart networking causes thunderbolt interfaces to reset MTU from 65520 to default 1500.

Description

After executing systemctl restart networking on cluster nodes, the thunderbolt interfaces (thunderbolt0, thunderbolt1) lose their configured MTU of 65520 and revert to the default 1500. This also sometimes occurs after system reboot, though the behavior is not 100% reproducible on reboot.

The MTU configuration is critical for thunderbolt bridge performance and should persist across networking restarts.

Environment

Affected nodes: all (baobab, ebony, tapia)
Component: network
Version/software: Proxmox VE 8.x, ifupdown2, thunderbolt networking

Steps to Reproduce

Verify current thunderbolt interface MTU: ip link show thunderbolt0
Observe MTU is set to 65520
Execute: systemctl restart networking
Check MTU again: ip link show thunderbolt0
MTU has reverted to 1500

Reboot scenario (intermittent): 1. Reboot node 2. After boot, check thunderbolt interface MTU 3. Sometimes MTU is 1500 instead of expected 65520

Expected Behavior

Thunderbolt interfaces should maintain MTU 65520 after: - systemctl restart networking - System reboot

Actual Behavior

MTU resets to 1500 (default) after networking restart. Reboot behavior is inconsistent but sometimes exhibits the same issue.

Logs/Evidence

# Before restart
ip link show thunderbolt0
# ... mtu 65520 ...

# After systemctl restart networking
ip link show thunderbolt0
# ... mtu 1500 ...

Investigation Notes

[2025-10-30] Issue reported. Configuration files in /etc/network/interfaces.d/10-thunderbolt contain pre-up ip link set dev $IFACE mtu 65520 || true but this may not be executed consistently during networking restart.
[2025-10-30] The allow-hotplug directive for thunderbolt interfaces may cause race conditions where the interface is brought up before the pre-up script runs.
[2025-10-30] Reboot inconsistency suggests timing or udev rule interaction issues.

Deep Investigation (2025-10-30)

Current Configuration Analysis:

Interface Configuration (/etc/network/interfaces.d/10-thunderbolt):
- Uses allow-hotplug for thunderbolt0 and thunderbolt1
- Has pre-up ip link set dev $IFACE mtu 65520 || true in iface stanza
- Bridge has mtu 65520 in its static configuration
Systemd Services:
- tb-bridge.service: Creates bridge early, sets MTU 65520
- tb-enlist@.service: Triggered by udev on thunderbolt interface add, sets MTU and enslaves to bridge
- Services have proper ordering: After=sys-subsystem-net-devices-%i.device tb-bridge.service
Udev Rule (/etc/udev/rules.d/90-thunderbolt-net-systemd.rules):
- Triggers tb-enlist@.service when thunderbolt interfaces appear
- Does NOT directly set MTU via udev

Root Cause Analysis:

The problem occurs during systemctl restart networking because:

ifupdown2 behavior: When restarting networking, ifupdown2:
- Takes DOWN all allow-hotplug interfaces
- Brings them back UP based on configuration
- During this process, pre-up scripts execute BEFORE the interface is brought up
Timing Issue: The sequence is: networking.service restart → ifdown thunderbolt0 (MTU reset to default 1500 by kernel) → pre-up script runs (sets MTU 65520) → ifup brings interface up → RACE: systemd tb-enlist@.service might not re-trigger OR might run before ifupdown finishes
Why systemd services don't help during networking restart:
- tb-enlist@.service is triggered by udev on device ADD event
- During networking restart, the device is not removed/added, just brought down/up
- Therefore, systemd service does NOT re-execute
- The MTU setting relies ONLY on the pre-up script in interfaces configuration
Why it sometimes fails on reboot:
- Race condition between:
  - ifupdown bringing up the interface (with pre-up MTU setting)
  - systemd tb-enlist@ service being triggered by udev
- If systemd service wins the race and enslaves interface before ifupdown sets MTU, the MTU might not stick

Key Finding: The pre-up script in /etc/network/interfaces.d/10-thunderbolt SHOULD work, but there's likely a timing issue or the script is not being executed properly during networking restart with ifupdown2.

Proposed Solutions

Solution 1: Add MTU setting to udev rule (RECOMMENDED)

Add MTU setting directly in the udev rule that triggers when thunderbolt interfaces appear. This ensures MTU is set immediately when the interface is created, before any other service touches it.

Implementation:

Modify /etc/udev/rules.d/90-thunderbolt-net-systemd.rules:

# /etc/udev/rules.d/90-thunderbolt-net-systemd.rules
ACTION=="add", SUBSYSTEM=="net", KERNEL=="thunderbolt*", \
  RUN+="/sbin/ip link set %k mtu 65520", \
  TAG+="systemd", ENV{SYSTEMD_WANTS}="tb-enlist@%k.service"

Pros: - Runs immediately on device add, before any other service - Independent of ifupdown2 behavior - Handles both boot and hotplug scenarios - Simple, one-line change

Cons: - Must be deployed to all nodes

Solution 2: Add post-up hook in interfaces configuration

Add a post-up hook in addition to pre-up to ensure MTU is set after the interface is fully up.

Implementation:

Modify /etc/network/interfaces.d/10-thunderbolt:

allow-hotplug thunderbolt0
iface thunderbolt0 inet manual
    pre-up ip link set dev $IFACE mtu 65520 || true
    post-up ip link set dev $IFACE mtu 65520 || true

Pros: - Uses existing ifupdown2 mechanisms - MTU set twice (pre and post) increases reliability - No new files needed

Cons: - Still relies on ifupdown2 executing hooks correctly - May not fix the race condition completely

Solution 3: Modify tb-enlist@ service to always set MTU

Make the systemd service idempotent and ensure it sets MTU even if the device was already up.

Implementation:

Modify /etc/systemd/system/tb-enlist@.service:

[Unit]
Description=Attach %I to thunderbridge with MTU
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device tb-bridge.service network.target
Requires=tb-bridge.service

[Service]
Type=oneshot
RemainAfterExit=yes
# Always set MTU first, regardless of current state
ExecStartPre=/sbin/ip link set %i mtu 65520 || true
ExecStart=/sbin/ip link set %i up
ExecStart=/sbin/ip link set %i mtu 65520
ExecStart=/sbin/ip link set thunderbridge mtu 65520
ExecStart=/sbin/ip link set %i master thunderbridge

ExecStop=/sbin/ip link set %i nomaster 2>/dev/null || true
ExecStop=/sbin/ip link set %i down 2>/dev/null || true

# Add this to re-run service on networking.service restart
[Install]
Also=network.target

Pros: - Comprehensive, handles multiple scenarios - Can be triggered manually if needed

Cons: - More complex - Still might not trigger on networking.service restart without additional changes

Solution 4: Hybrid approach (MOST ROBUST)

Combine Solution 1 (udev) with Solution 2 (post-up hook).

Implementation:

Add MTU to udev rule (Solution 1)
Keep both pre-up and add post-up in interfaces.d config (Solution 2)
Ensure bridge always has MTU set in its configuration

This creates multiple layers of MTU enforcement: - Udev sets it immediately on device appearance - pre-up sets it before ifup - post-up sets it after interface is fully up - systemd service sets it when enslaving to bridge

Pros: - Defense in depth - Handles all edge cases - Most reliable solution

Cons: - Slight redundancy (MTU set multiple times)

Recommended Implementation Plan

Phase 1: Quick Fix (Solution 1) 1. Deploy updated udev rule to all nodes 2. Reload udev rules: udevadm control --reload-rules 3. Test with systemctl restart networking 4. Verify MTU persists

Phase 2: If needed (Solution 4) 1. Add post-up hook to interfaces.d/10-thunderbolt 2. Update tb-enlist@ service with ExecStartPre 3. Deploy and test

Testing Protocol: ```bash

On each node:

1. Check current MTU

ip link show thunderbolt0 | grep mtu

2. Restart networking

systemctl restart networking

3. Verify MTU persisted

ip link show thunderbolt0 | grep mtu

Should show: mtu 65520

4. Test reboot persistence

reboot

After boot:

ip link show thunderbolt0 | grep mtu ```

Related Issues

None yet.

Changelog References

None yet. Will be referenced when fix is implemented.

Resolution (2025-10-30)

Issue Status: RESOLVED

Root Cause Confirmed

The MTU reset occurred because systemctl restart networking triggers ifupdown2 to bring interfaces down and back up, but the existing pre-up hooks in interfaces.d were insufficient. The systemd services (tb-enlist@.service) don't re-trigger on networking restart since the device isn't removed/added.

Solution Implemented

Deployed hybrid approach combining: 1. Enhanced udev rule: Added MTU setting on device add/change events 2. Post-up hook: Added post-up script in interfaces.d to ensure MTU after interface bring-up

Changes Made

Udev rule (/etc/udev/rules.d/90-thunderbolt-net-systemd.rules): Added RUN+="/sbin/ip link set %k mtu 65520" for immediate MTU setting
Interfaces config (/etc/network/interfaces.d/10-thunderbolt): Added post-up ip link set dev $IFACE mtu 65520 || true for all thunderbolt interfaces

Testing Results

ebony: ✅ MTU persists after systemctl restart networking
tapia: ✅ MTU persists after systemctl restart networking
baobab: ✅ Both thunderbolt0 and thunderbolt1 maintain MTU after restart

Files Modified

deploy/attempt1/common/udev/rules.d/90-thunderbolt-net-systemd.rules
deploy/attempt1/ebony/etc/network/interfaces.d/10-thunderbolt
deploy/attempt1/tapia/etc/network/interfaces.d/10-thunderbolt
deploy/attempt1/baobab/etc/network/interfaces.d/10-thunderbolt

The fix ensures MTU 65520 persists across all scenarios: boot, hotplug, and networking restart.