| |
N9LYA > NET44 07.09.26 16:00l 290 Lines 13610 Bytes #287 (0) @ WW
BID : 12960_N9LYA
Read: GUEST
Subj: N9LYA net44 status update
Path: IZ3LSV<IW0QNL<HB9ON<HB9ON<PI8ZTM<VE3CGR<PY2BIL<N9LYA
Sent: 260907/1341Z 12960@N9LYA.#SIN.IN.USA.NOAM LinBPQ6.0.25
================================================================
FIXING THE N9LYA-8 AXIP LINK TO N2NOV
N9LYA Homelab -- AMPRNet / AXIP Postmortem
2026-09-05 to 2026-09-07 -- N9LYA (Jerry Kutche) & Claude
================================================================
Charles Hargrove (N2NOV) reported he could not establish an AXIP
link to N9LYA-8 (LinBPQ, 44.48.0.42). What looked at first like a
single misconfiguration turned out to be SEVEN INDEPENDENT, STACKED
FAULTS across three systems -- a router, two Linux boxes, and
finally ARDC's own portal -- each masking the next. This is the
full trail, in the order things were actually found.
----------------------------------------------------------------
01. BUG 1 -- No route for our own block [FIXED]
----------------------------------------------------------------
RB5009 / missing static route
The router had generic catch-all routes for the WireGuard-based
"44Net Connect" tunnel (wg-44net, ARDC's newer replacement for the
legacy IPIP mesh, covering 44.0.0.0/9 and 44.128.0.0/10) -- but
nothing telling it what to do with traffic for our own specific
block once it arrived. Inbound WireGuard traffic for 44.48.0.42 had
nowhere to go.
/ip route add dst-address=44.48.0.40/29 gateway=192.168.1.16
----------------------------------------------------------------
02. BUG 2 -- Firewall silently dropping it anyway [FIXED]
----------------------------------------------------------------
RB5009 / asymmetric forward-filter coverage
A neighboring block (44.128.0.0/10) had a proper bidirectional
forward-chain accept pair. The /9 range covering our own block only
had an outbound-only rule (LAN->tunnel). Inbound connections were
silently dropped with no matching rule -- even with the route from
Bug 1 in place.
/ip firewall filter add chain=forward action=accept src-address=44.48.0.40/29
/ip firewall filter add chain=forward action=accept dst-address=44.48.0.40/29
----------------------------------------------------------------
03. BUG 3 -- Replies leaking out the wrong door [FIXED]
----------------------------------------------------------------
RB5009 / return-path routing mark scoped to the wrong chain
An existing mangle rule marked 44Net connections on the way in
(chain=prerouting, matches any packet) and routed their replies back
through the tunnel -- but only for chain "output", i.e. replies the
router itself originates. LinBPQ and URONode are separate LAN hosts,
so their replies traverse "forward", never "output" -- they fell
through to the default route and got masqueraded out the WAN as our
public IP instead of routing back through the tunnel with real 44.x
addressing.
Correction mid-fix: RouterOS only allows mark-routing in
prerouting/output, not forward (the routing decision already
happened by then). The actual fix is a second rule in prerouting --
conntrack marks are per-connection, so it correctly catches the
LAN-originated reply too.
/ip firewall mangle add chain=prerouting action=mark-routing \
connection-mark=44net-in new-routing-mark=44net-return passthrough=no
----------------------------------------------------------------
04. BUG 4 -- LinBPQ's own firewall, a second time [FIXED]
----------------------------------------------------------------
LinBPQ (192.168.1.42) / local INPUT chain missing IPIP
Separate from anything upstream: LinBPQ's own iptables (default-DROP)
had zero rule admitting protocol 4 (ipencap) from URONode. The
re-encapsulated relay envelope from URONode was dropped locally
before it could ever be unwrapped for decapsulation on ampr0.
iptables -I INPUT 1 -s 192.168.1.16 -p ipencap -j ACCEPT
----------------------------------------------------------------
05. BUG 5 -- A hardening pass from two weeks earlier
[Found earlier same day by a separate session, verified still fixed]
----------------------------------------------------------------
URONode / CSF blocking protocol 93/94
A separate session earlier the same day traced a real, dateable
regression: CSF (installed 2026-08-25 during a broader
security-hardening pass) only understands TCP/UDP/ICMP by default --
it silently dropped all inbound AX.25-encap (protocol 93) and IPIP
(protocol 94) traffic at the OS firewall level on URONode ever
since, with no exception ever written for it. Confirmed still in
place:
-A INPUT -p ax.25 -j ACCEPT
-A INPUT -p ipip -j ACCEPT
----------------------------------------------------------------
06. BUG 6 -- Replying with the wrong identity [FIXED]
----------------------------------------------------------------
LinBPQ / outbound AXIP presenting the LAN IP, not the AMPR identity
Captured LinBPQ's actual outbound replies: sourced from 192.168.1.42
(its plain LAN IP) for every peer, never 44.48.0.42. For any peer
dialing the real AMPR address directly, the reply wouldn't match
what their client expects -- and worse, since it wasn't sourced from
our own block, it never matched the connection-tracking state from
Bug 3, so it fell through to the ordinary masquerade path regardless.
/ip firewall nat add chain=srcnat protocol=udp src-address=192.168.1.42 \
src-port=10093 action=src-nat to-addresses=44.48.0.42
----------------------------------------------------------------
07. THE REAL ROOT CAUSE
----------------------------------------------------------------
NONE OF THE ABOVE COULD EVER HAVE MATTERED.
Checking ARDC's own portal (portal.ampr.org) directly: 44.48.0.40/29
-- the block N9LYA-7 and N9LYA-8 both live in -- was registered
MODE: PRIVATE, meaning it was never published to the public AMPRNet
routing table at all. Neighboring blocks under the same account
(44.48.5.0/29, 44.48.16.0/24, 44.48.18.0/26, 44.48.150.0/24) were all
correctly published, pointing at the real WAN IP. This one wasn't --
alongside a lease showing EXPIRED SINCE 2019-10-22.
No AMPR-connected router anywhere in the world -- including
Charles's own gateway -- could ever have known to route traffic for
44.48.0.42 toward this network, in either direction, regardless of
anything fixed locally. This explains why outbound connect attempts
left cleanly (confirmed via live packet capture, real SABM retries
going out) but never got a reply back: there was nowhere for the
world to send it.
Network : 44.48.0.40/29
Mode : Private -> Open
Lease : Expired 2019-10-22 -> Renewed
Fixed directly in the portal: Networks -> 44.48.0.40/29 -> Edit ->
Mode: Open, description filled in (was blank, portal-required),
lease renewed.
----------------------------------------------------------------
CONFIRMATION
----------------------------------------------------------------
Propagation was fast enough to confirm live, same session:
Convers test
44.68.41.2:3600 -- failed with "Aborted" all day, connected
clean on first retry after the portal fix. Live session, 15
real users, 9 channels, half a dozen countries.
URONode heard list
N2NOV-4 (24 pkts, Text) and N2NOV-7 (33 pkts, NetRom) -- his
station, genuinely heard, first time all day.
Also corrected along the way: an earlier working theory had
misattributed unrelated NET/ROM "NODES" broadcast traffic (decoded
from a live packet capture -- PID 0xCF, destination NODES) to
Charles's connection attempts. His real public IP (162.247.79.40)
was confirmed directly by him and used to correct LinBPQ's MAP
table entry.
----------------------------------------------------------------
INCIDENTAL CLEANUP, SAME SESSION
----------------------------------------------------------------
- URONode VM hard-locked mid-session -- 100% packet loss, matching
a known prior guest-kernel-freeze pattern (VM alive at the Proxmox
level, zero response to ping/TCP). Recovered cleanly with
"qm reset 110".
- FlexNet duplicate-launch race -- running "ax25 restart" manually
raced against the independent Keepalive-flexnet cron watchdog
(both call the same launcher, with no coordination), launching two
competing dosemu instances fighting over tap0. Cleaned up manually
(kill both, delete tap interfaces, clean single relaunch, restart
MSYS after).
- Fixed the race at the source -- the watchdog already flocked
/var/run/Keepalive-flexnet.lock; the init script's FlexNet launch
line now shares the same lock, so the two mechanisms can never
step on each other again.
- Bad bpq32.cfg lines found, two fixed -- VARA mode typos
(VARA23000 / VARA2000 -> VARA2300) on Port 4. One left
deliberately alone (MAP KA1VSC-15, missing a port number) pending
confirmation from that station.
----------------------------------------------------------------
SEARCH & DESTROY -- THE ACTUAL METHOD
----------------------------------------------------------------
None of the above came from reading config files and guessing.
Every bug was found the same way: assume nothing, watch real
traffic, and verify at every layer before moving to the next.
- Hex-dumped live packets and hand-decoded the AX.25 frame inside
them. Captured traffic that looked like it might be Charles's
connection attempt, then read the raw bytes byte-by-byte --
7-byte shifted-ASCII callsign fields, control/PID bytes -- and
found destination NODES, PID 0xCF (NET/ROM). It was routine
broadcast traffic from an unrelated station, not him at all.
Caught before it wasted more time on the wrong IP.
- Caught our own misattribution, twice, by re-verifying instead
of trusting timing. An earlier "confirmed working" claim (based
on a repeating "KISS Session Stats" log line) turned out to be
a stale, stuck one-way session -- identical numbers repeating,
not fresh successes. Both times, going back to raw packet
evidence rather than trusting the first plausible-looking log
line is what caught it.
- Traced one symptom through every layer in order, verifying real
traffic at each hop -- routing table -> firewall filter ->
NAT/masquerade -> connection-tracking mangle -> application
socket/MAP config -> external portal registration. A fix at one
layer was never assumed to mean the whole path worked; each hop
was re-checked with a live capture before moving to the next.
- Cross-referenced config files against live kernel state, not
just each other. Static ax25d.conf lock rules were checked
against live /proc/net/ax25 socket state; bpq32.cfg's MAP table
was checked against actual observed UDP source addresses in a
live capture, not assumed correct from the file alone.
- Learned the hard way to stop probing live-held resources
directly. An early raw stty/dd read against a serial port
LinBPQ already had open crashed the process. Every diagnostic
after that used passive methods only -- ss, journal logs,
interface byte counters -- never touching a resource something
else already held.
- Drove an interactive telnet-based node shell non-interactively
to run live connect tests and capture exact banner/error text
on demand, by scripting the login sequence and timed command
injection through a raw socket rather than waiting on manual
terminal access.
- Found and reconciled a second, independent investigation into
the same problem -- a separate session's same-day CSF fix,
found by searching the actual email thread with Charles rather
than assuming a clean slate, then verified it was still live
before building anything else on top of it.
- Backed up before every production change -- router
NAT/route/filter/mangle tables, bpq32.cfg, the ax25 init
script, the Uptime Kuma database -- each as a timestamped
snapshot, every time, before touching anything live.
- Went external once every local layer was exhausted. After the
router, both Linux boxes, and their firewalls all checked out
clean with no remaining explanation, the next step was checking
the one thing entirely outside this network: ARDC's own portal
registration. That's where the real root cause was.
----------------------------------------------------------------
MONITORING ADDED
----------------------------------------------------------------
Extended the existing generic n8n workflow ("Homelab Health Check
Alert" -- webhook-driven, per-host edge-triggered state tracking,
email + live tracker card on real transitions only) to cover
URONode two ways:
FULL OFFLINE
New Uptime Kuma webhook notification attached to the existing
URONode ping monitor (60s interval), custom body template mapped
to the workflow's contract:
{"host":"URONode",
"status":"{% if heartbeatJSON['status']==1 %}ok{% else %}fail{% endif %}",
"issues":["{{ msg }}"], "timestamp":"{{ heartbeatJSON['time'] }}"}
SERVICES UP, RADIO DEAD
New local script, cron every 5 min (URONode is sysvinit, no
systemd timers available) -- reports separately as
"URONode-Services" so a software failure doesn't get confused
with a full outage. Checks:
- ax25d / ax25ipd processes
- ampr-ripd process
- FlexNet dosemu process + tap0 carrier
- JNOS process
- ax0/ax1/ax2 link state
- beacon process count (expect 3)
/usr/local/bin/uronode-health-check.sh, posting JSON to the
shared homelab-health-alert webhook -- same alerting pipeline,
same edge-triggering logic, no new infrastructure.
================================================================
N9LYA homelab -- written up 2026-09-07
================================================================
73 Jerry N9LYA
https://n9lya.com
n9lya@n9lya.#sin.in.usa.noam
Read previous mail | Read next mail
| |
