Application Down Behind F5: VIP Reachable, Pool Healthy – So Where Did the Traffic Stop?

Category: F5
Technologies: F5 BIG-IP LTM, FortiGate, TCP, SNAT
Incident type: Application connectivity
Difficulty: Intermediate

Sanitization notice: The IP addresses are
representative private values. Customer names, hostnames, credentials,
and sensitive topology details have been removed or changed.

Incident summary

The application team reported that their service was down. Connection
tests to TCP ports 20055, 30055, and
40055 failed through the F5 VIP
10.200.252.190.

FortiGate logs showed a server reset (RST), which
initially suggested a backend issue. The server team found something
different: all three ports were listening, but no connection attempt
reached the servers and no session entered the ESTABLISHED
state.

The pool was healthy and the VIP was receiving traffic. The
investigation therefore focused on one question: where did the
flow stop between the F5 and the backend servers?

Environment

Traffic path

Client -> FortiGate -> F5 VIP -> F5 server-side leg -> FortiGate -> Backend server

This path contains two separate flows:

1. Client to F5 VIP
2. F5 to backend server

Proving that the first flow works does not prove that the second flow
is permitted.

Investigation

1. Check the VIP and
source translation

The virtual server, pool, and SNAT behavior were reviewed. The server
team then checked for connection attempts from the expected F5-side
source addresses.

No traffic was visible on the backend servers.

Finding: The servers were not rejecting the
connection; they were not receiving it.

2. Confirm that traffic
reaches the F5

A packet capture was taken on the F5 for the VIP and affected
ports:

tcpdump -nni 0.0:nnn 
  'host 10.200.252.190 and (port 20055 or port 30055 or port 40055)'

The capture showed client packets arriving at the F5 correctly.

Finding: Client routing and firewall policy up to
the VIP were working.

3. Trace the server-side flow

The next capture focused on the backend network:

tcpdump -nni 0.0:nnn 
  'net 192.168.10.0/24 and (port 20055 or port 30055 or port 40055)'

The expected connection did not reach the backend servers. The
failure was on the internal leg after the F5 accepted the client
request.

4. Eliminate common F5 causes

Nothing on the F5 explained why the backend servers saw no SYN
packets.

5. Do not rely only on pool
health

The pool showed UP, but the monitor did not prove
that production traffic on all three application ports was allowed end
to end.

Monitor traffic can use a different source address, port, policy, or
path. A successful monitor proves only that the configured monitor
succeeded.

6.
Recheck the firewall using the actual server-side flow

The FortiGate logs and policies were reviewed again with the correct
values:

• Source: F5 internal addresses
  172.22.72.251-254
• Destination: Backend servers
  192.168.10.20-23
• Ports: TCP 20055, 30055,
  40055

This exposed the missing part of the design.

Root cause

The FortiGate allowed client traffic to reach the F5 VIP, but there
was no policy permitting the separate connection from the F5 internal
addresses to the backend servers on the required application ports.

The firewall terminated the downstream flow. This produced the reset
seen in the logs, even though the backend servers never received the
connection.

Resolution

A FortiGate policy was added with the following scope:

The rule was limited to the required sources, destinations, and ports
rather than using broad ANY objects.

Verification

After the policy was applied:

• TCP connection tests succeeded on all three ports.
• F5 captures showed traffic on both client-side and server-side
  legs.
• FortiGate logs showed the new policy accepting the sessions.
• Backend servers received the connections and established
  sessions.
• The application team confirmed that the service was working
  normally.

What made this incident
misleading?

Key lessons

Trace each segment
separately

Client -> firewall -> VIP -> F5 server side -> firewall -> backend

At each step, confirm both packet arrival and departure.

Verify the source
address seen by the backend

Depending on the F5 design, the server may see a SNAT, automap, Self
IP-related, or other translated source. Firewall rules must match the
actual server-side source.

A healthy pool is not
an end-to-end test

Monitor success does not validate every production source,
destination port, firewall policy, or return path.

Confirm who generated the
reset

Do not rely only on the session-end label in a firewall log.
Correlate captures and logs to locate the packet carrying the RST.

Prevention checklist

• Document both client-side and
  server-side flows for every VIP.
• Record the source address the
  backend should see.
• Include monitor ports and
  application ports in firewall reviews.
• Verify zones, policy order, NAT, and
  return routing.
• Capture on both sides of the F5
  during service validation.
• Confirm that backend servers receive
  the expected SYN packets.
• Enable logging on relevant allow and
  deny policies.

Final takeaway

The traffic appeared to fail around the F5, but the decisive evidence
came from comparing its two traffic legs. The client reached the VIP,
while the backend never received a SYN.

When a VIP is reachable, pool members are healthy, and servers see no
connection attempt, trace the server-side flow using the actual
F5 source address. That check can quickly reveal the missing
link.