Showing logs "Missing OS Route" when place new CP Appliance (Suspect ARP issue)

CheckMates Post Link:
https://community.checkpoint.com/t5/General-Topics/Showing-logs-quot-Missing-OS-Route-quot-when-place-new-CP/m-p/58911#M11879%2Fjump-to%2Ffirst-unread-message

As previously we using L3 Switch (Name: POC L3) where 4 interface which comes from two internal switches (ABC and XYZ Switch) is terminated with upstream L3 Switch (Name: POC L3) internal VLAN interface.

Two internal switches (ABC and XYZ) both having one bond interface with mode LACP.

On Checkpoint external interface also having a bond interface with network 192.168.100.0/22.

Our requirement is to place Checkpoint Firewall (23500) in between internal switch (ABC and XYZ) and L3 Switch (Name: POC L3).

We face below challenges

Scenario 1

As per our requirement, we can not add two bonds on one VLAN, If we added then it will be creating two networks, for example, bond1.100, bond2.100 with respect to VLAN 100.

So we added 4 interfaces to one BOND as bond1 with mode LACP and then create a VLAN and added that bond1 but we face the issue because all interface is active because of mode LACP.

So on this scenario, If we remove the cable from any one internal switch, for example, ABC switch then XYZ switch working fine, we able to access server as well ssh from LAN/WAN.

Scenario 2:

As the first scenario does not work as expected so we change the configuration.

We create a two BOND, bond 1 and bond 2 with respect to the internal switch and then bridge between bond 1 and bond 2 so in this scenario, we only assign one IP address on the bridge interface as 172.16.100.1/22 which meet our requirement.

So in this scenario we able to ping internal server that behind ABC and XYZ switch.

Issue:

we check the logs found that the packet drop on Checkpoint internal bond interface, showing error “missing OS route”.

Base on the logs I suspect as the ARP cache is overflowing. The most possible reason – too much traffic on the network (generated by some application, by some hosts, or by related factors). (sk108587 )

By default, the arp cache size is 4096 ARP cache slot so maximum we can increase up to 4096*4=16384 arp cache slot.

Possible Solution

1. If we use two Checkpoint device and give cluster VIP address as 172.16.100.1/22 then we will not face challenges with bond interface configuration.

2. If we place the checkpoint gateway above the POC L3 Switch then we not face challenge related to bond.

3. We need to increase the arp cache size up to 16386 arp cache slot before that we need to check the below command to verify whether arp entry will increase or not after live the checkpoint gateway.

arp -an | wc -l (To check the line on arp cache size)

Check the message file as “neighbor table overflow”

From MAX POWER Book: One interesting side effect of not having a private transit network between the internal interface of the firewall and your internal core router is that the firewall will have to maintain far more IP address to MAC address mappings in its ARP cache than it otherwise would. This is especially true if there are hundreds or thousands of workstations/servers located on VLANs directly attached to the firewall’s various interfaces.

Such a large network will, of course, have far more broadcast traffic flooded to all stations on the network including the firewall, resulting in reduced overall network throughput.

However, one situation that can occur in this case will severely damage the firewall’s performance (almost to the point of looking like rolling outages in some cases): an overflow of the firewall’s ARP cache.

Please suggest us what about the possible reason for getting the error “missing OS route”.

NOTE: Message file is overwritten so we unable to find “neighbor table overflow” also we do not have any output or arp cache size during the issue.

Now we again plane to place the Checkpoint with the proper plane so need all your help.

Solution by Chinmaya (Admin checkpointengineer.com)

New Plane v2

As per our network architecture, it seems to be arp issue because the internal network is on one broadcast domain 172.16.100.0/22 and also we configure the Bridge interface where we added two bonds (total 4 interfaces).

So we change our plane.

New Plane v2

We revise the changes on the current diagram and introduce a new switch in between Checkpoint Gateway and internal Switch (ABC and XYZ), where the new switch will handle flat network traffic. (Broadcast)

We make Checkpoint gateway inside interface 172.16.100.1/22 as the default gateway for all North and South traffic Inspection.

But still, as the internal network still have one broadcast domain so still the checkpoint maintain the ARP but we avoid the bond interface related challenges.

So If I change the arp cache to maximum 16386 arp cache slot then is there any challenges?

New Plane v3

Now we make a new plane because still on our existing setup the internal network still have one broadcast domain and still checkpoint need to maintain the arp and that situation also we need to increase the arp cache. So we plane to place a new L3 switch and make default gateway as New L3 Switch.

One additional configuration: Add a static route for incoming traffic.

command: set static-route 172.16.100.0/22 nexthop gateway address <ip address of New L3 switch external interface> on