With my most populous post “Basic Checkpoint Gaia CLI Commands (Tips and Tricks)“, I would like to
collect some more advanced troubleshooting commands used in my daily work into this post. Actually, some of commands are not only for Checkpoint Gaia, it will be for SPLAT or IPSO platform as well. This post will keep updating as soon as I have something new.
1. fw ctl chain
Check Checkpoint Security Gateway packet inspection order/chain. For more details, check the post “How Firewalls (Security Gateways) Handle the Packets?“
in chain (18):
0: -7f800000 (f28854f0) (ffffffff) IP Options Strip (in) (ipopt_strip)
1: -7d000000 (f1796f10) (00000003) vpn multik forward in
2: – 2000000 (f177cb70) (00000003) vpn decrypt (vpn)
3: – 1fffff8 (f1787c00) (00000001) l2tp inbound (l2tp)
4: – 1fffff6 (f2886ca0) (00000001) Stateless verifications (in) (asm)
5: – 1fffff5 (f28bce30) (00000001) fw multik misc proto forwarding
6: – 1fffff2 (f17a4df0) (00000003) vpn tagging inbound (tagging)
7: – 1fffff0 (f177a150) (00000003) vpn decrypt verify (vpn_ver)
8: – 1000000 (f29049c0) (00000003) SecureXL conn sync (secxl_sync)
9: 0 (f282f810) (00000001) fw VM inbound (fw)
10: 1 (f28a6b30) (00000002) wire VM inbound (wire_vm)
11: 2000000 (f177b5e0) (00000003) vpn policy inbound (vpn_pol)
12: 10000000 (f2902cb0) (00000003) SecureXL inbound (secxl)
13: 7f600000 (f287ab70) (00000001) fw SCV inbound (scv)
14: 7f730000 (f2a13500) (00000001) passive streaming (in) (pass_str)
15: 7f750000 (f2c0bef0) (00000001) TCP streaming (in) (cpas)
16: 7f800000 (f2885890) (ffffffff) IP Options Restore (in) (ipopt_res)
17: 7fb00000 (f2fac050) (00000001) HA Forwarding (ha_for)
out chain (15):
0: -7f800000 (f28854f0) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: -78000000 (f1796ef0) (00000003) vpn multik forward out
2: – 1ffffff (f1779a10) (00000003) vpn nat outbound (vpn_nat)
3: – 1fffff0 (f2c0bd70) (00000001) TCP streaming (out) (cpas)
4: – 1ffff50 (f2a13500) (00000001) passive streaming (out) (pass_str)
5: – 1ff0000 (f17a4df0) (00000003) vpn tagging outbound (tagging)
6: – 1f00000 (f2886ca0) (00000001) Stateless verifications (out) (asm)
7: 0 (f282f810) (00000001) fw VM outbound (fw)
8: 1 (f28a6b30) (00000002) wire VM outbound (wire_vm)
9: 2000000 (f1779c30) (00000003) vpn policy outbound (vpn_pol)
10: 10000000 (f2902cb0) (00000003) SecureXL outbound (secxl)
11: 1ffffff0 (f17887b0) (00000001) l2tp outbound (l2tp)
12: 20000000 (f177d5b0) (00000003) vpn encrypt (vpn)
13: 7f700000 (f2c0e340) (00000001) TCP streaming post VM (cpas)
14: 7f800000 (f2885890) (ffffffff) IP Options Restore (out) (ipopt_res)
2. Proxy Arp
a. Use the Gaia portal.
Network Management -> Arp -> Proxy ARP
b. Use the command line (in Gaia):
add arp proxy ipv4-address 172.16.0.8 interface eth0 real-ipv4-address 172.16.0.22
Actually the GAIA command above convert it automatically to a file called local.arp
c. Use the command line (in expert mode):
Then insert the information directly to /opt/CPsuite-R76/fw1/conf/local.arp
echo “172.16.0.8 00:0c:29:f1:b7:74 172.16.0.22” >> $FWDIR/conf/local.arp
Verify the changes after a policy push with command “fw ctl arp”:
|
[[email protected]:0]#
fw ctl arp
(10.9.3.21) at 00-1c-7f-32-cc-15 (10.9.3.53) at 00-1c-7f-32-cc-15 (10.9.3.35) at 00-1c-7f-32-cc-15 (10.9.3.26) at 00-1c-7f-32-cc-15 (10.9.3.29) at 00-1c-7f-32-cc-15 (10.9.3.80) at 00-1c-7f-32-cc-15 (191.24.11.13) at 00-1c-7f-33-07-ae interface 191.24.11.116 (10.9.3.25) at 00-1c-7f-32-cc-15 (10.9.3.61) at 00-1c-7f-32-cc-15 (10.9.3.28) at 00-1c-7f-32-cc-15 (10.9.3.24) at 00-1c-7f-32-cc-15 (10.9.3.27) at 00-1c-7f-32-cc-15 FW-GAIA> show arp proxy all IP Address MAC Address / Interface Real IP Address 200.0.0.102 eth0 |
Reference: Checkpoint SPLAT Manual Proxy ARP Configuration Example
3. fw ctl zdebug drop
lists all dropped packets in real time gives an explanation why the packet is dropped
4. TCPDUMP
- tcpdump port 257 , <– on the firewall, this will allow you to see if the logs are passing from the firewall to the manager, and what address they are heading to.
- tcpdump -i WAN.15 <- to capture everything on this interface
- tcpdump -i eth1.16 icmp <– to capture just PINGs on this interface
- tcpdump -i Mgmt -vvv -s0 -w tcpdumpfile.log <– this captures the FULL packets to a file usefull for wireshark the -s0 stops the files being shortened
- tcpdump -i INT port 67 <– view dhcp requests
- tcpdump -eP -nni any host 10.9.4.30 <-disable both name and service port resolution while performing a capture, by using the -nn option; -e Print the link-level header on each dump line. This can be used, for example, to print MAC layer addresses for protocols such as Ethernet and IEEE 802.11. -p–no-promiscuous-mode.
- tcpdump -i any <- any can be used to tell tcpdump to listen on all interfaces
- tcpdump -n <- disable to lookup and translate hostnames and ports.
Reference: Understanding TCPDUMP Output
5. FW Monitor
- fw monitor -e ‘accept host(192.168.1.12);’ <– Show packets with IP 192.168.1.12 as SRC or DST
- fw monitor -e ‘accept src=192.168.1.12 and dst=192.168.3.3;’ <–Show all packets from 192.168.1.12 to 192.168.3.3
- fw monitor -pi ipopt_strip -e ‘accept udpport(53);’ <–Show UDP port 53 (DNS) packets, pre-in position is before ‘ippot_strip’
- fw monitor -m O -e ‘accept udp and (sport>1023 or dport>1023);’ <– Show UPD traffic from or to unprivileged ports, only show post-out
- fw monitor -e ‘accept net(192.168.1.0,24) and tracert;’ <–Show Windows traceroute (ICMP, TTL<30) from and to network 192.168.1.0/24
- fw monitor -v 23 -e ‘accept tcpport(80);’ <–Show Capture web traffic for VSX virtual system ID 23
- fw monitor -e ‘accept ip_p=50 and ifid=0;’ <–Show all ESP (IP protocol 50) packets on the interface with the ID 0. (List interfaces and corresponding IDs with fw ctl iflist)
- srfw monitor -o output_file.cap <–Show traffic on a SecuRemote/SecureClient client into a file. srfw.exe is in $SRDIR/bin (C:Program FilesCheckPointSecuRemotebin)
6. VPN tu
vpn tu or vpn tunnelutil ********** Select Option **********
(1) List all IKE SAs
(2) List all IPsec SAs
(3) List all IKE SAs for a given peer (GW) or user (Client)
(4) List all IPsec SAs for a given peer (GW) or user (Client)
(5) Delete all IPsec SAs for a given peer (GW)
(6) Delete all IPsec SAs for a given User (Client)
(7) Delete all IPsec+IKE SAs for a given peer (GW)
(8) Delete all IPsec+IKE SAs for a given User (Client)
(9) Delete all IPsec SAs for ALL peers and users
(0) Delete all IPsec+IKE SAs for ALL peers and users
(Q) Quit
7. Disk/File/Folder Commands
Checkpoint SK60080 displays some solutions to resolve excessive disk consumption on SPLAT/Gaia/IPSO/Lunix OS system. Here are some helpful commands:
a. df -h (view the partition table and its associated utilization)
b. du -h –max-depth=1 /opt | sort -n -r (examine disk space utilization at directory-level)
c. ls -1 $FWDIR/conf/db_versions/repository/ | wc -l (check the number of database revisions on a Security Management server)
d. ls -l $RTDIR/distrib/* | wc -l (counts the number of records)
e. evstop & evstart (Stop / start the Eventia / SmartEvent)
f. rm -r $RTDIR/distrib/* (Purge this directory of stale records)
8. Connections
CP-1> fw tab -t connections -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 77 948 179
Note:
The NAME Id is the actual table number.
The VALS colum is the current number of connections that are in the connections table at the time the command was run.
The PEAK number is the max number of connections that have been recorded since the last reboot.
The SLINKS table is a table of symbolic link that point to the real connection entry. There are usually 4 symbolic links per connection. This way no matter which direction the packet comes, there will be an entry for it. There is more to it than that, but that is the general idea.
CP-1> fw ctl pstat
System Capacity Summary:
Memory used: 8% (62 MB out of 696 MB) – below watermark
Concurrent Connections: 0% (79 out of 24900) – below watermark
Aggressive Aging is in detect mode
Hash kernel memory (hmem) statistics:
Total memory allocated: 71303168 bytes in 17408 (4096 bytes) blocks using 1 pool
Total memory bytes used: 9703728 unused: 61599440 (86.39%) peak: 18891512
Total memory blocks used: 2665 unused: 14743 (84%) peak: 4705
Allocations: 198489371 alloc, 0 failed alloc, 198382561 free
System kernel memory (smem) statistics:
Total memory bytes used: 117769900 peak: 120093268
Total memory bytes wasted: 996590
Blocking memory bytes used: 2530356 peak: 2557584
Non-Blocking memory bytes used: 115239544 peak: 117535684
Allocations: 433810 alloc, 28 failed alloc, 432937 free, 0 failed free
vmalloc bytes used: 114086588 expensive: no
Kernel memory (kmem) statistics:
Total memory bytes used: 56103032 peak: 66020104
Allocations: 198922588 alloc, 28 failed alloc
198815489 free, 0 failed free
External Allocations: 0 for packets, 0 for SXL
Cookies:
90753187 total, 0 alloc, 0 free,
7839 dup, 2107678 get, 160176 put,
91154457 len, 0 cached len, 0 chain alloc,
0 chain free
Connections:
231169 total, 7807 TCP, 4665 UDP, 182351 ICMP,
36346 other, 0 anticipated, 3 recovered, 79 concurrent,
948 peak concurrent
Fragments:
0 fragments, 0 packets, 0 expired, 0 short,
0 large, 0 duplicates, 0 failures
NAT:
80509/0 forw, 5266/0 bckw, 85750 tcpudp,
16 icmp, 10440-949656 alloc
Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 864451, retransmitted : 0, retrans reqs : 15, acks : 1826
Sync packets received:
total : 3614413, were queued : 30, dropped by net : 15
retrans reqs : 0, received 11745 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
Callback statistics: handled 11588 cb, average delay : 1, max delay : 5
9. Check Point SecureXL
To enable SecureXL, run the command:
CP[admin]# fwaccel on
To disable SecureXL, run the command:
CP[admin]# fwaccel off
Note: The fwaccel off command is not persistent and SecureXL will be enabled again after a reboot of the system. SecureXL can be permanently disabled through the CPconfig utility.
To check the number of accelerated connection and other SecureXL statistics: CP[admin]# netstat -f
To check the number of accelerated SA (VPN traffic): CP[admin]# netstat -s
To check overall SecureXL statistics: CP[admin]# fwaccel stat
10.View Checkpoint Log from CLI
fw log -n | morefw log -n -f | https
11. Revision Control Versions Location on Management Server
[[email protected]]# cd /opt/CPsuite-R75.20/fw1/conf/db_versions/repository/
[[email protected]]# ls1 11 12 13 14 15 16 17 18 2 3 4 5 6 7 8 9
[[email protected]]# cd database/
[[email protected]]# lsversioning_db.fws
12. Change user cli between BASH and CLISH
HostName> set user admin shell /bin/bash
HostName> save config
[[email protected]]# dbset passwd:admin:shell /etc/cli.sh
[[email protected]]# dbset :save
chsh -s /bin/bash admin
chsh -s /etc/cli.sh admin
for SPLAT the default shell is /bin/cpshell which is not showing from /etc/shells file.
chsh -s /bin/cpshell admin
13. Enable SFTP in Gaia
[[email protected]]# vi /etc/ssh/sshd_config
#Subsystem sftp /usr/libexec/openssh/sftp-server
to
Subsystem sftp /usr/libexec/openssh/sftp-server
[[email protected]]# /etc/init.d/sshd restart
Note: Please check my previous post: Enable SFTP to Checkpoint Gaia OS System for more details.
14. Installation of Hotfixes on Gaia or SPLAT
[[email protected]]# tar -zxvf Check_Point_Hotfix_VERSION_OS_sk104443.tgz
[[email protected]]# ./SecurePlatform_HOTFIX_NAME
[[email protected]]# reboot
a. install a hotfix on standby cluster member (CP2) then reboot it
b. failover from active cluster member (CP1) to standby cluster (CP2) after standby cluster finished rebooting
c. install hotfix on CP1 and reboot it.
|
[[email protected]:0]# md5sum Check_Point_R77.20.linux.tgz
d788583cf44389b83b0dd6990cb53f63 Check_Point_R77.20.linux.tgz [[email protected]:0]# tar -zxvf Check_Point_R77.20.linux.tgz Actions/ Actions/cpconfig Actions/CheckPackage Actions/CRSValidator Actions/GetPa …… [[email protected]:0]# ./UnixInstallScript *********************************************************** Welcome to Check Point R77_20_JUMBO_HF installation *********************************************************** Verifying installation environment for R77_20_JUMBO_HF…Done! The following components will be installed: * R77_20_JUMBO_HF Installation program is about to stop all Check Point Processes. Do you want to continue (y/n) ? y Stopping Check Point Processes…Done! Installing Security Gateway / Security Management R77_20_JUMBO_HF…Done! Installing GAIA R77_20_JUMBO_HF…Done! Installing Performance Pack R77_20_JUMBO_HF…Done! Installing Mobile Access R77_20_JUMBO_HF…Done! ************************************************************************ Package Name Status ———— —— Security Gateway / Security Management R77_20_JUMBO_HF Succeeded GAIA R77_20_JUMBO_HF Succeeded Performance Pack R77_20_JUMBO_HF Succeeded Mobile Access R77_20_JUMBO_HF Succeeded ************************************************************************ Installation program completed successfully. Do you wish to reboot your machine (y/n) ? y Broadcast message from admin (pts/2) (Mon Oct 26 16:37:44 2015): The system is going down for reboot NOW! Broadcast message from admin (pts/2) (Mon Oct 26 16:37:44 2015): The system is going down for reboot NOW! [[email protected]:0]# |
15. SSH Timeout Solutions
a. Increasing the timeout
set inactivity-timeout 720
b. Ignore Hangup
[[email protected]:0]# fw monitor -e “accept host(172.16.0.1);” -o test.cap & [1] 27524
[[email protected]:0]# ps -aux | grep “fw monitor”
admin 27524 0.7 2.1 88268 21256 pts/2 S 14:09 0:00 fw monitor -e accept host(172.16.0.1); -o test.cap
Disown the process with this command, specifying the PID:
disown 27524
Closing, or having the SSH session end due to timeout will no longer send a hangup to this process, since it is no longer a child process of the SSH session.
A new SSH session or console session can be started later, and the process can be killed manually to stop it.
kill 27524
[[email protected]:0]# nohup fw ctl kdebug -T -f -o debug.txt -m 10 -s 50000 & [1] 30209
nohup: appending output to ‘nohup.out’
This example is creating cyclic debug files, 10 files, with a maximum of 50000Kb. Again, the PID is displayed, and the output of the command is piped to the text file ‘nohup.out’.
The SSH session can be ended with “exit” or timed out, and the hangup sent to this child process will be ignored, the debug will continue running until we log in again and manually kill the PID.
16. ClusterXL Troubleshooting
16.1 Force a failover
cphaprob -d fail -s problem -t 0 register
Verify it’s in problem state with
cphaprob stat
and
cphaprob -i list
(you should see ‘fail’ in problem state)
Once you’ve finished your testing, run these two to reset it:
cphaprob -d fail -s ok reportcphaprob -d fail unregister
Reference: CheckPoint HA: How to force a failover (ClusterXL/VRRP)
16.2 cphaprob commands and troubleshooting ClustXL Problem
|
[[email protected]:0]# cphaprob -a if
Required interfaces: 5 Required secured interfaces: 1 eth1 UP non sync(non secured), multicast eth2 UP sync(secured), multicast Mgmt UP non sync(non secured), multicast eth3 UP non sync(non secured), multicast (eth3.106 ) eth3 UP non sync(non secured), multicast (eth3.102 ) Virtual cluster interfaces: 6 eth1 2.13.11.60 eth2 10.1.90.14 Mgmt 10.1.72.14 eth3.104 10.1.104.14 eth3.106 10.1.106.14 eth3.102 10.1.102.14 |
|
FW-CP1> cphaprob -i list
Built-in Devices: Device Name: Interface Active Check Current state: problem Device Name: HA Initialization Current state: OK Device Name: Recovery Delay Current state: OK Registered Devices: Device Name: Synchronization Registration number: 0 Timeout: none Current state: OK Time since last report: 64196.3 sec Device Name: Filter Registration number: 1 Timeout: none Current state: OK Time since last report: 63492.1 sec Device Name: cphad Registration number: 2 Timeout: none Current state: OK Time since last report: 2.68138e+06 sec Device Name: fwd Registration number: 3 Timeout: none Current state: OK Time since last report: 2.68137e+06 sec Device Name: routed Registration number: 4 Timeout: none Current state: OK Time since last report: 62898.8 sec |
Usually it was caused by the connection between firewall interface port and switch port. UDP port 8116 will help us to find out which one is not sending the keep-alive packets:
Cluster Control Protocol (CCP) runs on UDP port 8116, and allows cluster members to report their own states and learn about the states of other members, by sending keep-alive packets (applies only to ClusterXL clusters). Also CCP keeps cluster member sync state.
Following tcpdump shows cluster member 1 (00:00:00:00:fe:00) and cluster member 2 (00:00:00:00:fe:01) both are sending 8116 CCP packets. That is normal. If you only see one sending, you will have to check another one’s switch port vlan configuration. You may miss one vlan on switch trunk port, which has happened to me.
|
[[email protected]:0]# tcpdump -enni eth3.102 port 8116
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth3.102, link-type EN10MB (Ethernet), capture size 96 bytes 11:13:17.497801 00:00:00:00:fe:01 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 92: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 50 11:13:17.597743 00:00:00:00:fe:01 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 76: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 34 11:13:17.676067 00:00:00:00:fe:00 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 76: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 34 11:13:17.676182 00:00:00:00:fe:00 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 92: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 50 |
Note: Previous Troubleshooting Post – Checkpoint Cluster Member Down because interfaces show partially up
17. Permanent Change Global Kernel Parameters Value
Global kernel parameters exist to control (customize) the behavior of Security Gateway (kernel parameters are located in $FWDIR/boot/modules/fw*mod* kernel modules).
This control (customization) can be done on-the-fly using the fw ctl set int command (change takes effect immediately). However, the value of the kernel parameter returns to its default value after a reboot. At times, it may be required to control (customize) the behavior of Security Gateway permanently. In addition, it is necessary for some kernel parameters to be changed upon boot. fwkern.conf file is the one which holds all those kernel parameters value. If it is not existing in your system, you will need to create it manually.
The Security Gateway must be rebooted after any change in the $FWDIR/boot/modules/fwkern.conf file.
|
[[email protected]:0]#
cat /opt/CPsuite-R77/fw1/boot/modules/fwkern.conf
fwha_mac_magic=40 fwha_mac_forward_magic=41 fw_allow_simultaneous_ping=1 fwha_forw_packet_to_not_active=1 |
Reference:
- Check Point/SPLAT/Network Debug Cheat Sheet
- A tcpdump Primer with Examples
- Check Point fw monitor cheat sheet – 20141028
- Check Point CLI Reference Card – 20150617 by Jens Roesen
- Upgrading ClusterXL Deployments(R77)
扫一扫
9376
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
