Spikefish Solutions Blog

UPDATE: Turns out this doesn't work on the 600/1100 (wah waaaah). Need some more testing (Yeah, I totally tested this on a 600/1100 before posting) to see if i can work around libc issue.

I recently... well maybe not that recently.. spent a few months working on building cross compilers that matched up %100 to a given Checkpoint Gaia Embedded system. Meaning, same libc (glibc 2.5, what a pain!), compiler version (based on glibc output) and kernel heads version.

I thought this was needed so that everything would be compatible. Well, turns out I made things way harder then it should have been. I recently found out that glibc is basically backwards compatible. There may be edge cases where things don't end up right, but for the most part, it seems pretty darn backwards compatible.

So that got me thinking. I started downloading .deb files and extracting them on my 750 and pretty much everything worked. Granted there was a lot of tracing library dependencies. So knowing that all worked I switched gears. I bought a 32 gig microsd card and installed it. I did format it to ext4 since vfat isn't a linux friendly file system. Side note: of course 7xx doesn't have the mkfs.ext4. Sigh... I'll have to map out all the needed libraries for that and point out the download links.

So the next idea was, can we just install debian on the 750 in a chroot environment? It turns out, yeah. I used the Debootstrap to create the chroot. It took a little while as it needs perl and wget and a few other things. The default wget on Gaia Embedded doesn't support https so just to be safe I pulled wget down also.

Before continuing, this is not supported by anyone. I would only do this on a test box, and not on a production firewall.

Basically I downloaded all these utilities on a spare Linux box in our Miami office:



I put them on a linux box and extracted them using this... somewhat nasty process:




What I'm doing is expanding the .deb archive, which contains 3 or more files. The binaries are in a file called data.tar.gz (gziped) or data.tar.xz (lzma). I would have done this on Checkpoint Gaia Embedded but it doesn't include anything uncompress lzma. Kind of a brute force method to extract everything, but it worked. After that the raw files are ready to install on your Checkpoint firewall.


Next I just moved the files over to the Maimi Checkpoint firewall, so now I have this:




Debootstrap is really just a shell script so once you have everything you can just run it. You also don't have to run it in Miami, I won't tell anyone if you do.


I did make a small script to setup library and path so the debootstrap files are used first. The last item was to tell debootstrap where its shell include files were.


I put this in setup.sh:



This will suck in those settings for this login session on the Miami Checkpoint firewall.




That should be about all that is needed to run debootstrap. Next, just make the dir you want to install the OS into and run debootstrap.




Then fire off debootstrap.



With luck and about 15 min you'll have a fully installed OS. We'll need a few little tweaks to wrap this up.

We need to mount proc and sysfs inside the chroot.

I added these statements to my userScript to handle this at bootup (yes the Miami Checkpoint firewall):


You can just run those mount commands by hand also if you want. The mount /mnt/sd isn't
needed if the system is already up and running as it should auto mount. However,
the auto mount happens after userScript. Adding the mount to userScript is the workaround.

Now you're ready to jump in. Here I login to the chroot and then show python version
and perl version install.



The install is about 500meg. Fits just great on a 32gig sd card, but is way too big without it.

I should point out things will work just fine inside the chroot. Once you logout, some things
will work outside of the chroot (/mnt/sd/stable-chroot/usr/bin/lsof for example), others you
may need to create a shell script to add library search and path statements, or in the case of
perl and python, do something to tell it where the modules are located.

I hope I didn't loose you at inside the chroot and outside the chroot. Chroot is a linux command that CHanges the ROOT dir.

So outside the chroot means the dir structure would look like this for example:

inside the chroot it would look like this.

Here is an example:
First I login to the chroot (now I'm inside)
I create a file called TestFile
Notice how i'm in /
And we see the TestFile
Now I logout. Notice out the directory changes? I'm now outside the chroot.
Hope that clears things up!

One interesting thing I noticed was the default ip utilties package on gaia embedded says
it doesn't support netns (Network Name Space (think VSX)), but using the Debian ip utility I was able to create a netns name. I haven't looked into this any further.


I'll have to run through the whole process again to make sure I documented it correctly.

Wait what? You would like a tar file of the debootstrap dir?

ok ok ok. Here you go.

debootstrap - 600 / 1100 / 700 / 1400

Hi everyone, if you live in south Florida (or close enough) you should join the Meetup! The South Florida Checkpoint Zdebug Group - SFCZG for short! We're just getting started with the meetup. We're researching a location somewhere in Miami. Might expand to Fort Lauderdale at some point, but for now we'll be sticking to Miami for the Checkpoint meetup.

Join and drop a line!

There is an interesting post on cpug about tcp logging options. You can get a lot more details on what happened to a tcp connection, such as when it expired, why it closed etc. Take a look!

TCP State Logging sk101221

We're taking a moment away from Check Point firewall posts in Miami to describe how to make your very own firewalls... I mean Sheboygan Bier Brats. For those who don't know, my wife is from the German province of Wisconsin and this will cover how to make a proper brat!

 Ingredients:

• Miesfield's Jalapeno Bratwurst
• 12 Pack of PBR - 9 for this 3 for drinking
• 3 x Large yellow onions
• 1 x 8 ounce Kerry Gold unsalted butter. (I think it's too salty with salted butter)
• Alstertor Dusseldorf Mustard (because it comes in a beer mug)
• Bubbies Sauerkrat (in absence of Grandma Edna's home made)
• Brioche Buns (Sheboygan hard rolls (which are soft btw (don't ask)) if you can get them)
• A German from a local province to make this

Did you know Gaia Embedded includes a port scanner?

Hello everyone! Its time for another blog post from John <at> Spikefish Solutions. The last blog post contained a bit of silliness. Let me be Frank, that isn't going to stop. This write up will be a bit more technical. I ran across the need for a Hub and Spoke mode VPN with Check Point. I hit a few problems and had to get some advice from the local Check Point Diamond rep.

This guy was super knowledgeable and is a joy to work with. To protect his identity, we'll call him XML Smith.

That aside, let's make something clear. This write up is intended to be a very detailed write up on how to set up a Hub and Spoke VPN with a Check Point Firewall, which is located in Miami, FL, as the center gateway. This basically shows how Check Point Firewalls can act as a IPSec Proxy. What do I mean by IPSec Proxy?

==

What this write up is about:
The Firewalls on the outsides (Firewall A and Firewall C) don't know the remote networks don't exist on the Center Check Point Firewall. If you look closely you'll see Firewall A and Firewall C's encryption domains are not in Firewall B's encryption domain and yet we have Firewall A making a VPN to Firewall B and Firewall B making a VPN to Firewall C. What is really happening is there are 2 VPNs, but the Firewall A and Firewall C only know of a single VPN.

Something else interesting to point out is the Satellite gateways are Interoperability Devices (interop). In this demo they will be pfsense firewalls. I used this because I didn't have access to cisco devices and didn't feel like going GNS3 route. I also didn't want to use Check Point gateways to prove there was no black magic involved here.

Ok wait a sec..before moving on. The Check Point Diamond rep's name... I'm not a fan of XML. XML is a markup language that is hard for both humans and machines to parse. Let's call him JSON Smith instead. .. phew.. ok. That sounds much better. JSON Smith in Check Point Diamond is a great guy to have in your corner. Diamond is completely worth it, espically when you call them up and say come out here and upgrade P1 for me. :D

RIGHT! So let's talk lab overview. Engage network diagram (insert scifi sound here)!

Click to enlarge ( I hope ).

Network Overview

All clients: OpenBSD 5.9 (.100 address) - Default route is pointing to .1 btw
Interop Firewall A: Pfsense 2.3.1 - Release
Check Point Firewall B: Check Point R77.30 - Single Firewall + Management
Interop Firewall C: Pfsense 2.3.1 - Release
I should also point out all Firewalls have static routes for all networks listed.

I Created 5 Vlans for this (VL100 - VL 104). I used vlans off a single virtual switch for this lab. Also no devices are trunking.

The Orange cloud represents what would be the internet. The reason I'm pointing this out is it shows how you can even have a firewall on the internal network for this design.

Debugs:
All debugs have been started before passing traffic so that you can see everything.
This is all from the Check Point Firewall B in the Center of Miami.

• TCPDUMP on eth0, eth1, eth2.
• Firewall Monitor output
• ike.elg - vpn debug ikeon
• Export of all Tracker events during the passing of traffic.

Backups:
But wait, that isn't all! I also took backups of everything! This means you can setup this in a lab and quickly restore the backups if you want! Be warned the Checkpoint Firewall in Miami Beach (ok virtual Miami Beach) backup is 128Meg. Backup links and debug files will be located at end of post so scroll all the way down if that's what you're looking for.

Also, if you are restoring the checkpoint backup you will most likely need to install an eval as the license will have expired. You'll need a central all in one eval pointing to IP 192.168.20.10. Again call your local Checkpoint SE and they will be happy to help with this if you don't have access to usercenter to generate your own eval keys.

One last quick note, the key to this config is the contents of encryption domain A and C cannot be located in encryption domain B. This is really the main configuration item to the setup and a little odd from a normal domain based VPN setup on Check point Firewall.

So let's start with the policy shall we? Here we can see ping and ssh is allowed bidirectionally from VL100 to VL102. We can also see ping is allow bidirectionally between VL100 and VL104 (this is the hub and spoke part). Nothing else is allowed. Clients are all running stock OpenBSD 5.9 and I didn't feel like setting up any other services beside SSH.



Next lets walk through the settings of Check Point Firewall B, which is our Hub and Star of our VPN. This is what you would see after double clicking Check Point Firewall B (Which is a Check Point Firewall if I didn't make that clear. I hear Check Point in Miami is a good deal also).



Here is the Topology view



Next is the IPSec VPN tab



Link Selection under that.



This is what is under the Setup button.



VPN Advanced



Firewall B encryption domain



====

Firewall A!




Firewall A Topology



Firewall A encryption domain




====

Firewall C - General



Firewall C topology



Firewall C encryption domain.




====

And here is the IPSec VPN tab configuration!



Center gateway and Satellite - Yes, the Check Point Firewall in Miami. Yes I know that's getting annoying, at least there aren't ads (cough cough yet)!



Encryption ( encryption + hashing).



And then under the Advanced tab - VPN Routing! This is where magic happens assuming you setup the encryption domains correctly. Everything else is default on the VPN Community.




Ok so that's about it for the Check Point VPN Hub and Spoke configuration examples. Everything else is default. If you want to see the other windows let me know and I'll add them, but I get the feeling there are a lot of screen shots in here already.

Lets look at logs. Notice the bounces? Those are the VPN Routing Icons that tell you magic is happening!




Debugs:
eth0.cap
eth1.cap
eth2.cap
ike.elg
Firewall_Monitor.txt
Tracker-export.txt

Backups:
backup-FirewallA.localdomain-20160624013059.xml
backup-FirewallC.localdomain-20160624013128.xml
backup_FirewallB_23_Jun_2016_18_26.tgz

Note: I haven't tried restoring any of these so please let me know if there are problems.

all logins are as follows.

User: admin
Password: vpn123

No one who worked at checkpoint can use any other login for a lib install.

Hi everyone! This will be a quick write up covering physical aspects of a Check Point 750. And let's face it, you want to know what's on under the case right? Who wouldn't?!?!

Dear Checkpoint

    Over the past 90 days I have been working on testing out checkpoint's scada protections in the 1200R firewall. After seeing what is possible with just modbus at CPX Chicago, I can hardly wait to see what else can be done!

There is just one problem. App control requires internet access.*

Checkpoint's solution for application control assumes the firewall (and / or the management server?) will have internet access. This presents a major problem for scada systems of which many do not have internet access. Ok maybe some of them do, but lets ignore those for now.

So list the issues that remain stalemated.

1. 1200R doesn't seem to support offline updates without internet access. This is what i've been told but can't verify since I can't see to get the offline update package (yet). From what I understand it has something to do with verifying the contract over the internet.
2. You have to sign a new EULA to get access to an offline update package. Once you have something happens (magic!) and then you gain access the package. The rumblings i'm hearing is its a completely manual process to install them to the firewall. I'm guessing its a tar file.
3. Assuming issues 1 and 2 are resolved and worked into a process for updating, how will I know when a new package is out? The app wiki has no release dates on it. I also haven't found any place to get email alert about to signatures. Could be missing something here.
4. bonus points, why isn't there a smartupdate package I can download?

Hi all!

I've been working on a proof of concept with Aruba Clearpass and checkpoint. Hasn't been going so well so the blog post that goes with it will have to wait. So what to talk about until then?

Well a while back I ran across a couple of things that kind of blew my mind so I thought I would pass them along. I'll also be discussing a really interesting utility I found. Stick with me or scroll to the end if you're that kind of person. I bet you read all the Walking Dead comic wikipedia entries didn't you?

So.. lets get the mind blown parts out of the way. I'm sure you know what a DLL is as most of the world knows windows. On Linux systems we have shared objects or libraries. These are mostly located in /lib and /usr/lib.

Now thats all fine and dandy. Who cares right?

Well, the core of library on linux is libc. This is basically where almost all programs start. Its bad if this goes away. Very bad.

So where am I going with this? Well I found out libraries are executable! Granted most of the time its just going to crash when you run it. BUT! libc can run just fine! Lets run it and see what happens.

Hi everyone, I know you've been having these strange urges. You have these new feelings and you're not sure what to do about them. Everyone goes through this at one point. Its part of growing up. Meet me at camera three.

Ok, so we're here to talk about Gaia Embedded of course. Gaia Embedded is the OS that runs the SMB checkpoint firewalls. Its a combo of a uboot image, busybox, lua, sqlite3 databases and then all the normal stuff you would expect on a firewall. Your fw commands, environment variables and what not.

Another major difference is Gaia Embedded doesn't currently run on any x86/x64 cpu. As of right now it only runs on a ARM or MIPS CPU (that I know of). Meaning you can't just take an executable from say R77.20 Gaia and expect it to work on R77.20 for 1100.

First lets talk boot up. Gaia Embedded uses an image created via u-boot. This loads the kernel and the root file system, which is a rootfs (shocking!) and all the normal file systems.

Lets take a look! This is a small portion of /logs/boot_log. This provides a little hint of what is happening. Oh btw, this is a 1100 running R75.20.71.



So it looks like one partition has something related to boot loader environment, then we have kernel, root (-1 and -2). default_sw, logs, preset_cfg (maybe factory default boots here?), adsl (who uses that still?) and storage.

Lets compare to what we have running.


Looks like we found logs and storage. Maybe default_sfw is /pfrm2.0 (this is basically where most of the appliance lives).

So what else do we see? /tmp, /fwtmp,  /tmp/log/local are tmp file systems, meaning RAM based file systems. Technically virtual memory base file systems, but these boxes don't have a swap file so everything is for sure in RAM.

Now I want to point out there is no "/" in that listing. I'm pretty sure this is because / is a rootfs which is loaded by the kernel. Its kind of like a tmpfs, only it gets a list of files inserted into it before hand.

ok lets view /



Now.. something to notice. flash is a symbolic link (not a 3rd party app) to /pfrm2.0/config1. We'll come back to that.

Storage - this seems to be where online updates go? Not %100 sure. /logs its pretty much what it looks like. Logs..

Ok so lets talk black magic now...

Busybox. Busybox is a single application that will act differently based on how its called. You might be thinking, what do you mean by "how its called?". Let me show you with a shell script.



In this script i'm saying show me the first argument to this script and print it after the "is".

Guess what? There is also a $0, which is the name of the command (script in this case).



See how that changed? Using this logic a script with the exact same contents could act differently if there was a change for how it was called.

Ok so i changed the script and now we have:



Now lets make a copy of the script and call it hello.



Ok so we've proven we can change how something reacts based purely on its file name!

In comes Busybox! Busybox is a swiss army knife. Its a single binary that has a lot of programs built into it. This is done for massive disk space savings.



754k. So whats in there?




That is a lot of programs! So how does busybox know how to act? symbolic links!



So as you can see mv is a symbolic link to busybox.

You can feel free to poke around in there and see what else you can learn. Lets move on.

I started talking about the boot up process. Normally unix uses /etc/init.d/ stuff for booting. There are files in there but most of the heavy lifting is done in



This is where firewall kernel modules are loaded and all kinds of things happen.

If you need run a script at boot up, you'll need to create the following.



Ok, so what else can we talk about? Where do your configuration changes go that are made from clish or the webui?

Right here! ( /flash )



Notice a few things. shadow and passwd? these files are copied over to /etc on boot up or when changes are made via clish/webui.

The interesting one is system.db. This is a sqlite3 database. Want to read it? SURE!



Now you can view all the table schemes.. schema?.. whatever.. database output!

Something else interesting. Gaia Embedded on all platforms has a built in switch, fully managed switch!
I'm not going to dive into that right now, but you can split the ports and do basically anything a normal layer 3 switch would do. Cool stuff.

Now.. something odd i've noticed. If for some reason your doing dynamic routing on Gaia Embedded, keep this in mind when trouble shooting routing issues. If for some reason routed crashes it won't be restarted (well it depends on which routed process crashes but lets just say all of them crash). BUT!! If you login via cli and issue a show route, it will pause, then restart routed under the sheets, THEN show you the output.

This can be VERY confusing as it will look like all the sudden a issue fixed itself before you've had a chance to look at it. Want to see this in action? Setup a lab, get OSPF running. Kill routed then do a show route from clish.

Speaking of crashes!

When a process crashes on Gaia Embedded the kernel will use this sysctl to figure out how to generate the core file.



This means the core file will be piped into the shell script /pfrm2.0/bin/core_dump.sh.

Lets look at that shell script.



So... it pipes the core into cat and writes a file called /logs/core. I'm not following why they didn't just set kernel.core_pattern = /logs/core but sometimes its best to not ask questions. :)

Ok two things! You will only ever have a hit at while a single process crash and only the latest one because of this. That being said how do you know what process crashed? We only have a file called /logs/core.

We use the magic file command!

Lets tell sleep to go away most violently and check out core file. I'm going to tell sleep to sleep for 1000 seconds, then kill it with -6 (seg fault i think) %1 is the first job running in the background.



If you were trouble shooting something at this point I would say, create a cpinfo (cpinfo -o /logs/`hostname`.cpinfo.gz -z) and then download that core file also. If you aren't faint of heart I would also say do backtrace on said core file as well. You'll need gdb to do that. You can request it from checkpoint or use mine from the tools page. More on doing a backtrace later.

I'll update this with anything else i can think of, but for now thats all folks!

UPDATE: as of R77.20 HFA 20 checkpoint has added a scheduled backup option in the webui. Its under device -> system operations -> Periodic backup is OFF | Settings.. 

So I've had a fun journey going slightly insane trying to figure out why one backup method creates a meta string on the zip file and the other doesn't.

Well... so.... I made in important discovery. The zip file always had a meta header on it. Technically its called the comment field (--archive-comment). What happened? Well, I used unzip -l from cygwin and compared it to the output of unzip on Gaia Embedded. The unzip on Gaia Embedded doesn't print the meta header!! 

ARG! Well that was a waste. Note to self, saving a prompt would have been useful.


So.. down to details..

Here is the command to backup a centrally firewall managed.




And here is the command to backup a locally managed firewall.




Here is an option to backup without the policy.



Hurray! it works.. up next.. make this look less terrible. 

UPDATE: as of R77.20 HFA 20 checkpoint has added a scheduled backup option in the webui. Its under device -> system operations -> Periodic backup is OFF | Settings..

In the previous write up I showed how I think I found the backup command used in Gaia Embedded. How can we be sure this is the backup command? Well, this is what I did. I downloaded a backup via the webui then issued my backup command and compared the md5sums. Guess what I found? They're different! aaahh crap.

So how bad is it? File listing in the zip is the same. However, when I list the archive with zip -l I found this. The one from the webui has a meta section. The one created via clish (backup settings to tftp server 127.0.0.1) does not.

Check this out.., I've never seen this before (of course that means very little). This is on the top of the webui .zip:


Well... that is odd.. So the backup created via clish is different from the backup created by the webui on R75. Hopefully we haven't stumbled onto a bug that only effects R75 as R77 is out.

I'm going to go out on a limb and say the backup made via the webui is a better backup then from client.

Back to strace!

We know the webui runs on port 4434. Lets see what process is on that port:



Oh.. right.. no package.. sigh.. Well hopefully support doesn't see this. I uploaded lsof to /logs.

Boom! Now we know what process to strace.

This time I'm going to attach strace to a live process ( 910 ).
One thing I want to point out is I'm going to login to the webui and navigate all the way to the backup section first. Then I started the strace and hit the "Create Backup" button. I also did NOT download the file via the webui, so as to not pollute the strace output with all the stuff for the download . After the backup completed I hit CTRL-C on the strace. This is the full output of strace on the console:



Now... lets review our strace log.
Let go right to egrep -i backup /storage/thttpd.txt.




Ok, So the first line looks like it is us hitting the backup button, so the backup command has to be close to this line. However, I can't figure out what thttpd is doing using the unfilter log (not shown because of how long it is). Its like strace isn't seeing it. Its very possible strace has dorked something up and I need to reboot. I can't do that right now because the wife is watching Ill Tempered Masters of Tattooing on Hulu.

I think we're done for tonight.