Saturday, September 10, 2016

How to install Debian on Gaia Embedded - 700/1400 (not 1200R (ok and not 600/1100*))

UPDATE: Turns out this doesn't work on the 600/1100 (wah waaaah). Need some more testing (Yeah, I totally tested this on a 600/1100 before posting) to see if i can work around libc issue.

I recently... well maybe not that recently.. spent a few months working on building cross compilers that matched up %100 to a given Checkpoint Gaia Embedded system. Meaning, same libc (glibc 2.5, what a pain!), compiler version (based on glibc output) and kernel heads version.

I thought this was needed so that everything would be compatible. Well, turns out I made things way harder then it should have been. I recently found out that glibc is basically backwards compatible. There may be edge cases where things don't end up right, but for the most part, it seems pretty darn backwards compatible.

So that got me thinking. I started downloading .deb files and extracting them on my 750 and pretty much everything worked. Granted there was a lot of tracing library dependencies. So knowing that all worked I switched gears. I bought a 32 gig microsd card and installed it. I did format it to ext4 since vfat isn't a linux friendly file system. Side note: of course 7xx doesn't have the mkfs.ext4. Sigh... I'll have to map out all the needed libraries for that and point out the download links.

So the next idea was, can we just install debian on the 750 in a chroot environment? It turns out, yeah. I used the Debootstrap to create the chroot. It took a little while as it needs perl and wget and a few other things. The default wget on Gaia Embedded doesn't support https so just to be safe I pulled wget down also.

Before continuing, this is not supported by anyone. I would only do this on a test box, and not on a production firewall.

Basically I downloaded all these utilities on a spare Linux box in our Miami office:


I put them on a linux box and extracted them using this... somewhat nasty process:

for x in `ls *.deb` ; do ar xv $x ; tar -zxvf data.tar.gz ; tar -Jxvf data.tar.xz ; done

What I'm doing is expanding the .deb archive, which contains 3 or more files. The binaries are in a file called data.tar.gz (gziped) or data.tar.xz (lzma). I would have done this on Checkpoint Gaia Embedded but it doesn't include anything uncompress lzma. Kind of a brute force method to extract everything, but it worked. After that the raw files are ready to install on your Checkpoint firewall.

Next I just moved the files over to the Maimi Checkpoint firewall, so now I have this:

[Expert@FWCKP750]# ls -l
drwxr-xr-x 2 root root 4096 Sep 10 10:49 bin
drwxr-xr-x 5 root root 4096 Sep 10 10:13 etc
drwxr-xr-x 3 root root 4096 Sep 10 09:40 lib
drwxr-xr-x 2 root root 4096 Sep 10 09:33 sbin
drwxr-xr-x 7 root root 4096 Sep 10 09:47 usr
drwxr-xr-x 3 root root 4096 Sep 10 09:33 var
[Expert@FWCKP750]# pwd

Debootstrap is really just a shell script so once you have everything you can just run it. You also don't have to run it in Miami, I won't tell anyone if you do.

I did make a small script to setup library and path so the debootstrap files are used first. The last item was to tell debootstrap where its shell include files were.

I put this in

[Expert@FWCKP750]# pwd

[Expert@FWCKP750]# cat
declare -x DEBOOTSTRAP_DIR="/mnt/sd/debian/cnf/bootstrap/usr/share/debootstrap/"
declare -x LD_LIBRARY_PATH="/mnt/sd/debian/cnf/bootstrap/usr/lib/arm-linux-gnueabihf:/mnt/sd/debian/bootstrap/usr/lib:/mnt/sd/debian/bootstrap/lib/arm-linux-gnueabihf:/mnt/sd/debian/bootstrap/lib:.:/pfrm2.0/lib:/pfrm2.0/lib/iptables:"
declare -x PATH="/mnt/sd/cnf/debian/bootstrap/usr/bin:/mnt/sd/debian/bootstrap/usr/sbin:/mnt/sd/debian/bootstrap/sbin:/usr/local/bin:/usr/bin:/bin:/pfrm2.0/bin:/pfrm2.0/bin/cli:/pfrm2.0/bin/cli/provisioning:.:/usr/local/sbin:/usr/sbin:/sbin:/opt/fw1/bin"

This will suck in those settings for this login session on the Miami Checkpoint firewall.


That should be about all that is needed to run debootstrap. Next, just make the dir you want to install the OS into and run debootstrap.

mkdir /mnt/sd/stable-chroot

Then fire off debootstrap.

debootstrap --arch arghf stable /mnt/sd/stable-chroot

With luck and about 15 min you'll have a fully installed OS. We'll need a few little tweaks to wrap this up.

We need to mount proc and sysfs inside the chroot.

I added these statements to my userScript to handle this at bootup (yes the Miami Checkpoint firewall):

[Expert@FWCKP750]# ls -l userScript
-rwxr-xr-x 1 root root 120 Sep 10 12:01 userScript
[Expert@FWCKP750]# cat userScript
mount /dev/mmcblk1 /mnt/sd
mount proc /mnt/sd/stable-chroot/proc -t proc
mount sysfs /mnt/sd/stable-chroot/sys -t sysfs

You can just run those mount commands by hand also if you want. The mount /mnt/sd isn't
needed if the system is already up and running as it should auto mount. However,
the auto mount happens after userScript. Adding the mount to userScript is the workaround.

Now you're ready to jump in. Here I login to the chroot and then show python version
and perl version install.

[Expert@FWCKP750]# chroot /mnt/sd/stable-chroot bash -l
root@FWCKP750:/# echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin root@FWCKP750:/# cat /etc/debian_version 8.5
root@FWCKP750:/# python3 -V Python 3.4.2
root@FWCKP750:/# perl -v This is perl 5, version 20, subversion 2 (v5.20.2) built for arm-linux-gnueabihf-thread-multi-64int (with 81 registered patches, see perl -V for more detail) Copyright 1987-2015, Larry Wall Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5 source kit. Complete documentation for Perl, including FAQ lists, should be found on this system using "man perl" or "perldoc perl". If you have access to the Internet, point your browser at, the Perl Home Page. root@FWCKP750:/#

The install is about 500meg. Fits just great on a 32gig sd card, but is way too big without it.

I should point out things will work just fine inside the chroot. Once you logout, some things
will work outside of the chroot (/mnt/sd/stable-chroot/usr/bin/lsof for example), others you
may need to create a shell script to add library search and path statements, or in the case of
perl and python, do something to tell it where the modules are located.

I hope I didn't loose you at inside the chroot and outside the chroot. Chroot is a linux command that CHanges the ROOT dir.

So outside the chroot means the dir structure would look like this for example:

inside the chroot it would look like this.

Here is an example:
First I login to the chroot (now I'm inside)
[Expert@FWCKP750]# chroot /mnt/sd/stable-chroot bash -l
I create a file called TestFile
root@FWCKP750:/# touch TestFile
Notice how i'm in /
root@FWCKP750:/# pwd
And we see the TestFile
root@FWCKP750:/# ls
TestFile boot etc lib mnt proc run srv tmp var
bin dev home media opt root sbin sys usr
root@FWCKP750:/# exit
Now I logout. Notice out the directory changes? I'm now outside the chroot.
[Expert@FWCKP750]# pwd
[Expert@FWCKP750]# ls
TestFile dev lib opt run sys var
bin etc media proc sbin tmp
boot home mnt root srv usr
Hope that clears things up!

One interesting thing I noticed was the default ip utilties package on gaia embedded says
it doesn't support netns (Network Name Space (think VSX)), but using the Debian ip utility I was able to create a netns name. I haven't looked into this any further.

I'll have to run through the whole process again to make sure I documented it correctly.

Wait what? You would like a tar file of the debootstrap dir?

ok ok ok. Here you go.

debootstrap - 600 / 1100 / 700 / 1400

Thursday, September 8, 2016

Meetup group created!

Hi everyone, if you live in south Florida (or close enough) you should join the Meetup! The South Florida Checkpoint Zdebug Group - SFCZG for short! We're just getting started with the meetup. We're researching a location somewhere in Miami. Might expand to Fort Lauderdale at some point, but for now we'll be sticking to Miami for the Checkpoint meetup.

Join and drop a line!

Wednesday, July 13, 2016

Enhanced TCP logging options.

There is an interesting post on cpug about tcp logging options. You can get a lot more details on what happened to a tcp connection, such as when it expired, why it closed etc. Take a look!

TCP State Logging sk101221

Monday, July 4, 2016

4th of July Sheboygan Beer Brats

We're taking a moment away from Check Point firewall posts in Miami to describe how to make your very own firewalls... I mean Sheboygan Bier Brats. For those who don't know, my wife is from the German province of Wisconsin and this will cover how to make a proper brat!


Here are pics of everything.

Brats - out of package (yes warrenty voided)

This is really only 2... and it might have been white. The first one was yellow for sure!

Yes, the whole thing! Pretend you're German!

MUSTARD BEER MUG POWER!!! It's only 8 ounces btw.

BTW Bubbie's pickles are amazing. We're using Bubbie's Sauerkraut since we don't have a German Grandmother handy to make it for us. Heat and serve either on the side, or on the brat.

ok how to do this..

Hurl brats in a pot of ice water. This will help prevent bursting on the grill is what legends of old tell us. Do this for about 10 mins.

Important! The next 2 steps are done at the same time. Cook brats and heat up the da bier unions and butta.

Step 1 - cooking brats

Put your electric grill on medium low. You want to make sure the grill doesn't get too hot or the brats will burst. However, this should still be enough power to flip the breaker and kill power to the flat. 

This way, your Mobile Access Blade demo you're working on can go down hard and your wife can bring up (again!) how she has been saying for 3 weeks you need to get a UPS for your lab gear. You'll of course want to admit she is right, as always. Don't turn the servers back on, but reset the breaker.

If you can, don't use an electric grill! Charcoal is best.

Grill the brats for about 30 mins then turn them over and cook for another 30 min. This isn't a hard rule. The key is cook slow and make sure they're cooked. When you poke them, they should be firm and look like this just after turning.

Cooked, and ready to simmer in the butta and da bier!

While the brats are cooking put the da biers in the a pot with unions and da butta.

Step 2 Cooking the brine.

Put 9 x 12 oz of da biers, unions, butta in a pot on low. You don't want anything above a simmer.

Ok so let's assume steps 1 and 2 are complete. 

Ask the dog if she has found a job yet.

The lack of eye contact is not a good sign.

Patio lizard is not amused.

Use ray gun to increase dog's head to show your displeasure.

Now, take your brats and put them in the brine and let simmer for about an hour if you can stand it.

Now we didn't do this last step, but its pretty darn good. Do this if you have access to hard rolls (yes again they are soft and don't ask). Cut open the bun and only dip the open part of the bun in the brine, just enough to toast. Then grill the bun brine side down. This will take it to another level. We didn't do that this time.

Finished product. Yes, I know that's wine, we only had 3 da biers. We had no choice!

Wednesday, June 29, 2016

Gaia Embedded - includes a port scanner

Did you know Gaia Embedded includes a port scanner?

[Expert@FW1100]# /usr/bin/pscan
BusyBox v1.8.1 (2016-03-02 11:16:04 IST) multi-call binary

Usage: pscan [-p MIN_PORT] [-P MAX_PORT] [-t TIMEOUT] [-T MIN_RTT] HOST

Scan a host, print all open ports

        -p      Scan from this port (default 1)
        -P      Scan up to this port (default 1024)
        -t      Timeout (default 5000 ms)
        -T      Minimum rtt (default 5 ms, increase for congested hosts)


That's all for now. Night!

Friday, June 24, 2016

Check Point - Star VPN - HUB and Spoke VPN - To Center and Through Center of Miami

Hello everyone! Its time for another blog post from John <at> Spikefish Solutions. The last blog post contained a bit of silliness. Let me be Frank, that isn't going to stop. This write up will be a bit more technical. I ran across the need for a Hub and Spoke mode VPN with Check Point. I hit a few problems and had to get some advice from the local Check Point Diamond rep.

This guy was super knowledgeable and is a joy to work with. To protect his identity, we'll call him XML Smith.

That aside, let's make something clear. This write up is intended to be a very detailed write up on how to set up a Hub and Spoke VPN with a Check Point Firewall, which is located in Miami, FL, as the center gateway. This basically shows how Check Point Firewalls can act as a IPSec Proxy. What do I mean by IPSec Proxy?


What this write up is about:
The Firewalls on the outsides (Firewall A and Firewall C) don't know the remote networks don't exist on the Center Check Point Firewall. If you look closely you'll see Firewall A and Firewall C's encryption domains are not in Firewall B's encryption domain and yet we have Firewall A making a VPN to Firewall B and Firewall B making a VPN to Firewall C. What is really happening is there are 2 VPNs, but the Firewall A and Firewall C only know of a single VPN.

Something else interesting to point out is the Satellite gateways are Interoperability Devices (interop). In this demo they will be pfsense firewalls. I used this because I didn't have access to cisco devices and didn't feel like going GNS3 route. I also didn't want to use Check Point gateways to prove there was no black magic involved here.

Ok wait a sec..before moving on. The Check Point Diamond rep's name... I'm not a fan of XML. XML is a markup language that is hard for both humans and machines to parse. Let's call him JSON Smith instead. .. phew.. ok. That sounds much better. JSON Smith in Check Point Diamond is a great guy to have in your corner. Diamond is completely worth it, espically when you call them up and say come out here and upgrade P1 for me. :D

RIGHT! So let's talk lab overview. Engage network diagram (insert scifi sound here)!

Click to enlarge ( I hope ).

Network Overview

All clients: OpenBSD 5.9 (.100 address) - Default route is pointing to .1 btw
Interop Firewall A: Pfsense 2.3.1 - Release
Check Point Firewall B: Check Point R77.30 - Single Firewall + Management
Interop Firewall C: Pfsense 2.3.1 - Release
I should also point out all Firewalls have static routes for all networks listed.

I Created 5 Vlans for this (VL100 - VL 104). I used vlans off a single virtual switch for this lab. Also no devices are trunking.

The Orange cloud represents what would be the internet. The reason I'm pointing this out is it shows how you can even have a firewall on the internal network for this design.

All debugs have been started before passing traffic so that you can see everything.
This is all from the Check Point Firewall B in the Center of Miami.

  • TCPDUMP on eth0, eth1, eth2.
  • Firewall Monitor output
  • ike.elg - vpn debug ikeon
  • Export of all Tracker events during the passing of traffic.
Traffic for the vpn includes ping and SSH between VL100 and VL102 and a SSH attempt between VL100 and VL104.

Note: to view the ike.elg you need to download infoview from Check Point. If you don't have access call your local Check Point SE. I'm sure they'll be more then happy to get you a copy... or call Phoneboy, but make sure its like 3am in whatever timezone he is in. He loves that!

But wait, that isn't all! I also took backups of everything! This means you can setup this in a lab and quickly restore the backups if you want! Be warned the Checkpoint Firewall in Miami Beach (ok virtual Miami Beach) backup is 128Meg. Backup links and debug files will be located at end of post so scroll all the way down if that's what you're looking for.

Also, if you are restoring the checkpoint backup you will most likely need to install an eval as the license will have expired. You'll need a central all in one eval pointing to IP Again call your local Checkpoint SE and they will be happy to help with this if you don't have access to usercenter to generate your own eval keys.

One last quick note, the key to this config is the contents of encryption domain A and C cannot be located in encryption domain B. This is really the main configuration item to the setup and a little odd from a normal domain based VPN setup on Check point Firewall.

So let's start with the policy shall we? Here we can see ping and ssh is allowed bidirectionally from VL100 to VL102. We can also see ping is allow bidirectionally between VL100 and VL104 (this is the hub and spoke part). Nothing else is allowed. Clients are all running stock OpenBSD 5.9 and I didn't feel like setting up any other services beside SSH.

Next lets walk through the settings of Check Point Firewall B, which is our Hub and Star of our VPN. This is what you would see after double clicking Check Point Firewall B (Which is a Check Point Firewall if I didn't make that clear. I hear Check Point in Miami is a good deal also).

Here is the Topology view

Next is the IPSec VPN tab

Link Selection under that.

This is what is under the Setup button.

VPN Advanced

Firewall B encryption domain


Firewall A!

Firewall A Topology

Firewall A encryption domain


Firewall C - General

Firewall C topology

Firewall C encryption domain.


And here is the IPSec VPN tab configuration!

Center gateway and Satellite - Yes, the Check Point Firewall in Miami. Yes I know that's getting annoying, at least there aren't ads (cough cough yet)!

Encryption ( encryption + hashing).

And then under the Advanced tab - VPN Routing! This is where magic happens assuming you setup the encryption domains correctly. Everything else is default on the VPN Community.

Ok so that's about it for the Check Point VPN Hub and Spoke configuration examples. Everything else is default. If you want to see the other windows let me know and I'll add them, but I get the feeling there are a lot of screen shots in here already.

Lets look at logs. Notice the bounces? Those are the VPN Routing Icons that tell you magic is happening!



Note: I haven't tried restoring any of these so please let me know if there are problems.

all logins are as follows.

User: admin
Password: vpn123

No one who worked at checkpoint can use any other login for a lib install.

Sunday, June 19, 2016

Check Point 750 - see inside!

Hi everyone! This will be a quick write up covering physical aspects of a Check Point 750. And let's face it, you want to know what's on under the case right? Who wouldn't?!?!

I won't bother with packing material, but this is the box, just like all the other pics show.

Here is the front:

And here is the back:

6 x 10/100/1000 ports on a switch + WAN and DMZ ports.
802.11 b/g/n/ac MIMO 3x3, 1 radio band: 2.4Ghz/5Ghz
2 Console ports - 1 rj45 serial port and one usb port for console.

Here are all the cables it comes with:

Note: the black rectangle is a USB -> Mini USB Console Cable.
Basically, you can use this to console into the device if you for some reason
don't already own 3 to 5 USB -> 9pin serial converts.
You may need to download drivers from checkpoint to get it working.
I'll stick with my usb console cable because it has leds and blinky lights are fun!

Here is a picture with the Antenna properly installed. 

Why did it come with an extra one?

That might not be right, I'll have to do some research on that.

Now comes the hard part. How do we open this?
As it turns out, it's a highly complicated process involving up to two screws:

Ok screws are off. The top of the case should slide forward and lift off. Now we can see the sexy hardware on the inside!



.... Oh no! It's too sexy!

Ok, remove that last bit and this is what we have.
Here is a close up of the left side of the motherboard:

And here is the right:

Not much to see really. I want to take that heat sink off but the chicken is great with this one.

Oh, I should have listed what is on the inside.

ARMv7 Processor rev 4 (v7l) (Two cores)
1GB of ram
Not sure how big the storage is yet. I'll update with info.
Busybox 1.8.1
Linux Optix-700 3.10.20-al-5.0-pr2 (haven't check if this changed with R77.20 hfa 20).

I'll follow this up with some software info shortly.

Wednesday, June 15, 2016

An open letter to Check Point: App control and scada + offline updates

Dear Checkpoint

    Over the past 90 days I have been working on testing out checkpoint's scada protections in the 1200R firewall. After seeing what is possible with just modbus at CPX Chicago, I can hardly wait to see what else can be done!

There is just one problem. App control requires internet access.*

Checkpoint's solution for application control assumes the firewall (and / or the management server?) will have internet access. This presents a major problem for scada systems of which many do not have internet access. Ok maybe some of them do, but lets ignore those for now.

So list the issues that remain stalemated.

  1. 1200R doesn't seem to support offline updates without internet access. This is what i've been told but can't verify since I can't see to get the offline update package (yet). From what I understand it has something to do with verifying the contract over the internet.
  2. You have to sign a new EULA to get access to an offline update package. Once you have something happens (magic!) and then you gain access the package. The rumblings i'm hearing is its a completely manual process to install them to the firewall. I'm guessing its a tar file.
  3. Assuming issues 1 and 2 are resolved and worked into a process for updating, how will I know when a new package is out? The app wiki has no release dates on it. I also haven't found any place to get email alert about to signatures. Could be missing something here.
  4. bonus points, why isn't there a smartupdate package I can download?

I had no idea what an uphill battle this would become. I've been working with many people on this issue. I also don't want to diminish all the help I've received, but this is a major problem that remains unresolved.

So i guess after calling this an open letter I should wrap this up.

Check Point, come on, there has to be a way to resolve this. Lets find it and move on and start generating some really interesting reports!

Tuesday, May 10, 2016

My Linux mind blown and a really useful utility on Gaia.

Hi all!

I've been working on a proof of concept with Aruba Clearpass and checkpoint. Hasn't been going so well so the blog post that goes with it will have to wait. So what to talk about until then?

Well a while back I ran across a couple of things that kind of blew my mind so I thought I would pass them along. I'll also be discussing a really interesting utility I found. Stick with me or scroll to the end if you're that kind of person. I bet you read all the Walking Dead comic wikipedia entries didn't you?

So.. lets get the mind blown parts out of the way. I'm sure you know what a DLL is as most of the world knows windows. On Linux systems we have shared objects or libraries. These are mostly located in /lib and /usr/lib.

Now thats all fine and dandy. Who cares right?

Well, the core of library on linux is libc. This is basically where almost all programs start. Its bad if this goes away. Very bad.

So where am I going with this? Well I found out libraries are executable! Granted most of the time its just going to crash when you run it. BUT! libc can run just fine! Lets run it and see what happens.

[Expert@MGMT:0]# /lib/
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
Compiled by GNU CC version 4.1.2 20070626 (Red Hat 4.1.2-14).
Compiled on a Linux 2.6.9 system on 2015-01-29.
Available extensions:
        The C stubs add-on version 2.1.2.
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        GNU libio by Per Bothner
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        Native POSIX Threads Library by Ulrich Drepper et al
        RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see:

Wow.. who know all that info was in there?

So lets move on. Maybe you've seen people on checkpoint forms or else where say they can't run a command from cron but it works fine on the command line. Most of the time the fix is to source /etc/profile in your script, but do you know why this fixes it?

There are a lot of environment variables on a checkpoint login session. Here is what a normal environment looks like. No scatch that.. its too large.. cut to the chase. (run export on a normal session if you want to see) :)


This is basically a search path for libraries. Lets mess it up and see what happens.

[Expert@MGMT:0]# fw stat
Local host is not a FireWall-1 module
[Expert@MGMT:0]# unset LD_LIBRARY_PATH
[Expert@MGMT:0]# fw stat
fw: error while loading shared libraries: cannot open shared object file: No such file or directory

Notice how we're now getting a message saying it can't find 

So lets fix the environment again first.

[Expert@MGMT:0]# source /etc/profile
[Expert@MGMT:0]# fw stat
Local host is not a FireWall-1 module

Woot woot I say!

So here is where things get interesting. Lets pick on $FWDIR/bin/sendmail now. How can we tell what libraries sendmail is using?

[Expert@MGMT:0]# ldd $FWDIR/bin/sendmail =>  (0x00291000) => /opt/CPshrd-R77/lib/ (0x00ce6000) => /opt/CPshrd-R77/lib/ (0x00174000) => /opt/CPshrd-R77/lib/ (0x00a46000) => /lib/ (0x00769000) => /lib/ (0x003a7000) => /lib/ (0x00f07000) => /lib/ (0x0056b000) => /lib/ (0x00bf0000) => /lib/ (0x001d4000) => /usr/lib/ (0x00292000) => /lib/ (0x00d41000) => /opt/CPshrd-R77/lib/ (0x00110000)
        /lib/ (0x00370000) => /lib/ (0x00ed9000) => /lib/ (0x00198000)

ok one more unset.

[Expert@MGMT:0]# unset LD_LIBRARY_PATH
[Expert@MGMT:0]# /opt/CPsuite-R77/fw1/bin/sendmail
/opt/CPsuite-R77/fw1/bin/sendmail: error while loading shared libraries: cannot open shared object file: No such file or directory

How about that. Lets source /etc/profile and move on.

So the ldd program will print all the libraries a program uses. That's cool!.. OK I'm a nerd. :)

Game changer... 

wait.. what IS ldd? 

[Expert@MGMT:0]# file $(which ldd)
/usr/bin/ldd: Bourne shell script text executable

WHAT? Its a shell script? What is it running?

Take a look at it. The important part is this line.


Wait.. so ldd is running a library? mind blown again.. Let run it!

[Expert@MGMT:0]# /lib/
You have invoked `', the helper program for shared library executables.
This program usually lives in the file `/lib/', and special directives
in executable files using ELF shared libraries tell the system's program
loader to load the helper program from this file.  This helper program loads
the shared libraries needed by the program executable, prepares the program
to run, and runs it.  You may invoke this helper program directly from the
command line to load and run an ELF executable file; this is like executing
that file itself, but always uses this helper program from the file you
specified, instead of the helper program file specified in the executable
file you run.  This is mostly of use for maintainers to test new versions
of this helper program; chances are you did not intend to run this program.

  --list                list all dependencies and how they are resolved
  --verify              verify that given object really is a dynamically linked
                        object we can handle
  --library-path PATH   use given PATH instead of content of the environment
                        variable LD_LIBRARY_PATH
  --inhibit-rpath LIST  ignore RUNPATH and RPATH information in object names
                        in LIST

so...if you run across a linux system that doesn't have ldd, the shell script, installed you can use /lib/ (or whatever is in /lib). Too bad we don't have a system like that.

Oh but we do! Gaia Embedded!

[Expert@Gateway-ID]# which ldd
[Expert@Gateway-ID]# echo $PATH

Thanks Gaia Embedded! You sure save a lot of disk space keeping that shell script off the list of files to include. (Har!)

[Expert@Gateway-ID]# /lib/ --list $(which fw) => /lib/ (0x40001000) => /lib/ (0x4000c000) => /pfrm2.0/lib/ (0x4001b000) => /pfrm2.0/lib/ (0x40049000) => /pfrm2.0/lib/ (0x400cf000) => /pfrm2.0/lib/ (0x400fc000) => /pfrm2.0/lib/ (0x40108000) => /pfrm2.0/lib/ (0x40116000) => /pfrm2.0/lib/ (0x40124000) => /lib/ (0x40186000) => /lib/ (0x401a3000) => /lib/ (0x401c3000) => /pfrm2.0/lib/ (0x401dd000) => /lib/ (0x402b9000) => /pfrm2.0/lib/ (0x40368000) => /lib/ (0x4037d000)
        /lib/ (0x29aaa000)

Right well hopefully we all learned something there. 

Lets talk about this interesting command I found. First the issue.

We've all had this issue. Problem: Some process is putting data on the network. lsof is normally the goto (well unless your on Gaia Embedded (cough cough)) but here is the thing. What if the program is sending network data but you can't catch the process in the act? What if the process in question is firing off UDP packets and not leaving any traces?

Well here comes auditctl to the rescue. Real quick, I have Identity Awareness Captive Portal setup. Its currently pointing to Aruba Clearpass for Radius for user authentication. The issue? I don't know what process is firing off the radius request so I don't know what to debug! Keep in mind this is all lab. No idea how this effects production.

[Expert@MGMT:0]# auditctl -D
No rules
[Expert@MGMT:0]# auditctl -l
No rules

.. and so it begins..We'll be telling auditctl to log all socket calls (socket is the function used to write data on tcp/udp protocols).

[Expert@FWGW2:0]# auditctl -a exit,always -S socketcall
[Expert@FWGW2:0]# auditctl -l
LIST_RULES: exit,always syscall=socketcall

ok now, i'll login to the IA captive portal. The clearpass is on In hex that is "0A80020A". I don't understand all the information shown, but look at the comm= and exe= lines!!!

[Expert@FWGW2:0]# ausearch --host
 time->Tue May 10 20:02:29 2016
type=SOCKETCALL msg=audit(1462928549.402:1480): nargs=6 a0=8 a1=7fb5d2c0 a2=4000 a3=0 a4=7fb5d260 a5=7fb5d25c
type=SOCKADDR msg=audit(1462928549.402:1480): saddr=0200066D0A80020A0000000000000000
type=SYSCALL msg=audit(1462928549.402:1480): arch=40000003 syscall=102 success=yes exit=100 a0=c a1=7fb5d230 a2=1ad678 a3=0 items=0 ppid=5322 pid=2687 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="pdpd" exe="/opt/CPsuite-R77/fw1/bin/pdpd" subj=kernel key=(null)

woooooooow....I also fired off a telnet to on port 80. What did that find?

[Expert@FWGW2:0]# ausearch --host
time->Tue May 10 20:06:53 2016
type=SOCKETCALL msg=audit(1462928813.605:3494): nargs=3 a0=3 a1=9485d80 a2=10
type=SOCKADDR msg=audit(1462928813.605:3494): saddr=02000050080808080000000000000000
type=SYSCALL msg=audit(1462928813.605:3494): arch=40000003 syscall=102 success=no exit=-4 a0=3 a1=7fa76e30 a2=4964dc a3=9485d60 items=0 ppid=4876 pid=4920 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=tty1 ses=69 comm="telnet" exe="/usr/bin/telnet" subj=kernel key=(null)

.... : O ...

ok.. thats pretty amazing.

auditctl -D to delete that rule.

Go there for a read. Lots of stuff out there to peak around on.

Saturday, April 16, 2016

Gaia Embedded - How it works.

Hi everyone, I know you've been having these strange urges. You have these new feelings and you're not sure what to do about them. Everyone goes through this at one point. Its part of growing up. Meet me at camera three.

Ok, so we're here to talk about Gaia Embedded of course. Gaia Embedded is the OS that runs the SMB checkpoint firewalls. Its a combo of a uboot image, busybox, lua, sqlite3 databases and then all the normal stuff you would expect on a firewall. Your fw commands, environment variables and what not.

Another major difference is Gaia Embedded doesn't currently run on any x86/x64 cpu. As of right now it only runs on a ARM or MIPS CPU (that I know of). Meaning you can't just take an executable from say R77.20 Gaia and expect it to work on R77.20 for 1100.

First lets talk boot up. Gaia Embedded uses an image created via u-boot. This loads the kernel and the root file system, which is a rootfs (shocking!) and all the normal file systems.

Lets take a look! This is a small portion of /logs/boot_log. This provides a little hint of what is happening. Oh btw, this is a 1100 running R75.20.71.

Creating 11 MTD partitions on "nand_mtd":
0x00000000-0x000a0000 : "u-boot"
i2c driver was not initialized yet.
0x000a0000-0x00100000 : "bootldr-env"
0x00100000-0x00900000 : "kernel-1"
0x00900000-0x07a00000 : "rootfs-1"
0x07a00000-0x08200000 : "kernel-2"
0x08200000-0x0f300000 : "rootfs-2"
0x0f300000-0x16c00000 : "default_sw"
0x16c00000-0x18400000 : "logs"
0x18400000-0x18500000 : "preset_cfg"
0x18500000-0x18600000 : "adsl"
0x18600000-0x20000000 : "storage"

So it looks like one partition has something related to boot loader environment, then we have kernel, root (-1 and -2). default_sw, logs, preset_cfg (maybe factory default boots here?), adsl (who uses that still?) and storage.

Lets compare to what we have running.
[Expert@FW]# df -h
Filesystem                Size      Used Available Use% Mounted on
tmpfs                    20.0M    620.0k     19.4M   3% /tmp
tmpfs                    40.0M      7.4M     32.6M  18% /fwtmp
/dev/mtdblock7           24.0M      8.5M     15.5M  35% /logs
/dev/mtdblock10         122.0M     27.8M     94.2M  23% /storage
/dev/mtdblock5          113.0M     79.4M     33.6M  70% /pfrm2.0
tmpfs                    40.0M      1.1M     38.9M   3% /tmp/log/local

Looks like we found logs and storage. Maybe default_sfw is /pfrm2.0 (this is basically where most of the appliance lives).

So what else do we see? /tmp, /fwtmp,  /tmp/log/local are tmp file systems, meaning RAM based file systems. Technically virtual memory base file systems, but these boxes don't have a swap file so everything is for sure in RAM.

Now I want to point out there is no "/" in that listing. I'm pretty sure this is because / is a rootfs which is loaded by the kernel. Its kind of like a tmpfs, only it gets a list of files inserted into it before hand.

ok lets view /

[Expert@FW]# cd / ; ls -l
drwxr-xr-x    2 105      80              0 Apr 12 10:30 bin
lrwxrwxrwx    1 105      80              6 Dec 31  1969 data -> /flash
lrwxrwxrwx    1 root     root            8 Apr 13 05:20 dbg -> /tmp/dbg
drwxr-xr-x    5 5031     80              0 Apr 13 05:21 dev
drwxr-xr-x    7 105      80              0 Apr 15 07:07 etc
lrwxrwxrwx    1 root     root           16 Apr 13 05:20 flash -> /pfrm2.0/config1
drwxrwxrwt   12 root     root          860 Apr 16 11:20 fwtmp
lrwxrwxrwx    1 105      80             10 Dec 31  1969 init -> /sbin/init
drwxr-xr-x    3 105      80              0 Apr 13 05:21 lib
lrwxrwxrwx    1 105      80             11 Dec 31  1969 linuxrc -> bin/busybox
drwxr-xr-x    8 root     root            0 Apr 12 21:27 logs
drwxr-xr-x    8 105      80              0 Apr 13 05:20 mnt
lrwxrwxrwx    1 root     root           10 Apr 13 05:20 opt -> /fwtmp/opt
drwxr-xr-x   12 root     root            0 Dec 31  1969 pfrm2.0
dr-xr-xr-x   70 root     root            0 Dec 31  1969 proc
drwxr-xr-x    2 105      80              0 Apr 13 05:20 sbin
drwxr-xr-x    8 root     root            0 Apr 16 11:20 storage
drwxr-xr-x   10 root     root            0 Dec 31  1969 sys
drwxrwxrwt    4 root     root          320 Apr 16 11:20 tmp
drwxr-xr-x    2 root     root            0 Apr 13 05:21 usb
drwxr-xr-x    8 105      80              0 Apr 13 05:21 usr
drwxrwxrwx    8 105      80              0 Apr 16 10:46 var
drwxr-xr-x    2 root     root            0 Apr 13 05:21 web

Now.. something to notice. flash is a symbolic link (not a 3rd party app) to /pfrm2.0/config1. We'll come back to that.

Storage - this seems to be where online updates go? Not %100 sure. /logs its pretty much what it looks like. Logs..

Ok so lets talk black magic now...

Busybox. Busybox is a single application that will act differently based on how its called. You might be thinking, what do you mean by "how its called?". Let me show you with a shell script.

[Expert@FW]# echo "echo My Argument is \$1" > /tmp/
[Expert@FW]# cat /tmp/
echo My Argument is $1
[Expert@FW]# bash /tmp/ hello!
My Argument is hello!

In this script i'm saying show me the first argument to this script and print it after the "is".

Guess what? There is also a $0, which is the name of the command (script in this case).

[Expert@FW]# echo "echo My Argument is \$0" > /tmp/
[Expert@FW]# bash /tmp/ hello!
My Argument is /tmp/

See how that changed? Using this logic a script with the exact same contents could act differently if there was a change for how it was called.

Ok so i changed the script and now we have:

[Expert@FW]# cat /tmp/
if [ $0 == "hello" ] ; then
    echo My Argument is $1!
    echo "i don't know how $0 acts!"
[Expert@FW]# bash
i don't know how acts!

Now lets make a copy of the script and call it hello.

[Expert@FW]# cp hello
[Expert@FW]# bash hello howdy
My Argument is howdy!

Ok so we've proven we can change how something reacts based purely on its file name!

In comes Busybox! Busybox is a swiss army knife. Its a single binary that has a lot of programs built into it. This is done for massive disk space savings.

[Expert@FW]# ls -l /bin/busybox
-rwxr-xr-x    1 105      80         745216 Dec 31  1969 /bin/busybox

754k. So whats in there?

[Expert@FW]# /bin/busybox
BusyBox v1.8.1 (2015-04-26 16:47:09 IDT) multi-call binary
Copyright (C) 1998-2006 Erik Andersen, Rob Landley, and others.
Licensed under GPLv2. See source distribution for full notice.

Usage: busybox [function] [arguments]...
   or: [function] [arguments]...

        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as!

Currently defined functions:
        [, [[, addgroup, adduser, adjtimex, ar, arp, arping, ash,
        awk, basename, bunzip2, bzcat, bzip2, cal, cat, catv,
        chattr, chgrp, chmod, chown, chpasswd, chpst, chroot,
        chrt, chvt, cksum, clear, cmp, comm, cp, cpio, crond,
        crontab, cryptpw, cut, date, dc, dd, deallocvt, delgroup,
        deluser, df, dhcprelay, diff, dirname, dmesg, dnsd, dos2unix,
        du, dumpkmap, dumpleases, echo, ed, egrep, eject, env,
        envdir, envuidgid, ether-wake, expand, expr, fakeidentd,
        false, fbset, fdflush, fdformat, fdisk, fgrep, find, fold,
        free, freeramdisk, fsck, fsck.minix, ftpget, ftpput, fuser,
        getopt, getty, grep, gunzip, gzip, halt, hdparm, head,
        hexdump, hostid, hostname, httpd, hwclock, id, ifconfig,
        ifdown, ifup, inetd, init, insmod, install, ip, ipaddr,
        ipcalc, ipcrm, ipcs, iplink, iproute, iprule, iptunnel,
        kbd_mode, kill, killall, killall5, klogd, last, length,
        less, linux32, linux64, linuxrc, ln, loadfont, loadkmap,
        logger, login, logname, logread, losetup, ls, lsattr,
        lsmod, lzmacat, makedevs, md5sum, mdev, mesg, mkdir, mkfifo,
        mkfs.minix, mknod, mkswap, mktemp, modprobe, more, mount,
        mountpoint, mt, mv, nameif, netstat, nice, nmeter, nohup,
        nslookup, od, openvt, passwd, patch, pidof, ping, ping6,
        pipe_progress, pivot_root, poweroff, printenv, printf,
        pscan, pwd, raidautorun, rdate, readahead, readlink, readprofile,
        realpath, reboot, renice, reset, resize, rm, rmdir, rmmod,
        route, rpm, rpm2cpio, run-parts, runlevel, runsv, runsvdir,
        rx, sed, seq, setarch, setconsole, setkeycodes, setlogcons,
        setsid, setuidgid, sh, sha1sum, slattach, sleep, softlimit,
        sort, split, start-stop-daemon, stat, strings, stty, su,
        sulogin, sum, sv, svlogd, swapoff, swapon, switch_root,
        sync, sysctl, syslogd, tail, tar, taskset, tcpsvd, tee,
        telnet, telnetd, test, tftp, time, top, touch, tr, traceroute,
        true, tty, ttysize, udhcpc, udhcpd, udpsvd, umount, uname,
        uncompress, unexpand, uniq, unix2dos, unlzma, unzip, uptime,
        usleep, uudecode, uuencode, vconfig, vi, vlock, watch,
        watchdog, wc, wget, which, who, whoami, xargs, yes, zcat,


That is a lot of programs! So how does busybox know how to act? symbolic links!

[Expert@FW]# ls -l mv
lrwxrwxrwx    1 105      80              7 Dec 31  1969 mv -> busybox

So as you can see mv is a symbolic link to busybox.

You can feel free to poke around in there and see what else you can learn. Lets move on.

I started talking about the boot up process. Normally unix uses /etc/init.d/ stuff for booting. There are files in there but most of the heavy lifting is done in


This is where firewall kernel modules are loaded and all kinds of things happen.

If you need run a script at boot up, you'll need to create the following.


Ok, so what else can we talk about? Where do your configuration changes go that are made from clish or the webui?

Right here! ( /flash )

[Expert@FW]# ls -l
drwxr-xr-x    2 root     root            0 Dec 27 11:42 ace
-rw-r--r--    1 root     root           35 Dec 27 11:45 expert_pass_
drwxr-xr-x   10 root     root            0 Dec 27 11:42 fw1
-r--r--r--    1 root     root          373 Apr 12 10:28 passwd
-r-xr-xr-x    1 105      80            950 Sep  2  2015
-rw-------    1 root     root          255 Apr 12 10:28 shadow
drwxr-xr-x    4 root     root            0 Dec 27 11:43 sofaware
-rw-r--r--    1 root     root       760832 Apr 16 10:58 system.db
drwxr-xr-x    2 root     root            0 Dec 27 11:42 tmp
-rw-r--r--    1 root     root         1122 Dec 28 05:45 top_last_day_report.json
-rw-r--r--    1 root     root         1123 Dec 26 18:01 top_last_hour_report.json

Notice a few things. shadow and passwd? these files are copied over to /etc on boot up or when changes are made via clish/webui.

The interesting one is system.db. This is a sqlite3 database. Want to read it? SURE!

echo .dump | sqlite3 system.db > /logs/system-db.txt

Now you can view all the table schemes.. schema?.. whatever.. database output!

Something else interesting. Gaia Embedded on all platforms has a built in switch, fully managed switch!
I'm not going to dive into that right now, but you can split the ports and do basically anything a normal layer 3 switch would do. Cool stuff.

Now.. something odd i've noticed. If for some reason your doing dynamic routing on Gaia Embedded, keep this in mind when trouble shooting routing issues. If for some reason routed crashes it won't be restarted (well it depends on which routed process crashes but lets just say all of them crash). BUT!! If you login via cli and issue a show route, it will pause, then restart routed under the sheets, THEN show you the output.

This can be VERY confusing as it will look like all the sudden a issue fixed itself before you've had a chance to look at it. Want to see this in action? Setup a lab, get OSPF running. Kill routed then do a show route from clish.

Speaking of crashes!

When a process crashes on Gaia Embedded the kernel will use this sysctl to figure out how to generate the core file.

[Expert@FW]# sysctl kernel.core_pattern
kernel.core_pattern = |/pfrm2.0/bin/

This means the core file will be piped into the shell script /pfrm2.0/bin/

Lets look at that shell script.

[Expert@FW]# cat /pfrm2.0/bin/

cat > /logs/core

So... it pipes the core into cat and writes a file called /logs/core. I'm not following why they didn't just set kernel.core_pattern = /logs/core but sometimes its best to not ask questions. :)

Ok two things! You will only ever have a hit at while a single process crash and only the latest one because of this. That being said how do you know what process crashed? We only have a file called /logs/core.

We use the magic file command!

Lets tell sleep to go away most violently and check out core file. I'm going to tell sleep to sleep for 1000 seconds, then kill it with -6 (seg fault i think) %1 is the first job running in the background.

[Expert@FW]# sleep 1000 &
[1] 18008
[Expert@FW]# kill -6 %1
[1]+  Aborted                 (core dumped) sleep 1000

[Expert@FW]# ls -l /logs/core
-rw-r--r--    1 root     root       274432 Apr 16 11:56 /logs/core
[Expert@FW]# file /logs/core
/logs/core: ELF 32-bit LSB core file ARM, version 1 (SYSV), SVR4-style, from 'sleep'

If you were trouble shooting something at this point I would say, create a cpinfo (cpinfo -o /logs/`hostname`.cpinfo.gz -z) and then download that core file also. If you aren't faint of heart I would also say do backtrace on said core file as well. You'll need gdb to do that. You can request it from checkpoint or use mine from the tools page. More on doing a backtrace later.

I'll update this with anything else i can think of, but for now thats all folks!

Tuesday, April 12, 2016

Strace / Backsup / how magical strace is - Part 3 - The final!

UPDATE: as of R77.20 HFA 20 checkpoint has added a scheduled backup option in the webui. Its under device -> system operations -> Periodic backup is OFF | Settings.. 

So I've had a fun journey going slightly insane trying to figure out why one backup method creates a meta string on the zip file and the other doesn't.

Well... so.... I made in important discovery. The zip file always had a meta header on it. Technically its called the comment field (--archive-comment). What happened? Well, I used unzip -l from cygwin and compared it to the output of unzip on Gaia Embedded. The unzip on Gaia Embedded doesn't print the meta header!! 

ARG! Well that was a waste. Note to self, saving a prompt would have been useful.

So.. down to details..

Here is the command to backup a centrally firewall managed.

/pfrm2.0/bin/ full pc "Making cron jobs stuff" admin

And here is the command to backup a locally managed firewall.

/pfrm2.0/bin/ local_policy pc "Making cron jobs stuff" admin

Here is an option to backup without the policy.

/pfrm2.0/bin/  pc "Making cron jobs stuff" admin

I don't really understand the "pc" argument. Seems like it has control over where the backup gets stored locally. Not sure I see a point in changing it.

Right.. so I'm on a firewall with local policy.. is my current userScript. I moved crond down the list because it looks like if you make any crontab changes you'll need to restart crond. So to make everything work right we need to create all crontabs before starting cron on boot up.

So this crontab creates a backup every 5 mins. I did this because I was debugging and wanted to be able to show all the correct times. If you wanted to use this in production you would use a different time. If your not sure how to write a crontab this looks like a pretty good site. Crontab Examples

[Expert@FW]# cat /pfrm2.0/etc/userScript
ln -s /bin/busybox /bin/crond
mkdir -p /var/spool/cron/crontabs/
echo '*/5 * * * * /pfrm2.0/bin/ local_policy pc "Making cron jobs stuff" admin' >> /var/spool/cron/crontabs/root
[Expert@FW]#ls -l /storage/
-rw-r--r--    1 root     root      3089494 Apr 12 22:10 /storage/
[Expert@FW]# egrep -i cron /var/log/messages
2016 Apr 12 10:30:22 FW cron.notice crond[1832]: crond 1.8.1 started, log level 8
2016 Apr 12 22:02:23 FW cron.notice crond[7131]: crond 1.8.1 started, log level 8
2016 Apr 12 22:05:01 FW cron.notice crond[7131]: USER root pid 7147 cmd /pfrm2.0/bin/ local_policy pc "Making cron jobs stuff" admin
2016 Apr 12 22:10:01 FW cron.notice crond[7131]: USER root pid 7290 cmd /pfrm2.0/bin/ local_policy pc "Making cron jobs stuff" admin

Hurray! it works.. up next.. make this look less terrible. 

Saturday, April 9, 2016

Strace / Backsup / how magical strace is - Part 2

UPDATE: as of R77.20 HFA 20 checkpoint has added a scheduled backup option in the webui. Its under device -> system operations -> Periodic backup is OFF | Settings..

In the previous write up I showed how I think I found the backup command used in Gaia Embedded. How can we be sure this is the backup command? Well, this is what I did. I downloaded a backup via the webui then issued my backup command and compared the md5sums. Guess what I found? They're different! aaahh crap.

So how bad is it? File listing in the zip is the same. However, when I list the archive with zip -l I found this. The one from the webui has a meta section. The one created via clish (backup settings to tftp server does not.

Check this out.., I've never seen this before (of course that means very little). This is on the top of the webui .zip:

<Date>Apr 09, 2016 09:11:10 PM</Date>
<Comment> </Comment>
  Length      Date    Time    Name

Well... that is odd.. So the backup created via clish is different from the backup created by the webui on R75. Hopefully we haven't stumbled onto a bug that only effects R75 as R77 is out.

I'm going to go out on a limb and say the backup made via the webui is a better backup then from client.

Back to strace!

We know the webui runs on port 4434. Lets see what process is on that port:

[Expert@FW]# lsof -nni | grep 4434
-bash: lsof: command not found

Oh.. right.. no package.. sigh.. Well hopefully support doesn't see this. I uploaded lsof to /logs.

[Expert@FW]# tar -zxvf lsof_4.89.tgz
[Expert@FW]# cd cnf/bin/
[Expert@FW]# ./lsof -nni | grep 4434
thttpd     910   root    1u  IPv4   3439      0t0  TCP *:4434 (LISTEN)

Boom! Now we know what process to strace.

This time I'm going to attach strace to a live process ( 910 ).
One thing I want to point out is I'm going to login to the webui and navigate all the way to the backup section first. Then I started the strace and hit the "Create Backup" button. I also did NOT download the file via the webui, so as to not pollute the strace output with all the stuff for the download . After the backup completed I hit CTRL-C on the strace. This is the full output of strace on the console:

[Expert@FW]# strace -s 1024 -f -p 910 -o /storage/thttpd.txt
strace: Process 910 attached
strace: Process 17366 attached
strace: Process 17367 attached
strace: Process 17368 attached
strace: Process 17369 attached
strace: Process 17491 attached
strace: Process 910 detached

Now... lets review our strace log.
Let go right to egrep -i backup /storage/thttpd.txt.

[Expert@FW]# egrep -i backup /storage/thttpd.txt
17367 send(0, "<31>Apr  9 21:42:11 thttpd[17367]: POST data: backup.full_backup=false&backup.comments=&backup.password=&button.create_backup=apply&thispage=lm_backupRestore\n", 158, MSG_NOSIGNAL) = 158
17367 write(1, "backup.full_backup=false&backup.comments=&backup.password=&button.create_backup=apply&thispage=lm_backupRestore", 111) = 111
17369 read(0, "backup.full_backup=false&backup.comments=&backup.password=&button.create_backup=apply&thispage=lm_backupRestore", 1024) = 111

Ok, So the first line looks like it is us hitting the backup button, so the backup command has to be close to this line. However, I can't figure out what thttpd is doing using the unfilter log (not shown because of how long it is). Its like strace isn't seeing it. Its very possible strace has dorked something up and I need to reboot. I can't do that right now because the wife is watching Ill Tempered Masters of Tattooing on Hulu.

I think we're done for tonight.