Spikefish Solutions Blog

Hi everyone, I know you've been having these strange urges. You have these new feelings and you're not sure what to do about them. Everyone goes through this at one point. Its part of growing up. Meet me at camera three.

Ok, so we're here to talk about Gaia Embedded of course. Gaia Embedded is the OS that runs the SMB checkpoint firewalls. Its a combo of a uboot image, busybox, lua, sqlite3 databases and then all the normal stuff you would expect on a firewall. Your fw commands, environment variables and what not.

Another major difference is Gaia Embedded doesn't currently run on any x86/x64 cpu. As of right now it only runs on a ARM or MIPS CPU (that I know of). Meaning you can't just take an executable from say R77.20 Gaia and expect it to work on R77.20 for 1100.

First lets talk boot up. Gaia Embedded uses an image created via u-boot. This loads the kernel and the root file system, which is a rootfs (shocking!) and all the normal file systems.

Lets take a look! This is a small portion of /logs/boot_log. This provides a little hint of what is happening. Oh btw, this is a 1100 running R75.20.71.



So it looks like one partition has something related to boot loader environment, then we have kernel, root (-1 and -2). default_sw, logs, preset_cfg (maybe factory default boots here?), adsl (who uses that still?) and storage.

Lets compare to what we have running.


Looks like we found logs and storage. Maybe default_sfw is /pfrm2.0 (this is basically where most of the appliance lives).

So what else do we see? /tmp, /fwtmp,  /tmp/log/local are tmp file systems, meaning RAM based file systems. Technically virtual memory base file systems, but these boxes don't have a swap file so everything is for sure in RAM.

Now I want to point out there is no "/" in that listing. I'm pretty sure this is because / is a rootfs which is loaded by the kernel. Its kind of like a tmpfs, only it gets a list of files inserted into it before hand.

ok lets view /



Now.. something to notice. flash is a symbolic link (not a 3rd party app) to /pfrm2.0/config1. We'll come back to that.

Storage - this seems to be where online updates go? Not %100 sure. /logs its pretty much what it looks like. Logs..

Ok so lets talk black magic now...

Busybox. Busybox is a single application that will act differently based on how its called. You might be thinking, what do you mean by "how its called?". Let me show you with a shell script.



In this script i'm saying show me the first argument to this script and print it after the "is".

Guess what? There is also a $0, which is the name of the command (script in this case).



See how that changed? Using this logic a script with the exact same contents could act differently if there was a change for how it was called.

Ok so i changed the script and now we have:



Now lets make a copy of the script and call it hello.



Ok so we've proven we can change how something reacts based purely on its file name!

In comes Busybox! Busybox is a swiss army knife. Its a single binary that has a lot of programs built into it. This is done for massive disk space savings.



754k. So whats in there?




That is a lot of programs! So how does busybox know how to act? symbolic links!



So as you can see mv is a symbolic link to busybox.

You can feel free to poke around in there and see what else you can learn. Lets move on.

I started talking about the boot up process. Normally unix uses /etc/init.d/ stuff for booting. There are files in there but most of the heavy lifting is done in



This is where firewall kernel modules are loaded and all kinds of things happen.

If you need run a script at boot up, you'll need to create the following.



Ok, so what else can we talk about? Where do your configuration changes go that are made from clish or the webui?

Right here! ( /flash )



Notice a few things. shadow and passwd? these files are copied over to /etc on boot up or when changes are made via clish/webui.

The interesting one is system.db. This is a sqlite3 database. Want to read it? SURE!



Now you can view all the table schemes.. schema?.. whatever.. database output!

Something else interesting. Gaia Embedded on all platforms has a built in switch, fully managed switch!
I'm not going to dive into that right now, but you can split the ports and do basically anything a normal layer 3 switch would do. Cool stuff.

Now.. something odd i've noticed. If for some reason your doing dynamic routing on Gaia Embedded, keep this in mind when trouble shooting routing issues. If for some reason routed crashes it won't be restarted (well it depends on which routed process crashes but lets just say all of them crash). BUT!! If you login via cli and issue a show route, it will pause, then restart routed under the sheets, THEN show you the output.

This can be VERY confusing as it will look like all the sudden a issue fixed itself before you've had a chance to look at it. Want to see this in action? Setup a lab, get OSPF running. Kill routed then do a show route from clish.

Speaking of crashes!

When a process crashes on Gaia Embedded the kernel will use this sysctl to figure out how to generate the core file.



This means the core file will be piped into the shell script /pfrm2.0/bin/core_dump.sh.

Lets look at that shell script.



So... it pipes the core into cat and writes a file called /logs/core. I'm not following why they didn't just set kernel.core_pattern = /logs/core but sometimes its best to not ask questions. :)

Ok two things! You will only ever have a hit at while a single process crash and only the latest one because of this. That being said how do you know what process crashed? We only have a file called /logs/core.

We use the magic file command!

Lets tell sleep to go away most violently and check out core file. I'm going to tell sleep to sleep for 1000 seconds, then kill it with -6 (seg fault i think) %1 is the first job running in the background.



If you were trouble shooting something at this point I would say, create a cpinfo (cpinfo -o /logs/`hostname`.cpinfo.gz -z) and then download that core file also. If you aren't faint of heart I would also say do backtrace on said core file as well. You'll need gdb to do that. You can request it from checkpoint or use mine from the tools page. More on doing a backtrace later.

I'll update this with anything else i can think of, but for now thats all folks!