Showing posts with label strace. Show all posts
Showing posts with label strace. Show all posts

Tuesday, April 12, 2016

Strace / Backsup / how magical strace is - Part 3 - The final!

UPDATE: as of R77.20 HFA 20 checkpoint has added a scheduled backup option in the webui. Its under device -> system operations -> Periodic backup is OFF | Settings.. 

So I've had a fun journey going slightly insane trying to figure out why one backup method creates a meta string on the zip file and the other doesn't.

Well... so.... I made in important discovery. The zip file always had a meta header on it. Technically its called the comment field (--archive-comment). What happened? Well, I used unzip -l from cygwin and compared it to the output of unzip on Gaia Embedded. The unzip on Gaia Embedded doesn't print the meta header!! 

ARG! Well that was a waste. Note to self, saving a prompt would have been useful.



So.. down to details..


Here is the command to backup a centrally firewall managed.



/pfrm2.0/bin/backup_settings.sh full pc "Making cron jobs stuff" admin


And here is the command to backup a locally managed firewall.



/pfrm2.0/bin/backup_settings.sh local_policy pc "Making cron jobs stuff" admin


Here is an option to backup without the policy.



/pfrm2.0/bin/backup_settings.sh  pc "Making cron jobs stuff" admin


I don't really understand the "pc" argument. Seems like it has control over where the backup gets stored locally. Not sure I see a point in changing it.

Right.. so I'm on a firewall with local policy.. SOOOOO...here is my current userScript. I moved crond down the list because it looks like if you make any crontab changes you'll need to restart crond. So to make everything work right we need to create all crontabs before starting cron on boot up.

So this crontab creates a backup every 5 mins. I did this because I was debugging and wanted to be able to show all the correct times. If you wanted to use this in production you would use a different time. If your not sure how to write a crontab this looks like a pretty good site. Crontab Examples

[Expert@FW]# cat /pfrm2.0/etc/userScript
ln -s /bin/busybox /bin/crond
mkdir -p /var/spool/cron/crontabs/
echo '*/5 * * * * /pfrm2.0/bin/backup_settings.sh local_policy pc "Making cron jobs stuff" admin' >> /var/spool/cron/crontabs/root
/bin/crond
[Expert@FW]#ls -l /storage/Gateway-ID-7F70949E_R75.20.71_983004120_2016-Apr-12-22_10_02.zip
-rw-r--r--    1 root     root      3089494 Apr 12 22:10 /storage/Gateway-ID-7F70949E_R75.20.71_983004120_2016-Apr-12-22_10_02.zip
[Expert@FW]# egrep -i cron /var/log/messages
2016 Apr 12 10:30:22 FW cron.notice crond[1832]: crond 1.8.1 started, log level 8
2016 Apr 12 22:02:23 FW cron.notice crond[7131]: crond 1.8.1 started, log level 8
2016 Apr 12 22:05:01 FW cron.notice crond[7131]: USER root pid 7147 cmd /pfrm2.0/bin/backup_settings.sh local_policy pc "Making cron jobs stuff" admin
2016 Apr 12 22:10:01 FW cron.notice crond[7131]: USER root pid 7290 cmd /pfrm2.0/bin/backup_settings.sh local_policy pc "Making cron jobs stuff" admin

Hurray! it works.. up next.. make this look less terrible. 

Saturday, April 9, 2016

Strace / Backsup / how magical strace is - Part 2

UPDATE: as of R77.20 HFA 20 checkpoint has added a scheduled backup option in the webui. Its under device -> system operations -> Periodic backup is OFF | Settings..

In the previous write up I showed how I think I found the backup command used in Gaia Embedded. How can we be sure this is the backup command? Well, this is what I did. I downloaded a backup via the webui then issued my backup command and compared the md5sums. Guess what I found? They're different! aaahh crap.

So how bad is it? File listing in the zip is the same. However, when I list the archive with zip -l I found this. The one from the webui has a meta section. The one created via clish (backup settings to tftp server 127.0.0.1) does not.

Check this out.., I've never seen this before (of course that means very little). This is on the top of the webui .zip:

Archive:  FW_R75.20.71_983004120_2016-Apr-09-21_11_10.zip
<meta_data_record>
<UID>MAC_HERE</UID>
<BoardModel>L50</BoardModel>
<Hostname>FW</Hostname>
<Version>R75.20.71_983004120</Version>
<Date>Apr 09, 2016 09:11:10 PM</Date>
<HasPolicy>2</HasPolicy>
<HasPassword>0</HasPassword>
<User>admin</User>
<Comment> </Comment>
</meta_data_record>
  Length      Date    Time    Name

Well... that is odd.. So the backup created via clish is different from the backup created by the webui on R75. Hopefully we haven't stumbled onto a bug that only effects R75 as R77 is out.

I'm going to go out on a limb and say the backup made via the webui is a better backup then from client.

Back to strace!

We know the webui runs on port 4434. Lets see what process is on that port:

[Expert@FW]# lsof -nni | grep 4434
-bash: lsof: command not found
[Expert@FW]#


Oh.. right.. no package.. sigh.. Well hopefully support doesn't see this. I uploaded lsof to /logs.

[Expert@FW]# tar -zxvf lsof_4.89.tgz
cnf/bin/lsof
[Expert@FW]# cd cnf/bin/
[Expert@FW]# ./lsof -nni | grep 4434
thttpd     910   root    1u  IPv4   3439      0t0  TCP *:4434 (LISTEN)
[Expert@FW]#

Boom! Now we know what process to strace.

This time I'm going to attach strace to a live process ( 910 ).
One thing I want to point out is I'm going to login to the webui and navigate all the way to the backup section first. Then I started the strace and hit the "Create Backup" button. I also did NOT download the file via the webui, so as to not pollute the strace output with all the stuff for the download . After the backup completed I hit CTRL-C on the strace. This is the full output of strace on the console:

[Expert@FW]# strace -s 1024 -f -p 910 -o /storage/thttpd.txt
strace: Process 910 attached
strace: Process 17366 attached
strace: Process 17367 attached
strace: Process 17368 attached
strace: Process 17369 attached
strace: Process 17491 attached
strace: Process 910 detached
[Expert@FW]#


Now... lets review our strace log.
Let go right to egrep -i backup /storage/thttpd.txt.

[Expert@FW]# egrep -i backup /storage/thttpd.txt
17367 send(0, "<31>Apr  9 21:42:11 thttpd[17367]: POST data: backup.full_backup=false&backup.comments=&backup.password=&button.create_backup=apply&thispage=lm_backupRestore\n", 158, MSG_NOSIGNAL) = 158
17367 write(1, "backup.full_backup=false&backup.comments=&backup.password=&button.create_backup=apply&thispage=lm_backupRestore", 111) = 111
17369 read(0, "backup.full_backup=false&backup.comments=&backup.password=&button.create_backup=apply&thispage=lm_backupRestore", 1024) = 111
[Expert@FW]#



Ok, So the first line looks like it is us hitting the backup button, so the backup command has to be close to this line. However, I can't figure out what thttpd is doing using the unfilter log (not shown because of how long it is). Its like strace isn't seeing it. Its very possible strace has dorked something up and I need to reboot. I can't do that right now because the wife is watching Ill Tempered Masters of Tattooing on Hulu.

I think we're done for tonight.


Thursday, April 7, 2016

Strace / Backsup / how magical strace is - Part 1

UPDATE: as of R77.20 HFA 20 checkpoint has added a scheduled backup option in the webui. Its under device -> system operations -> Periodic backup is OFF | Settings..

So... I made blog post about how to use a symbolic link to enable crond. I'll follow up this with a blog post about busybox based on the feed back I got from that posting.

I think this will be an interesting post. I'm going to point out i'm doing this on a live firewall.

so.. strace.. this tool does not ship on any checkpoint firewall. That being said its pretty easy to get it on Gaia. Just install CentOS 5.11, install it and copy over the strace binary. Gaia Embedded however.. thats a little more difficult. So if you check the tools page you'll find a download link for the 600, 1100, and 1200R. I should point out this is only for learning and most likely should not be used on a production firewall. Use this on a lab to learn how things really work because lets face it. If you don't know how something works when its working its much harder to understand why its broken if its broken.

So back to backsup. So you're thinking you spelled that wrong. I'll explain that later, it will make sense, just stick with me.

ok rock and roll. So we've already discussed how to enable crond so that we can schedule jobs on Gaia Embedded. The next step is to setup a backsup job. Lets see what options we have in clish and then try to figure out what command to use.



Well.. thats not very cool. TFTP or USB? TFTP?? what year is this? Should I hook up a serial connection to the encode the backup via uuencode and pipe the backsup into hylafax over slip (I don't really know if thats possible but i'm guessing so). Ok, joking aside...

lets just finish the backsup command and see what happens. Lets send it to 127.0.0.1.

[Expert@FW]# clish
FW> backup settings to ?
usb  - Save the backup file on a USB device
tftp - Send the backup file to a TFTP server
FW> backup settings to


ok, so what happened? the tftp timed out.. no surprise there. But what is interesting is the file name. Lets get out of clish (because I already set bashUser on in expert) and see if we can find that file.

FW> exit
Terminated
[Expert@FW]# find / -name FW_R75.20.71_983004120_2016-Apr-07-21_03_43.zip
/storage/FW_R75.20.71_983004120_2016-Apr-07-21_03_43.zip
[Expert@FW]#


Ok! Now we're cooking with gas! So we know we could stop here. We could add a cronjob that simply calls clish -c "backup settings to tftp server 127.0.0.1" and then upload said backup from there, but come on. We don't want to fire off a needless tftp command.

Here is where strace comes in, IN YOUR LAB!.. ehem...

Lets upload strace for the 600 / 1100 since this is a 1100 firewall. I've put strace in /logs/.

[Expert@FW]# ls -l strace-4.11.tgz
-rw-r--r--    1 root     root       293698 Apr  7 21:13 strace-4.11.tgz
[Expert@FW]# tar -zxvf strace-4.11.tgz
cnf/bin/strace
[Expert@FW]#


ok so strace installed.

strace has a lot of arguments. I should also point out it does some things that can cause problems on production system. I would only use this in a lab environment as it has a chance of crashing or causing unexpected bad things to happen. But this is our lab! so what do we care?

So full steam ahead!

[Expert@FW]# /logs/cnf/bin/strace -f -o /logs/strace-output.txt  -s 1024 clish -c "backup settings to tftp server 127.0.0.1"



This should complete without issue (assuming you're running in bash because you issued a bashUser on from expert, logged out and back in).

So the arguments are as follows.
-f == Mean follow any child processes. What is a child process? Well.. you can't do it with one process so many programs will create sub processes to handle small tasks and then return to the main process. This options means trace those child processes as well.
-o == This is where our strace output file will go. In this case /logs/strace.txt
-s == This is the max size of each line logged. So each line should be no longer then 1024 .. characters. might be a little excessive.
The final arguments are the command we want to trace, which in this case is clish -c "backup settings to tftp server 127.0.0.1"

You'll need to wait a little while. This eats a lot of cpu. After about 5-10 mins we have the following.


ok.. yeah.. -s 1024 was a bit much but we'll just work with what we have.

[Expert@FW]# ls -lh /logs/strace-output.txt
-rw-r--r--    1 root     root        15.4M Apr  7 21:28 /logs/strace-output.txt
[Expert@FW]#



So short cut. I know from playing with this before hand that we're looking for execve calls mostly.

[Expert@FW]# egrep '^[0-9]+ +execve' /logs/strace.txt > /logs/execve.txt
[Expert@FW]#


Ok so what do we have in /logs/execve.txt?

[Expert@FW]# wc -l /logs/execve.txt
      133 /logs/execve.txt
[Expert@FW]#



Not too bad.. Get to the good part.


577   execve("/bin/clish", ["clish", "-c", "backup settings to tftp server 127.0.0.1"], [/* 23 vars */]) = 0
579   execve("/usr/bin/id", ["id", "-u"], [/* 21 vars */]) = 0
581   execve("/pfrm2.0/bin/pt", ["/pfrm2.0/bin/pt", "--list"], [/* 22 vars */]) = 0
583   execve("/pfrm2.0/bin/lua", ["lua", "-e", "require ('cli.pt')('--list',  nil)"], [/* 21 vars */]) = 0
585   execve("/usr/bin/awk", ["awk", "-F:", "-v", "U=admin", "$1==U { print $7; exit; }", "/etc/passwd"], [/* 22 vars */]) = 0
587   execve("/usr/bin/tty", ["tty"], [/* 22 vars */]) = 0
588   execve("/pfrm2.0/bin/is_under_fw.sh", ["is_under_fw.sh", "577"], [/* 22 vars */]) = 0
589   execve("/pfrm2.0/bin/ppnames.sh", ["ppnames.sh", "577"], [/* 21 vars */]<unfinished ...>
590   execve("/bin/grep", ["grep", "^fw$"], [/* 21 vars */] <unfinished ...>
592   execve("/usr/bin/awk", ["awk", "$1==\"PPid:\" { print $2}"], [/* 21 vars*/] <unfinished ...>
593   execve("/bin/ps", ["ps", "--noheader", "-o", "comm", "575"], [/* 21 vars*/]) = 0
595   execve("/usr/bin/awk", ["awk", "$1==\"PPid:\" { print $2}"], [/* 21 vars*/]) = 0
596   execve("/bin/ps", ["ps", "--noheader", "-o", "comm", "32500"], [/* 21 vars */]) = 0
598   execve("/usr/bin/awk", ["awk", "$1==\"PPid:\" { print $2}"], [/* 21 vars*/]) = 0
599   execve("/bin/ps", ["ps", "--noheader", "-o", "comm", "32499"], [/* 21 vars */]) = 0
601   execve("/usr/bin/awk", ["awk", "$1==\"PPid:\" { print $2}"], [/* 21 vars*/]) = 0
602   execve("/bin/ps", ["ps", "--noheader", "-o", "comm", "906"], [/* 21 vars*/]) = 0
604   execve("/usr/bin/awk", ["awk", "$1==\"PPid:\" { print $2}"], [/* 21 vars*/]) = 0
605   execve("/bin/ps", ["ps", "--noheader", "-o", "comm", "1"], [/* 21 vars */]) = 0
607   execve("/usr/bin/awk", ["awk", "$1==\"PPid:\" { print $2}"], [/* 21 vars*/]) = 0
608   execve("/usr/bin/tty", ["tty"], [/* 23 vars */]) = 0
610   execve("/bin/grep", ["grep", "sfwsh\\.bin"], [/* 23 vars */] <unfinished...>
611   execve("/usr/bin/tty", ["tty"], [/* 23 vars */] <unfinished ...>
609   execve("/bin/ps", ["ps", "--noheaders", "-t", "/dev/pts/0"], [/* 23 vars*/]) = 0
613   execve("/pfrm2.0/bin/pt", ["pt", "users", "-f", "username", "admin", "-F", "role"], [/* 23 vars */] <unfinished ...>
614   execve("/usr/bin/head", ["head", "-n", "1"], [/* 23 vars */] <unfinished...>
615   execve("/bin/grep", ["grep", "-v", "{}"], [/* 23 vars */] <unfinished ...>
617   execve("/pfrm2.0/bin/lua", ["lua", "-e", "require ('cli.pt')('users', '-f', 'username', 'admin', '-F', 'role',  nil)"], [/* 22 vars */] <unfinished ...>
577   execve("/pfrm2.0/bin/sfwsh.bin", ["/pfrm2.0/bin/sfwsh.bin", "-c", "backup settings to tftp server 127.0.0.1"], [/* 25 vars */]) = 0
620   execve("/bin/sh", ["sh", "-c", "BKUP_TARGET=tftp backup_settings_cli.sh"], [/* 26 vars */]) = 0
620   execve("/pfrm2.0/bin/cli/backup_settings_cli.sh", ["backup_settings_cli.sh"], [/* 27 vars */]) = 0
623   execve("/bin/sh", ["sh", "-c", "export CPDIR=/opt/fw1 ; export FWDIR=/opt/fw1 ; PATH=/usr/sbin:/opt/fw1/bin:${PATH}; export PATH;/opt/fw1/bin//cpprod_util FwIsLocalMgmt 2>&1 ; echo RC=$?"], [/* 28 vars */] <unfinished ...>
624   execve("/opt/fw1/bin//cpprod_util", ["/opt/fw1/bin//cpprod_util", "FwIsLocalMgmt"], [/* 28 vars */]) = 0
634   execve("/bin/sh", ["sh", "-c", "/pfrm2.0/bin/backup_settings.sh local_policy pc \" \" admin \"\" \"\" 2>&1 ; echo RC=$?"], [/* 28 vars */] <unfinished ...>
635   execve("/pfrm2.0/bin/backup_settings.sh", ["/pfrm2.0/bin/backup_settings.sh", "local_policy", "pc", " ", "admin", "", ""], [/* 28 vars */]) = 0
638   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
639   execve("/bin/rm", ["/bin/rm", "-rf", "/fwtmp/backup_settings_status"], [/* 28 vars */]) = 0
641   execve("/bin/df", ["df", "/logs", "-m"], [/* 28 vars */] <unfinished ...>
642   execve("/usr/bin/tr", ["tr", "-s", " "], [/* 28 vars */] <unfinished ...>
643   execve("/usr/bin/cut", ["cut", "-f4", "-d "], [/* 28 vars */] <unfinished ...>
644   execve("/usr/bin/tail", ["tail", "-n", "1"], [/* 28 vars */] <unfinished...>
658   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
660   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
662   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
663   execve("/opt/fw1/bin/cp_write_syslog.sh", ["/opt/fw1/bin/cp_write_syslog.sh", "[System", "Operations]", "Starting", "settings", "backup", "process..."], [/* 28 vars */]) = 0
663   execve("/usr/bin/logger", ["logger", "-t", "CHECKPOINT", "-p", "info", "--", "[System Operations] Starting settings backup process..."], [/* 27 vars */]) = 0
665   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
667   execve("/usr/sbin/fw_printenv", ["/usr/sbin/fw_printenv", "-n", "activePartition"], [/* 28 vars */]) = 0
669   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
671   execve("/usr/sbin/fw_printenv", ["/usr/sbin/fw_printenv", "-n", "hw_mac_addr"], [/* 28 vars */]) = 0
673   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
675   execve("/usr/sbin/fw_printenv", ["/usr/sbin/fw_printenv", "-n", "unitModel"], [/* 28 vars */]) = 0
677   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
678   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
679   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
681   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
682   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
683   execve("/bin/rm", ["/bin/rm", "-rf", "/storage/*.zip"], [/* 28 vars */])= 0
693   execve("/usr/sbin/fw_printenv", ["/usr/sbin/fw_printenv", "-n", "activeConfig"], [/* 28 vars */]) = 0
694   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
695   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
696   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
697   execve("/bin/date", ["/bin/date", "+%b %d, %Y %r"], [/* 28 vars */]) = 0
698   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
699   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
700   execve("/bin/mkdir", ["/bin/mkdir", "-p", "/pfrm2.0/config1/addtional_settings_tmp"], [/* 29 vars */]) = 0
701   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
702   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
703   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
705   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
706   execve("/usr/bin/dirname", ["dirname", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/user.dhcpd.conf.*"], [/* 29 vars */]) = 0
707   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
708   execve("/bin/cp", ["/bin/cp", "-a", "///pfrm2.0/etc/user.dhcpd.conf.*", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/"], [/* 29 vars */]) = 0
709   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
710   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
711   execve("/usr/bin/dirname", ["dirname", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/opt/fw1/boot/modules/*.conf"], [/* 29 vars */]) = 0
712   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
713   execve("/bin/cp", ["/bin/cp", "-a", "///pfrm2.0/opt/fw1/boot/modules/*.conf", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/opt/fw1/boot/modules/"], [/* 29 vars */]) = 0
714   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
715   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
716   execve("/usr/bin/dirname", ["dirname", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/logging.config"], [/* 29 vars */]) = 0
717   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
718   execve("/bin/cp", ["/bin/cp", "-a", "///pfrm2.0/etc/logging.config", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/"], [/* 29 vars */]) = 0
719   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
720   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
721   execve("/usr/bin/dirname", ["dirname", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/userScript"], [/* 29 vars */]) = 0
722   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
723   execve("/bin/cp", ["/bin/cp", "-a", "///pfrm2.0/etc/userScript", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/"], [/* 29 vars */]) = 0
724   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
728   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
729   execve("/usr/bin/dirname", ["dirname", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/dropbear_rsa_host_key"], [/* 29 vars */]) = 0
730   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
731   execve("/bin/cp", ["/bin/cp", "-a", "///pfrm2.0/etc/dropbear_rsa_host_key", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/"], [/* 29 vars */]) = 0
732   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
733   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
734   execve("/usr/bin/dirname", ["dirname", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/webManifest"], [/* 29 vars */]) = 0
735   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
736   execve("/bin/cp", ["/bin/cp", "-a", "///pfrm2.0/etc/webManifest", "/pfrm2.0/config1/addtional_settings_tmp//pfrm2.0/etc/"], [/* 29 vars */]) = 0
737   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
738   execve("/bin/cp", ["/bin/cp", "-a", "/pfrm2.0/bin/restore_future_settings_hook.sh", "/pfrm2.0/config1"], [/* 29 vars */]) = 0
739   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
740   execve("/pfrm2.0/bin/firmTool", ["/pfrm2.0/bin/firmTool", "-c", "/dev/mtd4"], [/* 29 vars */]) = 0
743   execve("/usr/bin/cut", ["cut", "-d", "_", "-f", "1"], [/* 29 vars */]) =0
746   execve("/usr/bin/cut", ["cut", "-d", "_", "-f", "2"], [/* 29 vars */]) =0
749   execve("/usr/bin/cut", ["cut", "-d", "_", "-f", "3"], [/* 29 vars */]) =0
752   execve("/usr/bin/cut", ["cut", "-d", "_", "-f", "4"], [/* 29 vars */]) =0
753   execve("/bin/hostname", ["hostname"], [/* 29 vars */]) = 0
754   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
757   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
758   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
759   execve("/pfrm2.0/bin/zip", ["/pfrm2.0/bin/zip", "-ry", "/storage/settings_backup.zip", "ace", "addtional_settings_tmp", "expert_pass_", "fw1", "passwd","restore_future_settings_hook.sh", "shadow", "sofaware", "system.db", "tmp", "top_last_day_report.json", "top_last_hour_report.json", "restore_future_settings_hook.sh", "-x", "./fw1/state/local/FW1/*", "-x", "./sofaware/gui/logs.properties", "-qz"], [/* 29 vars */]) = 0
763   execve("/pfrm2.0/bin/unzip", ["/pfrm2.0/bin/unzip", "-qz", "/storage/settings_backup.zip"], [/* 29 vars */]) = 0
764   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
765   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
766   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
767   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
768   execve("/bin/mv", ["/bin/mv", "/storage/settings_backup.zip", "/storage/FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip"], [/* 29 vars */]) = 0
769   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
770   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
771   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 29 vars */]) = 0
772   execve("/bin/rm", ["/bin/rm", "-rf", "/pfrm2.0/config1/addtional_settings_tmp"], [/* 29 vars */]) = 0
773   execve("/bin/sh", ["sh", "-c", "cat /fwtmp/backup_file_location 2>&1 ; echo RC=$?"], [/* 28 vars */] <unfinished ...>
774   execve("/bin/cat", ["cat", "/fwtmp/backup_file_location"], [/* 28 vars */]) = 0
775   execve("/bin/sh", ["sh", "-c", "cat /fwtmp/backup_file_location 2>&1 ; echo RC=$?"], [/* 28 vars */] <unfinished ...>
776   execve("/bin/cat", ["cat", "/fwtmp/backup_file_location"], [/* 28 vars */]) = 0
777   execve("/bin/sh", ["sh", "-c", "echo `/bin/date +%Y-%b-%d-%T`: 'Uploading FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip to the TFTP server 127.0.0.1'>> /logs/backup_settings 2>&1 ; echo RC=$?"], [/* 28 vars */] <unfinished ...>
779   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0
780   execve("/bin/sh", ["sh", "-c", "ls /storage//FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip 2>&1 ; echo RC=$?"], [/* 28 vars */] <unfinished ...>
781   execve("/bin/ls", ["ls", "/storage//FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip"], [/* 28 vars */]) = 0
782   execve("/bin/sh", ["sh", "-c", "cd /storage/; tftp -pl FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip 127.0.0.1 2>&1 ; echo RC=$?"], [/* 28 vars */] <unfinished ...>
783   execve("/usr/bin/tftp", ["tftp", "-pl", "FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip", "127.0.0.1"], [/* 29 vars */]) = 0
784   execve("/bin/sh", ["sh", "-c", "echo `/bin/date +%Y-%b-%d-%T`: 'tftp: timeout' >> /logs/backup_settings 2>&1 ; echo RC=$?"], [/* 28 vars */]) = 0
786   execve("/bin/date", ["/bin/date", "+%Y-%b-%d-%T"], [/* 28 vars */]) = 0

Ok.. that was super long. So what are we looking at? These are sub processes created by the clish command!

So we're pretty sure our magic backsup command is somewhere in here because I'm guessing the sub system that is uploading the backsup to a tftp server is different from the process that creates the backsup.  

So the first few pages of lines seem like validation tests. 
Then we see this..

577   execve("/pfrm2.0/bin/sfwsh.bin", ["/pfrm2.0/bin/sfwsh.bin", "-c", "backup settings to tftp server 127.0.0.1"]



This looks like our clish command! We must be getting close now..

623   execve("/bin/sh", ["sh", "-c", "export CPDIR=/opt/fw1 ; export FWDIR=/opt/fw1 ; PATH=/usr/sbin:/opt/fw1/bin:${PATH}; export PATH;/opt/fw1/bin//cpprod_util FwIsLocalMgmt 2>&1 ; echo RC=$?"], [/* 28 vars */] <unfinished ...>
624   execve("/opt/fw1/bin//cpprod_util", ["/opt/fw1/bin//cpprod_util", "FwIsLocalMgmt"], [/* 28 vars */]) = 0


This looks like a check for local vs central management. hmm. so backup command maybe different base on how the firewall is managed. This firewall is locally managed FYI.

634   execve("/bin/sh", ["sh", "-c", "/pfrm2.0/bin/backup_settings.sh local_policy pc \" \" admin \"\" \"\" 2>&1 ; echo RC=$?"]


ok.. this is what we're looking for.

Lets back up.

[Expert@FW]# ls -lh /storage/FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip
-rw-r--r--    1 root     root         2.9M Apr  7 21:28 /storage/FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip
[Expert@FW]#


 So this is our backsup file created from the clish command to backup to tftp? Lets remove it first.

[Expert@FW]# rm /storage/FW_R75.20.71_983004120_2016-Apr-07-21_26_10.zip
[Expert@FW]#


Ok now.. that backsup command.

635   execve("/pfrm2.0/bin/backup_settings.sh", ["/pfrm2.0/bin/backup_settings.sh", "local_policy", "pc", " ", "admin", "", ""]


How do we run this from cli? Well each "," is showing each argument to /pfrm2.0/bin/backup_settings.sh.

ok ok ok.

So looks like is script local_polic pc " " admin "" ""
hmmm

let see what that does..

[Expert@FW]# /pfrm2.0/bin/backup_settings.sh local_policy pc " " admin "" ""
[Expert@FW]#


Well... something happened, that command took a few seconds to run. Do we have anything special in the /storage dir?

[Expert@FW]# ls -l /storage/
-rw-r--r--    1 root     root      3089185 Apr  7 22:27 FW_R75.20.71_983004120_2016-Apr-07-22_27_06.zip
drwxr-xr-x    2 root     root            0 Apr  5 21:30 lib
[Expert@FW]


oh nice! is that a valid backup? We'll have to do more testing to find out. Its getting a bit late and i'm running out of stream.




* I misspelled backup. See i told you everything would be explained!

Wednesday, April 6, 2016

Tools Tools Tools!

I know I said the next post was going to be on backups but I found out that one of my tools was broken (strace on the 600/1100) so I decided to write up a blog on tools. This is kind of rehash but I wanted to put this in a final resting place. I'll be updating this as other tools are compiled as well.

No 700 packages yet. - April/06/2016

Gaia Embedded I think is missing a few tools. I know, you're thinking just these 4? Well these were just kind of my starting point. I have others that I haven't posted but if someone wants something compiled I'll be more then happy to do so. I'll be releasing the compilers once i get everything packaged up nicely. I lost my source code for the 1200R compiler so i'll need to restart that mess again. (ugh).

All files in the tars are prefixed with cnf/ so no need to worry about trashing a local binary. They also don't require any libraries not already installed which is a big plus (so far!).

I would expand them under /logs/ or /mnt/sd if you have a external SD card formatted with EXT4 (stick to /logs if your using dos/vfat).

so for example. Upload the files to /logs/
then run

tar -zxvf FILENAME_HERE

Note: These packages are not supported by checkpoint (or anyone really). Also note using 3rd party apps are not supported on checkpoint firewalls.

dosfstools - this is for people running a 600/1100/1200R that have an external SD card installed. If you reboot (reboot command) these devices there is a good chance the dos/vfat file system on the SD will become corrupted at some point because checkpoint shutdown process is not so great. I would reformat with EXT4 if possible. If you need access to the data of a hosed dos file system and you're remote this gives you a way. Its a bit of a gamble. I mean there is no guarantee that repairing the file system will give you access to your data. 2nd option would be to dd off the unmounted SD card, upload to a remote server, mount via a loop and use a native dos file system checker.

strace - strace is a magical tool. You can watch what happens real time and catch error when debugs aren't useful. If you know what procmon on windows is, its basically the same thing. Basically you get to log almost everything a process does. Like, files it opens (or tries), libraries, external processes it might spawn. All that jazz. How many times have you thought, "how should xyz work normally" when something is broken in production but works in a lab? This will shed light. Next blog will be on how to use this so stay tuned!

gdb - if you find you have a core file (/logs/core) run file against it to find out what process crashed then use gdb to create a backtrace. This in addition with the core file and a cpinfo will help speed along any support ticket.

lsof - LiSt Open Files. this is another great tool for seeing all files/network connections/devices a giving process has open. Want to see all process that have a network connection open and not resolve ips? lsof -in

tail - I know, I know.. John.. everything has tail. Oh.. but does your tail include -F support? well.. this one does. If you know a -F is more then a grade a passive aggressive teacher gives you then this is for you!

600 / 1100 downloads

600 / 1100 dosfstools

600 / 1100 gdb

600 / 1100 lsof

600 / 1100 strace

600 / 1100 tail

1200R downloads

1200R dosfstools

1200R gdb

1200R lsof

1200R strace

1200R tail