I've been working on a proof of concept with Aruba Clearpass and checkpoint. Hasn't been going so well so the blog post that goes with it will have to wait. So what to talk about until then?
Well a while back I ran across a couple of things that kind of blew my mind so I thought I would pass them along. I'll also be discussing a really interesting utility I found. Stick with me or scroll to the end if you're that kind of person. I bet you read all the Walking Dead comic wikipedia entries didn't you?
So.. lets get the mind blown parts out of the way. I'm sure you know what a DLL is as most of the world knows windows. On Linux systems we have shared objects or libraries. These are mostly located in /lib and /usr/lib.
Now thats all fine and dandy. Who cares right?
Well, the core of library on linux is libc. This is basically where almost all programs start. Its bad if this goes away. Very bad.
So where am I going with this? Well I found out libraries are executable! Granted most of the time its just going to crash when you run it. BUT! libc can run just fine! Lets run it and see what happens.
[Expert@MGMT:0]# /lib/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20070626 (Red Hat 4.1.2-14).
Compiled on a Linux 2.6.9 system on 2015-01-29.
Available extensions:
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
GNU libio by Per Bothner
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
[Expert@MGMT:0]#
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20070626 (Red Hat 4.1.2-14).
Compiled on a Linux 2.6.9 system on 2015-01-29.
Available extensions:
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
GNU libio by Per Bothner
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
[Expert@MGMT:0]#
Wow.. who know all that info was in there?
So lets move on. Maybe you've seen people on checkpoint forms or else where say they can't run a command from cron but it works fine on the command line. Most of the time the fix is to source /etc/profile in your script, but do you know why this fixes it?
There are a lot of environment variables on a checkpoint login session. Here is what a normal environment looks like. No scatch that.. its too large.. cut to the chase. (run export on a normal session if you want to see) :)
LD_LIBRARY_PATH
This is basically a search path for libraries. Lets mess it up and see what happens.
[Expert@MGMT:0]# fw stat
Local host is not a FireWall-1 module
[Expert@MGMT:0]# unset LD_LIBRARY_PATH
[Expert@MGMT:0]# fw stat
fw: error while loading shared libraries: libfw1_light.so: cannot open shared object file: No such file or directory
[Expert@MGMT:0]#
Notice how we're now getting a message saying it can't find libfw1_light.so?
So lets fix the environment again first.
[Expert@MGMT:0]# source /etc/profile
[Expert@MGMT:0]# fw stat
Local host is not a FireWall-1 module
[Expert@MGMT:0]#
Woot woot I say!
So here is where things get interesting. Lets pick on $FWDIR/bin/sendmail now. How can we tell what libraries sendmail is using?
[Expert@MGMT:0]# ldd $FWDIR/bin/sendmail
linux-gate.so.1 => (0x00291000)
libcpprod50.so => /opt/CPshrd-R77/lib/libcpprod50.so (0x00ce6000)
libProdUtils.so => /opt/CPshrd-R77/lib/libProdUtils.so (0x00174000)
libOS.so => /opt/CPshrd-R77/lib/libOS.so (0x00a46000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00769000)
libresolv.so.2 => /lib/libresolv.so.2 (0x003a7000)
libdl.so.2 => /lib/libdl.so.2 (0x00f07000)
libpam.so.0 => /lib/libpam.so.0 (0x0056b000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00bf0000)
libm.so.6 => /lib/libm.so.6 (0x001d4000)
libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x00292000)
libc.so.6 => /lib/libc.so.6 (0x00d41000)
libDataStruct.so => /opt/CPshrd-R77/lib/libDataStruct.so (0x00110000)
/lib/ld-linux.so.2 (0x00370000)
libaudit.so.0 => /lib/libaudit.so.0 (0x00ed9000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00198000)
[Expert@MGMT:0]#
ok one more unset.
[Expert@MGMT:0]# unset LD_LIBRARY_PATH
[Expert@MGMT:0]# /opt/CPsuite-R77/fw1/bin/sendmail
/opt/CPsuite-R77/fw1/bin/sendmail: error while loading shared libraries: libProdUtils.so: cannot open shared object file: No such file or directory
[Expert@MGMT:0]#
How about that. Lets source /etc/profile and move on.
So the ldd program will print all the libraries a program uses. That's cool!.. OK I'm a nerd. :)
Game changer...
wait.. what IS ldd?
[Expert@MGMT:0]# file $(which ldd)
/usr/bin/ldd: Bourne shell script text executable
[Expert@MGMT:0]#
WHAT? Its a shell script? What is it running?
Take a look at it. The important part is this line.
RTLDLIST=/lib/ld-linux.so.2
Wait.. so ldd is running a library? mind blown again.. Let run it!
[Expert@MGMT:0]# /lib/ld-linux.so.2
Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked `ld.so', the helper program for shared library executables.
This program usually lives in the file `/lib/ld.so', and special directives
in executable files using ELF shared libraries tell the system's program
loader to load the helper program from this file. This helper program loads
the shared libraries needed by the program executable, prepares the program
to run, and runs it. You may invoke this helper program directly from the
command line to load and run an ELF executable file; this is like executing
that file itself, but always uses this helper program from the file you
specified, instead of the helper program file specified in the executable
file you run. This is mostly of use for maintainers to test new versions
of this helper program; chances are you did not intend to run this program.
--list list all dependencies and how they are resolved
--verify verify that given object really is a dynamically linked
object we can handle
--library-path PATH use given PATH instead of content of the environment
variable LD_LIBRARY_PATH
--inhibit-rpath LIST ignore RUNPATH and RPATH information in object names
in LIST
[Expert@MGMT:0]#
so...if you run across a linux system that doesn't have ldd, the shell script, installed you can use /lib/ld-linux.so.2 (or whatever ld-linux.so is in /lib). Too bad we don't have a system like that.
Oh but we do! Gaia Embedded!
[Expert@Gateway-ID]# which ldd
[Expert@Gateway-ID]# echo $PATH
/usr/local/bin:/usr/bin:/bin:/pfrm2.0/bin:/pfrm2.0/bin/cli:/pfrm2.0/bin/cli/provisioning:.:/usr/local/sbin:/usr/sbin:/sbin:/opt/fw1/bin
[Expert@Gateway-ID]#
Thanks Gaia Embedded! You sure save a lot of disk space keeping that shell script off the list of files to include. (Har!)
[Expert@Gateway-ID]# /lib/ld-linux.so.3 --list $(which fw)
libdl.so.2 => /lib/libdl.so.2 (0x40001000)
librt.so.1 => /lib/librt.so.1 (0x4000c000)
libgssapi_krb5.so.2 => /pfrm2.0/lib/libgssapi_krb5.so.2 (0x4001b000)
libkrb5.so.3 => /pfrm2.0/lib/libkrb5.so.3 (0x40049000)
libk5crypto.so.3 => /pfrm2.0/lib/libk5crypto.so.3 (0x400cf000)
libcom_err.so.3 => /pfrm2.0/lib/libcom_err.so.3 (0x400fc000)
libkrb5support.so.0 => /pfrm2.0/lib/libkrb5support.so.0 (0x40108000)
libyajl.so.1 => /pfrm2.0/lib/libyajl.so.1 (0x40116000)
libluajit-5.1.so.2 => /pfrm2.0/lib/libluajit-5.1.so.2 (0x40124000)
libnsl.so.1 => /lib/libnsl.so.1 (0x40186000)
libpthread.so.0 => /lib/libpthread.so.0 (0x401a3000)
libresolv.so.2 => /lib/libresolv.so.2 (0x401c3000)
libstdc++.so.6 => /pfrm2.0/lib/libstdc++.so.6 (0x401dd000)
libm.so.6 => /lib/libm.so.6 (0x402b9000)
libgcc_s.so.1 => /pfrm2.0/lib/libgcc_s.so.1 (0x40368000)
libc.so.6 => /lib/libc.so.6 (0x4037d000)
/lib/ld-linux.so.3 (0x29aaa000)
[Expert@Gateway-ID]#
Right well hopefully we all learned something there.
Lets talk about this interesting command I found. First the issue.
We've all had this issue. Problem: Some process is putting data on the network. lsof is normally the goto (well unless your on Gaia Embedded (cough cough)) but here is the thing. What if the program is sending network data but you can't catch the process in the act? What if the process in question is firing off UDP packets and not leaving any traces?
Well here comes auditctl to the rescue. Real quick, I have Identity Awareness Captive Portal setup. Its currently pointing to Aruba Clearpass for Radius for user authentication. The issue? I don't know what process is firing off the radius request so I don't know what to debug! Keep in mind this is all lab. No idea how this effects production.
[Expert@MGMT:0]# auditctl -D
No rules
[Expert@MGMT:0]# auditctl -l
No rules
[Expert@MGMT:0]#
.. and so it begins..We'll be telling auditctl to log all socket calls (socket is the function used to write data on tcp/udp protocols).
[Expert@FWGW2:0]# auditctl -a exit,always -S socketcall
[Expert@FWGW2:0]# auditctl -l
LIST_RULES: exit,always syscall=socketcall
[Expert@FWGW2:0]#
[Expert@FWGW2:0]# auditctl -l
LIST_RULES: exit,always syscall=socketcall
[Expert@FWGW2:0]#
ok now, i'll login to the IA captive portal. The clearpass is on 10.128.2.10. In hex that is "0A80020A". I don't understand all the information shown, but look at the comm= and exe= lines!!!
woooooooow....I also fired off a telnet to 8.8.8.8 on port 80. What did that find?
[Expert@FWGW2:0]# ausearch --host 10.128.2.10
time->Tue May 10 20:02:29 2016
type=SOCKETCALL msg=audit(1462928549.402:1480): nargs=6 a0=8 a1=7fb5d2c0 a2=4000 a3=0 a4=7fb5d260 a5=7fb5d25c
type=SOCKADDR msg=audit(1462928549.402:1480): saddr=0200066D0A80020A0000000000000000
type=SYSCALL msg=audit(1462928549.402:1480): arch=40000003 syscall=102 success=yes exit=100 a0=c a1=7fb5d230 a2=1ad678 a3=0 items=0 ppid=5322 pid=2687 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="pdpd" exe="/opt/CPsuite-R77/fw1/bin/pdpd" subj=kernel key=(null)
[Expert@FWGW2:0]#
time->Tue May 10 20:02:29 2016
type=SOCKETCALL msg=audit(1462928549.402:1480): nargs=6 a0=8 a1=7fb5d2c0 a2=4000 a3=0 a4=7fb5d260 a5=7fb5d25c
type=SOCKADDR msg=audit(1462928549.402:1480): saddr=0200066D0A80020A0000000000000000
type=SYSCALL msg=audit(1462928549.402:1480): arch=40000003 syscall=102 success=yes exit=100 a0=c a1=7fb5d230 a2=1ad678 a3=0 items=0 ppid=5322 pid=2687 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="pdpd" exe="/opt/CPsuite-R77/fw1/bin/pdpd" subj=kernel key=(null)
[Expert@FWGW2:0]#
[Expert@FWGW2:0]# ausearch --host 8.8.8.8
----
time->Tue May 10 20:06:53 2016
type=SOCKETCALL msg=audit(1462928813.605:3494): nargs=3 a0=3 a1=9485d80 a2=10
type=SOCKADDR msg=audit(1462928813.605:3494): saddr=02000050080808080000000000000000
type=SYSCALL msg=audit(1462928813.605:3494): arch=40000003 syscall=102 success=no exit=-4 a0=3 a1=7fa76e30 a2=4964dc a3=9485d60 items=0 ppid=4876 pid=4920 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=tty1 ses=69 comm="telnet" exe="/usr/bin/telnet" subj=kernel key=(null)
----
----
time->Tue May 10 20:06:53 2016
type=SOCKETCALL msg=audit(1462928813.605:3494): nargs=3 a0=3 a1=9485d80 a2=10
type=SOCKADDR msg=audit(1462928813.605:3494): saddr=02000050080808080000000000000000
type=SYSCALL msg=audit(1462928813.605:3494): arch=40000003 syscall=102 success=no exit=-4 a0=3 a1=7fa76e30 a2=4964dc a3=9485d60 items=0 ppid=4876 pid=4920 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=tty1 ses=69 comm="telnet" exe="/usr/bin/telnet" subj=kernel key=(null)
----
.... : O ...
ok.. thats pretty amazing.
auditctl -D to delete that rule.
http://linux.die.net/man/8/auditctl
Go there for a read. Lots of stuff out there to peak around on.
