Offline audit trail analysis
Todd, Charles
CTODD at ball.com
Tue Sep 11 21:20:07 UTC 2007
Karen,
I'm most impressed. I've been rewriting the audit trails as English
text, so I can shove them into syslog and a detailed review could have
more context. I REALLY, REALLY like your --karen as root-- and --karen
as dan-- lines. Very impressive. I've also toyed in my head with
strings that might look like --karen as dan-- via
ssh at badhost.hacker.com, but so much state information has to be kept
that might not be available when things roll off the archive. You've
obviously put some thought into usability. I added a keyword such as
AUDIT_SRO to every record where I was testing access to
security-relevant objects which makes for an easy grep later. I also
include the raw event ID for more details if I want them later.
My focus is the investigator looking at my audit trails four years from
now trying to see the kinds of things an individual did while on my
network. Presumably, they've been identified as an insider threat and
an investigation is underway. I'm certain to have changed IP address
resolution and user names around. OS upgrades will certainly have
happened.
Thanks for the reponse and continued ideas. My idea for getting "the
boss" to let me share is to go to my security officer and say "if I
release this code, it will help increase uniformity in the community
will make the higher-echelon security folks feel like we know what we're
doing in our shop." Once, they're on board with sharing, then your
management chain might have fewer arguments. (we can take that
discussion offline if you want)
Charlie Todd
Ball Aerospace & Technologies Corp.
________________________________
From: Wieprecht, Karen M. [mailto:Karen.Wieprecht at jhuapl.edu]
Sent: Tuesday, September 11, 2007 4:59 PM
To: Todd, Charles; linux-audit at redhat.com
Subject: RE: Offline audit trail analysis
This isn't probably as savvy as some of you would like, but I
started writing it before there were any auparse libraries to help with
this kind of stuff. I've done similar stuff on Solaris, Mac OSX,
irix, etc. My linux version scripting isn't finished (not enough hours
in the day), but it's still useful for my log reviews. At some point I
hope to get out internal people to bless it enough to share with whoever
might be interested in the open source world, but I'm not there yet.
here's the general thing I've setting up:
I use scripting to do the following on each machine :
- rotate the log
- (I want to try a script segment to make sure all related audit
event records are not interleaved)
- run an ausearch on it with some flags (interpret, all
messages) and some shell commands to get all related records for a
particular event onto a single line
- save this processed file (with user IDs, etc. particular to
that system) as something like audit.log.hostname.2006-09-10.ausearched
- Store the original and "ausearched" files in a central place
so I have them all gathered for my weekly log review
Then weekly, I use a "reviewlogs" script to post process the
"ausearched" "single line" data into something more readable. It also
filters some noise I don't care about. The Chronological results look
like this from various test sessions:
karen FAILED TO mv /etc/nsswitch.conf
/etc/nsswitch.conf.mine on patton, Permission denied on
01/15/2007 14:05:16.582
dan FAILED TO console login into patton on 01/17/2007
13:21:45.133
dan successful console login into patton on
01/17/2007 13:21:59.152
dan successful console logout from patton on
01/17/2007 13:22:26.059
karen FAILED TO ssh login into patton from ? on
01/17/2007 13:34:26.517
karen successful ssh login into patton from
patton.jhuapl.edu on 01/17/2007 13:34:31.387
karen FAILED TO touch /etc/nsswitch.conf on
patton, Permission denied on 01/17/2007 13:34:57.759
karen FAILED TO touch /etc/nsswitch.conf on
patton, Permission denied on 01/17/2007 13:34:57.759
karen successful SU LOGIN (PAM authentication) as root
on patton on 01/17/2007 13:35:15.140
karen successful SU LOGIN (PAM accounting) as root
on patton on 01/17/2007 13:35:15.141
karen successful SU LOGIN (PAM session open) as root
on patton on 01/17/2007 13:35:15.141
-- karen as root -- successful SU LOGIN (PAM
authentication) as dan on patton on 01/17/2007 13:35:31.364
-- karen as root -- successful SU LOGIN (PAM
accounting) as dan on patton on 01/17/2007 13:35:31.366
-- karen as root -- successful SU LOGIN (PAM session
open) as dan on patton on 01/17/2007 13:35:31.367
-- karen as dan -- FAILED TO rm /etc/hosts
on patton, Permission denied on 01/17/2007 13:35:39.129
-- karen as dan -- FAILED TO rm /etc/hosts
on patton, Permission denied on 01/17/2007 13:35:40.680
-- karen as root -- successful SU LOGIN (PAM setcred)
as dan on patton on 01/17/2007 13:35:44.177
-- karen as root -- successful CLOSED SU as dan
on patton on 01/17/2007 13:35:44.178
karen successful SU LOGIN (PAM setcred) as root
on patton on 01/17/2007 13:35:46.089
karen successful CLOSED SU as root on patton on
01/17/2007 13:35:46.089
Note that it's easy to spot users trying to make it look like
other users tried to do stuff. When I review the logs, I usually sort
the reformatted lines so that similar entries are all grouped (which
makes it really easy to let chunks of stuff I don't care about scroll by
quickly). If I see something suspicious, I'll go back and look at the
stuff in chronological order to see what was going on. Here's what the
above sample looks like sorted (and if I had a lot of data, the chunks
of similar stuff would be more apparent):
-- karen as dan -- FAILED TO rm /etc/hosts
on patton, Permission denied on 01/17/2007 13:35:39.129
-- karen as dan -- FAILED TO rm /etc/hosts
on patton, Permission denied on 01/17/2007 13:35:40.680
-- karen as root -- successful CLOSED SU as dan
on patton on 01/17/2007 13:35:44.178
-- karen as root -- successful SU LOGIN (PAM
accounting) as dan on patton on 01/17/2007 13:35:31.366
-- karen as root -- successful SU LOGIN (PAM
authentication) as dan on patton on 01/17/2007 13:35:31.364
-- karen as root -- successful SU LOGIN (PAM session
open) as dan on patton on 01/17/2007 13:35:31.367
-- karen as root -- successful SU LOGIN (PAM setcred)
as dan on patton on 01/17/2007 13:35:44.177
dan FAILED TO console login into patton on 01/17/2007
13:21:45.133
dan successful console login into patton on
01/17/2007 13:21:59.152
dan successful console logout from patton on
01/17/2007 13:22:26.059
karen FAILED TO mv /etc/nsswitch.conf
/etc/nsswitch.conf.mine on patton, Permission denied on
01/15/2007 14:05:16.582
karen FAILED TO ssh login into patton from ? on
01/17/2007 13:34:26.517
karen FAILED TO touch /etc/nsswitch.conf on
patton, Permission denied on 01/17/2007 13:34:57.759
karen FAILED TO touch /etc/nsswitch.conf on
patton, Permission denied on 01/17/2007 13:34:57.759
karen successful CLOSED SU as root on patton on
01/17/2007 13:35:46.089
karen successful SU LOGIN (PAM accounting) as root
on patton on 01/17/2007 13:35:15.141
karen successful SU LOGIN (PAM authentication) as root
on patton on 01/17/2007 13:35:15.140
karen successful SU LOGIN (PAM session open) as root
on patton on 01/17/2007 13:35:15.141
karen successful SU LOGIN (PAM setcred) as root
on patton on 01/17/2007 13:35:46.089
karen successful ssh login into patton from
patton.jhuapl.edu on 01/17/2007 13:34:31.387
We don't have a ton of Linux systems (yet), so this strategy
works well for reviewing a handful of systems from a central location.
The scripts save me a lot of time interpretting any one given log entry
as well as keeping me from having to log into N systems one at a time
and running M commands on each one each to decide if someone was messing
around on my machines or not.
Karen Wieprecht
________________________________
From: linux-audit-bounces at redhat.com
[mailto:linux-audit-bounces at redhat.com] On Behalf Of Todd, Charles
Sent: Tuesday, September 11, 2007 3:32 PM
To: linux-audit at redhat.com
Subject: Offline audit trail analysis
Thanks to Steve for being our biggest target for questions on
this list!
Has anyone talked about sane ways to do offline analysis of
Linux audit logs? Presumably, this would be on another Linux system,
but maybe not the same host, and probably not on the same release or
with the same username/IP address access. Conceptually, ausearch would
save and optionally read a system's "configuration" to be saved for
interpretation later.
My goal is central logging, but doing the reporting/analysis on
the central host. That way, I can see a user across the Enterprise (or
at least in the Linux hosts), but with all the power of ausearch for
refining the report. Ideally, I would do an ausearch -ts <date> -te
<date> --raw --config-to=<hostname.ausearch.config> and it would do
things like saving the syscall lookup table, lookup users referenced in
the reported audit trail, and resolve IP addresses references in the
reported audit trail. Maybe one config file could be written for each
data type in an existing format (e.g. users in /etc/passwd format, hosts
in /etc/hosts format, etc.). I'm mainly after whether or not anyone has
considered extending ausearch for this kind of processing?
This way, an archive of raw logs could be kept along with the
exact system configuration which allows offloading the audit trail
analysis to a trusted location, rather than risk side effects from a
rootkit.
Charlie Todd
Ball Aerospace & Technologies Corp.
This message and any enclosures are intended only for the
addressee. Please
notify the sender by email if you are not the intended
recipient. If you are
not the intended recipient, you may not use, copy, disclose, or
distribute this
message or its contents or enclosures to any other person and
any such actions
may be unlawful. Ball reserves the right to monitor and review
all messages
and enclosures sent to or from this email address.
This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-audit/attachments/20070911/2685f38f/attachment.htm>
More information about the Linux-audit
mailing list