[linux-security] Re: IPMASQ and lock-up of all terminals ---- Sum mary and update

[hgtaesml umail furryterror org (Zygo Blaxell)]

In article <27BC18174C3CD2118F6000A0C99E423E026A17F3 CRPHEX02 NAVSSES NAVY MIL>,
 <MeriwetherDJ nswccd navy mil> wrote:
>Well, last night, my box was hit again.. same symptoms:
>
>All attempts to connect remotely receive a connection, but a login prompt
>never comes up.

What about other servers that this machine is running?  Do they all hang, or
can you get a response from one of the other ports?  Do a 'netstat -antu'
while the machine is up, then try every listed port next time it's
not feeling well.

>When I went to the console and turned on the monitor, I had the login
>prompt, but written on to the screen was the message 
>IPMASQ: Reverse ICMP: Checksum error from xxx.xxx.xxx.xxx

But could you get another login prompt, or actually log in?  If you can
hit ENTER and/or Ctrl-D and get more login prompts, but as soon as you
enter a user name or password it hangs, then it's probably syslogd
(login (actually PAM) will hang as soon as it tries to syslog the username
you're logging in with).  

If you can't even get further login: prompts, then something more
low-level has failed.

Try enabling Magic-SysRq support in your kernel 
('echo 1 > /proc/sys/kernel/sysrq' may be required) then press Alt-SysRq-T
to get a list of tasks (you may need Alt-SysRq-8 first, to cause all log
messages to be displayed on the screen instead of intercepted by klogd).

Incidentally, you can also use this feature unmount and sync your disks,
with Alt-SysRq-U and Alt-SysRq-S, respectively.  Sometimes Alt-SysRq-U
unhangs things, particularly swap-related deadlocks.

>So, on this occasion, I thought I would post a summary of the responses I
>got, and ask a few specific lingering questions.
>
>Summary:
>Some commented on having the same problem until they switched from one
>distribution to another, or from one version to another. (this doesn't help
>me too much as the same problem happened in two different versions 6.0 and
>6.2 beta)

Presumably you're counting two different revisions of Red Hat as changing
distributions?  That's not a big change.  Debian or Slackware will give
you different behavior (although not necessarily different in a good way ;-).

>Some suggested I check my binaries for trojans, and other signs of attack.
>I found none.

>Some suggested the problem lay with syslogd locking up. Several specific
>conditions that have caused this phenomenon in the past were mentioned, but
>none of the circumstances fit my case.  (the machine using itself in
>resolv.conf, disk full,  etc.)

One case that always seemed to hang syslogd for me is syslogging to a
remote server from a Red Hat machine.  If the network between the log
client and log server goes down, syslogd inevitably hangs.  This problem
seems to have gone away in recent (in the last year) versions of Debian's
sysklogd package.  IIUC Debian and Red Hat have had different sysklogd
code bases for some years now.

>Another mentioned the possibility that a fork bomb or a DOS attack may have
>caused the system to run out of processes (of which they reported the
>default to be 512).   I know of no way of verifying this theory, nor do I
>know of anyway to defend against such an attack.  Help in this arena would
>be appreciated.

The defense against DOS attacks is to better control resource usage or
increase capacity until the DOS doesn't consume all of your resources
(though the latter case is more useful for "unintentional" DOS, e.g. when
your machine is simply crushed by the amount of legitimate work it has
to do).

To verify the theory, try running '(date; ps xaf) >> /var/log/ps-log' 
out of crontab every minute until the next time this problem happens.  
This may give you a clue if the fork bomb occurs slowly enough.

You can get some useful information by simply leaving 'top' running on
a virtual console.  At least you would have a partial process list just
before the crash.

>Continuing questions:
>
>1) What exactly is a Reverse ICMP?  (That message has been on the terminal
>screen 3 out of 3 times I have had this problem.)

Probably an ICMP that isn't a Forward ICMP.  ;-)  Actually an ICMP packet
that is going through the firewall from outside to inside, as opposed to
inside-to-outside (i.e. an ICMP "reply").

>2) Is there a way to directly test whether syslogd is the culprit?  Is there
>a way I can correct it?

If you have a shell prompt on the machine, try "echo boo! | logger".  If
that hangs, your problem is syslogd.  Note this will require you to leave
an ssh logged in under screen, or some similar thing, because you need to
be already logged in in order to do this.

To correct the problem, '/etc/rc.d/init.d/syslogd restart' (or 
'killall -9 syslogd; syslogd').  

Note that if your machine is unable to schedule any processes
(e.g. because the swap disk died), it can still accept() several hundred
incoming network connections.  

>>From the symptoms described so far, the problem could be nearly anything
from deliberate DOS to bad disk controller firmware, particularly if
console log messages are being routed through klogd.  A deadlock caused
by masquerading code isn't outside the realm of possibility.

>3) Is there a way of directly testing whether I am the victim of an
>occasionally fork bomb or DOS attack?  Is there a way I can correct this?

Logging...lots of logging.

You'd probably notice a fork bomb with tools as basic as 'ps' or 'top'--if
you got there fast enough.  If you're too late...well, you can't ask the
machine if it's running a fork bomb because it's too busy forking to
respond to you.  ;-)