[Freeipa-devel] Maintaining Identity in a large cluster

[Matthew Booth]

Hi,

I was recently asked to look into the problem of auditing user activity 
on a large cluster. There are a number of issues, but the one I'm 
highlighting here is the problem of maintaining a who a real user is in 
a large cluster.

If you talk to a cluster designer, their cluster is *a* machine. It is 
not a collection of machines, but a single entity. Security is 
important, but it is provided only at the boundary. Separate components 
of the machine (anybody else might call them servers) would have no more 
business authenticating each other than a CPU core would have 
authenticating another core. If I need to become root on an 8-core box, 
I wouldn't expect to enter my password 8 times. Now imagine there are 
hundreds of 'cores'.

A key administration tool of any cluster is a distributed shell. There 
are many implementations, but usage always boils down to something like:

# clustersh node[1-512] rpm -Uvh /clusterfs/newpkg.rpm

Key points to note:
* The user is root
* The user expects the command to run as root on nodes 1-512.

 From a security standpoint this is fine. The cluster trusts its network 
because it's entirely separate from any other network (it's just a bus 
in our big machine, right?). The user had to log on to the cluster in 
the first place with their own credentials, so access control is 
maintained. The problem comes when you introduce auditing.

Under other circumstances, 'best practise' would be to insist that a 
user log on as themselves, then escalate their privileges to root via an 
approved method. The audit system can tag them as they log in, and all 
subsequent actions can be made accountable. This doesn't work on a big 
cluster because the system administrator can't be expected to enter 
their password 512 times.

The solution is typically ssh keys shared across the cluster. The effect 
of this is that anyone who can perform an identity change on any machine 
can become anonymous on the cluster just by logging on to another node 
after the identity change. In practise, most/all users will be able to 
perform an identity change. If they are administrators this will be to 
root. If they are users, this will be to a processing user.

The problem extends beyond just cluster shell operations. For example, 
MPI jobs will typically be initiated on 1 node but executed on many. 
Again, it cannot be expected to require an authenticated privilege 
escalation for each target node.

The challenge is to maintain the identity of a real user across the 
cluster without compromising the 'single machine' model of the cluster. 
I'm not convinced this can be done today. It occurred to me that 
kerberos might be useful in this space if sshd can be coerced into:

1. Allowing principal mbooth/root to log in directly as root.
2. Setting the audit context to mbooth.

Has this problem been given any thought to date?

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat, Global Professional Services

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490