RE: [PATCH 2.5.64] Real-time futexes (priority inheritance/protection/robust support) take 4

["Perez-Gonzalez, Inaky" <inaky perez-gonzalez intel com>]

> -----Original Message-----
> From: Ulrich Drepper
>
> > - Does not modify original futexes - a new interface is added
> 
> This is the killer.

I know, but it is the only way I found - either that or you help me convince
Rusty, that I doubt, and I understand his reasons. OTOH, you still need the
normal futexes (see below, for barriers).

Reasons for it:

* We are completely modifying the interface to access the futex
functionality, and in a way, it is a different functionality; we cannot use
counts any more, we need to use fixed values [PIDs] and atomic
compare-and-swap. This is diametrically different and some arches don't
support it (the atomic compare-and-swap, I mean).

* We cannot wake more than one waiter at a time: "correctness" && real time
wise this is calling for trouble.

Why I thus decided to go to a second interface is because we are changing so
much stuff we'd break more than one teeth and that is a no-no. And there is
no way I could find to do a decent and effective implementation using the
current interface that could provide owner identification for PI, PP,
deadlock-detection and robust support across different address spaces
without going to the kernel every time - I'd love to hear of one, though.

So it makes sense to have a separate interface, a new one, that we still
access through the sys_futex() syscall. I guess you would agree with me here
that blending this into the actual futex interface is almost impossible, not
to speak of how much stuff it would break.

> If it's not the same futex interface which is used it is not possible to
> use the non-rt version of the libpthread and the rt version on the same
> system.  The whole thing is complicated by statically binaries.

Agreed - that's what one of my buddies is looking for - how to do a simple
switch, that consistently, and across different processes with redirect each
instance of a mutex to the proper futex API (unless using rtfutex for all
cuts it well enough and without any impact - falling back in compile time to
the normal futex interface for the arches that do not support atomic
compare-and-swap).

> The problem is obvious: what do you do for mutex/condvars/barriers/etc

For mutexes: two options:

1. on _init time, you determine what interface is to be used and set a bit
or setup a function pointer in the mutex structure that indicates which
interface to use for all the call redirection to be done properly (for
example, pthread_mutex_t.__data.__kind). The trick here is to get it right
so that (a) overhead is close to nil [I know, understand and agree on you
being anal about this - me too] (b) it is correctly propagated/shared. If we
are using shared memory to share the mutex, well, the other guy will see the
same bit in pthread_mutex_t [in this case is where a function pointer does
not work, cross that out] and do the proper call forwarding. 

2. or use the rtfutex interface all the time on arches that support atomic
CAS instead of having to switch - falling back to the futex interface
[compile time]. This would also depend on if we can get good performance out
of the rtfutex, close to that of the futex. I think so, although I still
haven been able to measure it properly.

On condvars, I am not that sure - but to me it looks like they can use the
same mechanism as mutexes. It even makes sense to enable PI/PP/robust.

Barriers cannot use the rtfutex interface: think about it, it makes little
sense. For once, you need to wake up N waiters, not just one of them -
barriers don't care about the order, so it makes little sense to do it. In
the same motto, PI/PP does not makes sense for barriers, neither deadlock
detection or robust futex support. Asides, rtfutexes can be woken up only by
the same process that locked them.

Rwlocks: the only thing they can use is deadlock detection, maybe, and also
maybe, robust support - but not that well. How do you determine or record
the fact that two thousand readers own the lock ... well, it'd need a lot of
glue, so rwlocks can keep using the normal futex interface or internally
glue rtfutex and do their stuff manually (as it is done now, btw).

Am I missing anything ... oh, spinlocks. These guys play their own game and
do their own stuff, although it might make sense to convert them to do the
loop using rtfutex_trylock() for so many loops and then falling back to the
kernel to wait if we are too concerned about burning sooooo much CPU from
other processes.

In sum - the idea is not that rtfutex replaces the futex interface - it is
meant to live along with it, as each one has an specific use.

> actually initializes the object to do some more work and tell the kernel
> something.  But if the static initializers are used not even this is
> possible.  For pshared object it is possible since there is no static
> initializer to create such objects.

I think that the scenarios/methods I explained get it done - without needing
any help in the kernel or whatsoever. Let me know if I missed anything. 

> If you insist on not having a unified object the only possibility is to
> have separate libpthreads.  Because of the confusion this creates and
> the support nightmare I strongly oppose this but of course cannot
> prevent it from happening in a separate project.

I am known to be nuts, but not to that extent ...

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)