find-requires.pl will automatically add dependencies on every shared library
loaded by any binary in the package. pan, for example, automatically gets a
dependency on libgdk-x11-2.0.so.0, by the virtue of linking with it.
This dependency is provided by the gtk2 RPM which, in turn, automatically
has a dependency on libX11.so.6, libXext.so.6, and others, by the virtue of
linking against those libraries. This is sufficient to require
XFree86-libs.
I don't think you need anything special to correctly resolve runtime
dependency of X applications. If you use a substitute X11 server+libraries,
as long as the substitute installs libX11.so.6, et al, everything will work
correctly.
find-provides.pl automatically declares a provides for all matching shared
libraries installed by the package, so you do not need to do anything
explicit with the substitute X11 servers/runtime libraries. As long as they
install shared libraries that have the same names (even if they are
installed in different directories!), all runtime dependencies will be
satisfied.
As far as development libraries go, it does look like an explicit Requires:
may be necessary.
You might be able to get away with adding "Provides: XFree86-devel" to
substitute X development libraries. It shouldn't prevent them from
co-habiting with the real XFree86-devel, or other substitutes, and RPMs that
explicitly require XFree86-devel should still be buildable.