Re: Memory and performance with an alpha based Linux box

[roman songdog eskimo com (Bill Roman)]

Dave Gilbert wrote:
> 
> On Fri, 18 Jun 1999, Jeff Bolle wrote:
> 
> > I believe alphalinux supports up to 2 gigs of ram, which is a limitation
> > thrust upon it by the intel side (i believe, there are probably many more
> > knowledgeable people on here than me who know more on that)
> 
> The problem is actually PCI ; 32 bit PCI cards only have a limited memory
> space that they can see (2GB or is it 4...) and you have to teach drivers
> about that so drivers and things which are trying to talk to SCSI cards
> and the like need to know that if they have a lot of RAM that they can't
> DMA into any old chunk of it.

Jeff and Dave both have part of the story; here are a few more details.

Most common PCI cards can only generate 32-bit memory addresses.  This is
sufficient to span 4 GB.  Part of this address space is occupied by
memory-mapped devices (a good example being video card frame buffers), so
less than 4 GB is available as a region to be mapped to system memory for
PCI devices to do DMA transfers.  The simplest hardware design allocates
such regions in chunks whose size is a power of two, hence the 2 GB limit.

Because of their 32-bit address capabilities, Intel architecture systems
have generally blurred the distinction between the 4 GB PCI memory address
space and their 4 GB system memory space.  DMA devices pretty much deal
with system memory addresses, without any consideration that the PCI
and system memory address spaces are actually two different things.
This is often referred to as "direct mapped" DMA.

Even the first Alpha systems had more than 32 physical address bits
out of the processor; the EV6 has 44.  Alpha system chipsets allocate
completely separate physical address space for each PCI bus, and must
provide translation from bus addresses to memory addresses.  They have
the capability of using direct mapped translation just like Intel;
this works fine for small (2 GB or less) system memory.

But to accommodate large memory (as well as for other reasons), Alphas
also have "scatter/gather" DMA address translation.  The bus address space
that is used for DMA is split into pages, and a single-level page table
translates bus addresses to memory addresses.  This has the advantage
that the DMA device's view of the destination buffer can be made to
match the program's virtually contiguous (but physically discontiguous)
buffer; it has the disadvantage that there is additional CPU overhead
to manage the scatter/gather map.

There was some discussion recently of implementing scatter/gather DMA
in Linux; it would be beneficial on Alpha, and probably also on (excuse
my language) Merced.  Maybe we can look forward to this in 2.3?

There are quite a few details I've glossed over here: the legacy ISA
memory hole above 640K, and the "Physical Address Extension" (PAE)
feature of newer Intel processors, as well as Intel's recent "invention"
of scatter/gather DMA in connection with AGP.  If you're eyes haven't
glazed over already, but your interest has been piqued, check out the
processor and chipset specs on Intel's web site.

-- 
Bill Roman  (roman@songdog.eskimo.com / roman@songdog.uucp)   running linux