The machine where I ran the original test has 96 GB main memory.  A 32-bit
program runs out of memory when it hits 2 GB so I don't think any swapping
was involved.  I also saw that when memory use got up to about 2 GB, the
system was still using 100% of one CPU.  So it is actively trying to do
something but is not thrashing.

I compiled the latest run-time via ./quick_configure and then linked the
system (compiled with 6.3.74123) with the newest run-time.  It did not improve
things.

If I get dtrace permission on an appropriate machine, I can get the stack
trace for every malloc failure, which might be useful information.