I ran the system in the original report under truss and confirmed that malloc
is indeed failing:

/1@1:   -> libc:malloc(0x400028, 0x1, 0x13af4, 0x13800)
/1:     brk(0x7F50EFE0)                                 = 0
/1:     brk(0x7F90EFE0)                                 Err#12 ENOMEM
/1@1:   <- libc:malloc() = 0

In the output above, we see a call to malloc requesting 0x400028 bytes.
Malloc calls `brk', which fails with an ENOMEM error.  Malloc then returns 0
(null).  So malloc is failing.  Yet the program continues to run long after
this failure before it finally reports an "out of memory" error.