We just ran into this problem in production.  We were running a 32-bit
application which needed more than 2 GB of virtual memory.  The application
memory use got near 2 GB and then it kept running for more than 60 hours
without reporting "out of memory", though we know that it needed more memory.
We finally killed it, compiled it as a 64-bit application, and re-ran it.
This time the app used about 5 GB memory at its maximum and finished in about
1.5 hours.

Please consider trying to improve the "out of memory" behavior for 6.5.