PR# 18591 Memory leak

Hi

for some time I have had problems with one of my programs that it consumes a lot of memory. It crach when it is out of memory and I have seen allocation levels between 4 and 5 GB. I have tried to pinpoint the problem. Then I have noticed two things:

1. I allocate slowly more memory. I have identified 

SPECIAL [!ANY]

as the cause of it. It is slowly increasing the number of it allocated. I have guessed that it is part of a LIST or an ARRAY, I use both LISTs and ARRAYs in my program. But since neither SPECIAL nor ANY gives me any clue of the connection to my types/classes I wonder if you have any idea of how I can get more information from the Eiffel System about it? I have attached some code I use                   to extract information from the memory and also examples of the result I get.

2. After several hours execution, in this particular case about 3 hours the program allocates several hundres of MB each minutes. I have compared the output of the allocated classes during this period, see attached files, and I can not get any clue where the "memory leak" take place. It is pnly a smaall increase of different Eiffel calsses.
Can I get more information from the Eiffel System regarding this compared to what I write in the example code attached?
I write the different classes and then the total memory consumed.



My own ideas of how to proceed is

1. To make a samll test case where I can reproduce the problem.

2. With your help extract more memory information from my program, e.g. can you suggest what more (nore than total memory and all classes and the instances of them) info I can write to my logfiles.

Regards

Anders

We have been able to reproduce the problem and are still investigating to understand the reason.

Hi, have you been able to reproduce the problem?

That is the deafult folder for test cases, at least in my system.

I have not taken the time to but my guess is that it takes between 5 and 10 minutes.

Anders

Sorry, I usually do not extract EIFGENs folders and did not see that. How long does it take to fail on your setup?

Hi

Have you looked in the zip-file in the EIFGENs/seriali...report/Cluster folder?

It contains a file called new_test_set.e . That one uses the IALIZER class.

Regards

Anders

The code you sent us is not runnable, that is to say the root class does nothing, and there are not test cases that use the IALIZER class. Could you send us the remaining files?

I noticed that I used an obsolete class. I substituted it but got the same problem. See exception log attached.

Hi

finally I have managed to do a small test case that recreates the memory problem I have had for a long time. 
I have attached the dump I get so you can see the result of my execution.

One have to run the test case for some minutes until it breaks.

I have no idea of why it fails after some iteration.

The code in the IALIZER class is taken from eiffelroom (might be some minor changes since I took it some years ago)

Regards

Anders

Hi, I think i finally found the problem. It seems that the serialize classes can not be used in parallell frpm two threads even if they are diferent objects. WHen I added semaphores around them the problem disapperaed. 
I will try to reproduce the problem in a small program and file a report.
However, I am struggeling with converting to SCOOP instead and it will peobqably take some time until I can make take care of this problem again.

REgards

Anders

Hi

I have now narrowed the problem to two loops. One loop in each of two threads, e.g. a producer/consumer pattern.

I would like to be able to do according to the following:

1. Take a snapshot of the memory in one of the loops

2. execute some iterations in the loop

3. Take a new snapshot of the memory 

4. Make a comparsion of the two to try to understand where the leak is.

Is this possible with the available tools? 

If so, can yo ugive some indications of how to do it.


Regards

Anders

HI,

I have been able to use the tool but it would be nice with some documentation/recommendation about how to use.

I have still problems with the memory leak. I have made testcases to pinpoint it but at the moment it takes about 20 hours of execution at my fast computer until it appears so it is not that easy to find. 
THe test case takes about the same amount of memory for about 20 hours until it suddenly consumes all available memory in a few seconds. 
Well, I will continue to my investigation.....

Sorry, it seems that we missed your question. Were you able to use the memory analyzer library and to find/solve your memory leak?

I found the `memory_analyzer' library but not much documentation. I was at least able to incorporate it in my application and look at it. But does it exist any user documentation?

Regards

Anders

The way we actually solve this type of problem is by using the `memory_analyzer' library. This is a graphical tool that let you take memory snapshot and compare them. That way you know what changes between two snapshot. Then for each object of a certain kind, you can actually find out the referrers and usually this kind give you a clue where the memory leak is.

In your case, because the leak seems to be on SPECIAL objects and that you have more than 30000 it might be difficult to find out which ones are the newly created one. So ideally, the snapshots have to be done at the very early stage of the program execution so that manual inspect can be done.


In addition, the major causes of memory leaks are:
- keeping object in an observer pattern structure without removing them when not needed anymore
- calling some C externals that might prevent the object to be garbage collected
- having objects being referenced from a once routine
- putting objects as part of an agent that is kept forever.

Attachments for problem report #18591