[kaffe] Re: OutOfMemoryError while compiling GNU Classpath

[Dalibor Topic]

Dalibor Topic wrote:
> Dalibor Topic wrote:
>> Kiyo Inaba wrote:
>>> Hi Ito-san,
>>>
>>> You wrote:
>>>>> I have found that recent Kaffe causes OutOfMemoryError in the
>>>>> process of this compiling.
>>>> I tested this on a machine with rich memory, and have found out
>>>> that, in order to compile GNU Classpath 0.96.1 with ecj,
>>>
>>> Since I gave up compiling GNU Classpath everytime, I did not notice
>>> this trouble. But, even without zzip library if we need 384MB, the
>>> increase of 60MB or so should not make much difference ;-)
>>>
>>> Kiyo
>>> P.S. I am so happy to know the weekly snap helps tracking down these
>>>      problem.
>>> P.P.S. Of course, it is better to write some special note in FAQ about
>>>      this memory issue.
>>
>> GNU Classpath CVS head is a bit tough on the compilers, agreed. ;) But 
>> this seems to be a kaffe issue, so I'll investigate to see if the new 
>> zzip based code is leaking memory.
> 
> OK, running this through valgrind (and we could use a volunteer to make 
> Kaffe valgrind clean now that the wrestling with the compiler warnings 
> is coming to an end ...) gives me:
> 
> ==10731== Thread 1:
> ==10731==
> ==10731== 156 (36 direct, 120 indirect) bytes in 1 blocks are definitely 
> lost in loss record 4 of 25
> ==10731==    at 0x4022765: malloc (vg_replace_malloc.c:149)
> ==10731==    by 0x41E79B2: (within /lib/tls/i686/cmov/libc-2.6.1.so)
> ==10731==    by 0x41E8208: __nss_database_lookup (in 
> /lib/tls/i686/cmov/libc-2.6.1.so)
> ==10731==    by 0x48EAFDB: ???
> ==10731==    by 0x48EC11C: ???
> ==10731==    by 0x4194EEB: getpwuid_r (in /lib/tls/i686/cmov/libc-2.6.1.so)
> ==10731==    by 0x419489D: getpwuid (in /lib/tls/i686/cmov/libc-2.6.1.so)
> ==10731==    by 0x40DC566: 
> Java_gnu_classpath_VMSystemProperties_getUserName 
> (gnu_classpath_VMSystemProperties.c:317)
> ==10731==    by 0x8151E2B: ???
> ==10731==    by 0x81625FC: ???
> ==10731==    by 0x81570C2: ???
> ==10731==    by 0x40ABA3B: engine_callMethod (sysdepCallMethod.h:46)
> ==10731==
> ==10731==
> ==10731== 100 bytes in 1 blocks are possibly lost in loss record 7 of 25
> ==10731==    at 0x4021AA4: calloc (vg_replace_malloc.c:279)
> ==10731==    by 0x427E115: zzip_file_open (in 
> /usr/lib/libzzip-0.so.13.0.49)
> ==10731==
> ==10731==
> ==10731== 4,532,544 (46,800 direct, 4,485,744 indirect) bytes in 468 
> blocks are definitely lost in loss record 19 of 25
> ==10731==    at 0x4021AA4: calloc (vg_replace_malloc.c:279)
> ==10731==    by 0x427E115: zzip_file_open (in 
> /usr/lib/libzzip-0.so.13.0.49)
> ==10731==
> ==10731==
> ==10731== 1,885,752 bytes in 198 blocks are possibly lost in loss record 
> 22 of 25
> ==10731==    at 0x4022765: malloc (vg_replace_malloc.c:149)
> ==10731==    by 0x428C3F0: (within /usr/lib/libz.so.1.2.3.3)
> ==10731==    by 0x428C571: inflateInit2_ (in /usr/lib/libz.so.1.2.3.3)
> ==10731==    by 0x427E39C: zzip_file_open (in 
> /usr/lib/libzzip-0.so.13.0.49)
> ==10731==
> ==10731==
> ==10731== 13,402,112 bytes in 409 blocks are possibly lost in loss 
> record 25 of 25
> ==10731==    at 0x4022765: malloc (vg_replace_malloc.c:149)
> ==10731==    by 0x427E14C: zzip_file_open (in 
> /usr/lib/libzzip-0.so.13.0.49)
> ==10731==
> ==10731== LEAK SUMMARY:
> ==10731==    definitely lost: 46,836 bytes in 469 blocks.
> ==10731==    indirectly lost: 4,485,864 bytes in 337 blocks.
> ==10731==      possibly lost: 15,287,964 bytes in 608 blocks.
> ==10731==    still reachable: 990,462 bytes in 56 blocks.
> ==10731==         suppressed: 0 bytes in 0 blocks.
> 
> So Ito-san is right, it's my fesh zziplib code. Sorry about that, I'll 
> get on it right away, and thank you very much for spotting it.

I've committed a fix now.

It was a thinko. The old jar.h API did not really foresee a way to close 
a jar entry, so I didn't expose that functionality from zziplib. But it 
is necessary with zziplib to make sure that the resources allocated for 
a jar entry are freed when one's done using it. Since no jar entries 
were closed, the lost memory accumulated quickly.

I've made sure that all the jar entries that are opened also get closed 
when we are finished using them. That brings the memory consumption for 
compiling GNU classpath down to ~210M on x86-linux-jit3-kaffegc. I hope 
that the results are similar on i386-freebsd, too.

Thank you very much for spotting this embarassing bug, and I hope the 
fix will work for you, too.

cheers,
dalibor topic