Re: xz -9 : Cannot allocate memory

[Corinna Vinschen <corinna-cygwin at cygwin dot com>]

On Aug 30 17:03, Christopher Faylor wrote:
> On Fri, Aug 30, 2013 at 09:00:48PM +0200, Corinna Vinschen wrote:
> >On Aug 30 19:46, Corinna Vinschen wrote:
> >> On Aug 30 12:58, Achim Gratz wrote:
> >> > Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> >> > > Yes, looks normal and expected from what you observed.  mmap commits
> >> > > memory top-down and that was apparently the first free slot big enough
> >> > > to fullfil the request.  The default heap size is 384 Megs and then
> >> > > there's apparently not enough space anywhere.
> >> > 
> >> > Sorry if I'm dense, but does this mean that setting the default heap size to
> >> > zero does in fact mean it's trying to use 374MiB... wait, yes it is: the
> >> > memory map is the same when I'm setting the initial heap size to 384MiB with
> >> > peflags.
> >> > 
> >> > So, even though that rather large heap is essentially unused at the point of
> >> > failure and there is enough memory free just beyond the heap, the allocation
> >> > still fails because both the existing heap and the free space are both
> >> > smaller than what's requested?
> >> 
> >> Well, good question.  You could debug the sbrk function and see why
> >> it fails to reserve the space.
> >
> >I just debugged this and it seems our sbrk implementation has a serious
> >problem to extend the heap if the new chunk of memory requires to commit
> >some of the existing heap and to reserve and commit some more space.
> >It tries to reserve memory using the wrong address and the wrong size.
> >It also uses a too simple method to commit the memory.  I'll apply a fix
> >shortly.
> 
> It looked fine to me.
> 
> The only code change that I did make was to change your coercion of
> pointers to ptrdiff_t types into (char *), since that is more in keeping
> with the rest of the code.
> 
> I had a hard time reading the code with all of the cygheap-> pointers so
> I changed two functions to be a user_info_heap methods.  That, IMO,
> makes the code faster and more readable.

Good idea, that was an overdue change, I guess.

What still bugs me in sbrk is the fact that it prefers to reserve
subsequent chunks of heap in "chunk"-sized chunks (uh oh).  When using
the default heap size of 384 Megs, "chunk" and thus each subsequent heap
chunk will be 384 Megs as well if the space is still available.

That's an awful lot of memory, and it it could easily break later mmap's
or thread stack reservations on 32 bit.  Given that the default heap is
384 Megs already, I would prefer if we reserve subsequent heap memory in
chunks of 1 or 2 Megs only, to lower the pressure on the VM.

Sounds good?

> You probably saw that I fixed the USR1/USR2 issue.  That was a serious signal
> bug.  I think both of those warrant a new release in fact.

I try to do that on Sunday.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Attachment: pgpulUrUy4scf.pgp
Description: PGP signature