Tracking memory alloc spike

[cfmorley]

Hi,

I have a VPS with GoDaddy which runs CentOS 5.6 on virtuozzo.

I have a weird issue that every minute exactly 167,000 pages of memory get allocated and then freed. This is causing random memory alloc failures for some processes - e.g. perl scripts/compiler etc.

"cat /proc/user_beancounters" around the time of a spike gives:

Version: 2.5
uid resource held maxheld barrier limit failcnt
10016161: kmemsize 8470374 9958352 9223372036854775807 9223372036854775807 0
lockedpages 0 0 9223372036854775807 9223372036854775807 0
privvmpages 96259 264370 261120 261120 1120
shmpages 661 661 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numproc 73 75 32567 32567 0
physpages 37188 95980 9223372036854775807 9223372036854775807 0
vmguarpages 0 0 9223372036854775807 9223372036854775807 0
oomguarpages 37188 95980 9223372036854775807 261120 0
numtcpsock 30 31 9223372036854775807 9223372036854775807 0
numflock 6 6 9223372036854775807 9223372036854775807 0
numpty 3 3 255 255 0
numsiginfo 0 1 1024 1024 0
tcpsndbuf 549328 549328 9223372036854775807 9223372036854775807 0
tcprcvbuf 491520 491520 9223372036854775807 9223372036854775807 0
othersockbuf 80128 85576 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 8472 9223372036854775807 9223372036854775807 0
numothersock 51 55 9223372036854775807 9223372036854775807 0
dcachesize 1139847 1152993 9223372036854775807 9223372036854775807 0
numfile 2665 2697 9223372036854775807 9223372036854775807 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 61 61 9223372036854775807 9223372036854775807 0

Should it be possible that maxheld privvmpages > limit ?!

I killed all user processes including crond - leaving only sshd and my shell & the behaviour still exists... spike of 100,000+ pages (550MB).

It happens after linux reboot & container reboot.

I wrote a small C program to allocate & commit 500MB memory. Then on the 1 minute "tick" I get exactly 1 extra on privvmpages failcnt - so it is one alloc.

Any ideas anyone? (GoDaddy support think there is nothing wrong with the VM)
Should privvmpages maxheld ever go above limit? Does this help indicate what the issue is?

P.S. VPS seemed fine @ 8am UK 27th September (ran big build script ok) but broken by 29th September (lots of alloc problems in scripts)

Thanks for any help!

Chris

[cfmorley]

Update:

No help from GoDaddy so I reprovisioned the server (very inconvenient).

The new build still has the memory spike as a clean build - 168,000 pages exactly once per minute.

It must be kernel/module. Interestingly after the rebuild the virtuozzo beancounters have significantly different limits (some +50%, some now unlimited) set so it appears (for the moment) that the spike is not causing malloc fails now.

The spike is clearly intentional then. I guess it is there to clear out buffers/cache/dirty pages or perhaps force dirty page garbage collection somewhere - most likely to return such dirty pages back to the host & reduce vm instance memory use.

Anyone comment on that or if a strategy like that would return memory to the host more regularly with Parallels Virtuozzo?

Thanks