Home Forum Developers General discussions Memory allocation problem

JRPassphrase Registration Control

In order to register on this site, you must first submit the passphrase below.

TODO list for each hardware target can be found as sticky topic in the corresponding forum


nunne
useravatar
User Info

Memory allocation problem

Hi,

I have been seeing a memory allocation problem in the latest versions of switchfin.
I dont see this problem in earlier version, but then asterisk will fail on other stuff like dialing more than 3-4 panasonic sip phones at once tongue


Luckily the latest version of switchfin will not crash when memory allocation fails (like it did some revisions ago!).

Here is the serial output from the shell:

Code:

asterisk: page allocation failure. order:6, mode:0xd0

Stack info:
SP: [0x034bfdac] ��.
Memory from 0x034bfda0 to 034c0000
034bfda0: 000000d0  034bfdac  00000000 [001db300] 00035dae  063fe5c0  00000000  000200d0
034bfdc0: 063fe7a0  00000006  000000d0  00000001  00000042  001da8d4  00000000  034be008
034bfde0: 00000010  00000050  00000040  034be000  034be000  01429780  000200d0  001db304
034bfe00: 00000040  00000000  063fe5c0  00000000  00000040  00000010  00000000  001da8d4
034bfe20: 00000000  00000000  00000000  001da8d4  03ae86e0  0003dd08  00029000  0277361c
034bfe40: 0277361c  00000006  00000073  00000000  04000021  034bfe6c  00b2fd30  00000010
034bfe60: 00000000  00000001  04000021  03ae86e0  0003e062  00000000  0277361c  00000000
034bfe80: 00000003  00000073  00000000  04000021  02c9cf91  ffffffc0  00029000  00000004
034bfea0: 04000021  00000000  034bfeb0  001cd210  00000004  034ba1b8  0003e3fc  00000000
034bfec0: 00029000  05a61250  034be000  00000000  04000021  00000003  00000000  05a61250
034bfee0: 00029000  00000003  04000021  00000000  ffa008be  0003e3b8  000000c0  00000000
034bff00: ffffe000  00000000  00000000  05a61250  05a60b98  05a60b9c  04000021  00000000
034bff20: 00000000  0418d2c8  00008000  00002000  00000000  034c0000  0418d2c8  0418d2c8
034bff40: 041ac02e  ffa00e44  02003065  0419bb07  05be23b7  0419bb06  05be23b6  00000000
034bff60: 00000000  00000000  00000000  00000000  00000000  00000000  7ffff000  000000c0
034bff80: 00000137  00000000  00000000  00000000  00000000  0000005b  00001802  00000001
034bffa0: ffffffc0  00000006  034b9f10  00000001  00000000  034ba1ac  034ba1b8  05a61250
034bffc0: 0594e8d8  05a61250  00000000  05977718  000000c0  00000000  05a60b9c  00000000
034bffe0: 00000000  04000021  00000003  00029000  00000000  00000000  000000c0  00000006
Return addresses in stack:
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
active_anon:0 inactive_anon:0 isolated_anon:0
active_file:6 inactive_file:4 isolated_file:0
unevictable:90 dirty:2 writeback:0 unstable:0
free:19676 slab_reclaimable:86 slab_unreclaimable:395
mapped:0 shmem:0 pagetables:0 bounce:0
DMA free:78704kB min:1300kB low:1624kB high:1948kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:16kB unevictable:360kB isolated(anon):0kB o
lowmem_reserve[]: 0 0 0
DMA: 401*4kB 277*8kB 291*16kB 322*32kB 320*64kB 306*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 78684kB
100 total pagecache pages
26624 pages RAM
760 pages reserved
8 pages shared
6117 pages non-shared
Allocation of length 167936 from process 13592 (asterisk) failed
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
active_anon:0 inactive_anon:0 isolated_anon:0
active_file:6 inactive_file:4 isolated_file:0
unevictable:90 dirty:2 writeback:0 unstable:0
free:19671 slab_reclaimable:86 slab_unreclaimable:400
mapped:0 shmem:0 pagetables:0 bounce:0
DMA free:78668kB min:1300kB low:1624kB high:1948kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:16kB unevictable:360kB isolated(anon):0kB o
lowmem_reserve[]: 0 0 0
DMA: 401*4kB 277*8kB 290*16kB 322*32kB 320*64kB 306*128kB 1*256kB 0*512kB 0*1024k
100 total pagecache pages

and it spews out these errors. the call doesnt really seem to "fail".. but I have not verified it. I am currently stresstesting the system, so that is why I see this error after just an hour or so. (making about 10 calls/minute to different phones with a call file to /var/spool/asterisk/outgoing).


and in earlier versions i never really got anything usefull from my asterisk log. but now I did! probably because it didnt crash tongue


Code:

[Mar 27 07:12:08] ERROR[13071]: /root/2012-03-26/switchfin/build_br4/asterisk-1.4.42/include/asterisk:369 _ast_calloc: Memory Allocation Failure in function ast_udptl_new_with_bindaddr at line 753 of udptl.c

Sounds like this might be the trouble? is it running out of "bigger" DMA-blocks? My kernel settings are default for the BR4-target. And it says its allowed to allocate 1MB of memory, and it fails on chunks of 160-210kb of memory. While there is 75-80mb of free memory (my BR4 board has 128mb memory instead of 64mb).

But is it a DMA allocation problem? that to many small chunks are reserved so my little "bigger" one fails? anyone have any input?

Or is there a problem in udptl.c? I dont see anything clear in it about memory / port allocation that I can change.

I will try it with trying to allow 2mb chunks of memory to be allocated instead of 1mb in kernel and see if that helps.


Administrator has disabled public posting
admin
useravatar
User Info

Re: Memory allocation problem

Hi nunne,

>asterisk: page allocation failure. order:6, mode:0xd0
>DMA: 401*4kB 277*8kB 291*16kB 322*32kB 320*64kB 306*128kB 1*256kB 0*512kB 0*1024kB >0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 78684kB

It suggests memory fragmentation. You have lot of free small blocks and cannot allocate 256kB one. mISDN is more prone to memory fragmentation.

Are you using the default SLUB memory allocator in Switchfin?
If not please try it.

Best Regards
Dimitar


Administrator has disabled public posting
nunne
useravatar
User Info

Re: Memory allocation problem

I have not changed anything of this in kernel. Where in the kernel config is this?

And is there an easy way to manually re-allocate the blocks?

Because in latests SVN it will still crash. I havent even built mISDN for the build Im using on the BR4 now just to try.

But after hammering 10+10 calls per minute for maybe 6-7h it will SEGV.
The system becomes unresponsive or at least asterisk crashes like mad.

Im thinking its the memory allocation thing that makes it crash, because after it starts SEGV i see this in the asterisk console:


Code:

[Mar 27 12:03:38] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:00] WARNING[856]: chan_sip.c:3622 retrans_pkt: Retransmission timeout reached on transmission 9c3d09ab-4c5bff160bf22dd136810080f0d179e5@192.168.3.35 for seqno 2 (Critical Response) -- See https://w[Mar 27 12:04:00] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:02] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:03] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:05] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:06] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:09] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:11] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:12] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:14] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)
[Mar 27 12:04:15] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

and it just goes on like this "forever". This is in asterisk 1.8.. But it's the same in 1.4 (I tired 1.8 yesterday to see if it was the same behaviour smile).



And in the shell i get a lot of the "memory allocation failure", where is also says that it's out of memory. And that it kills "asterisk process XXX", kills dropbear etc. etc.

So I guess here something weird happens.. Because killing dropbear leaves access to the pbx "hard" big_smile


Administrator has disabled public posting
admin
useravatar
User Info

Re: Memory allocation problem

Hi Nunne,

I still would not use Asterisk 1.8 in the production boxes.
Do you have CONFIG_SLUB=y in your config.linux-2.6.x ?
mISDN is compiled by default on BR4 Appliance.

Best Regards
Dimitar


Administrator has disabled public posting
nunne
useravatar
User Info

Re: Memory allocation problem

Hi!

I actually choose SLOB instead of SLUB (found it in kernel config). And it actually produced better results!
Maybe it's because Im on 128MB RAM instead of 64MB?

It still produced memory leaks (this on 1.8), but it took a lot longer.. Almost 24h.
But maybe my test is a bit unrealistic.

Plus i use spooler-module.. it's disabled by default. So maybe it has memory leaks problems?

I engage in 10 individual calls plus 1 group call of 10 phones per minute. That is 43200 "channels" in 24 hours.
Maybe this is a to big peak for the box? and of course is unrealstic since only "smaller" customers use this box.

What do you think? Maybe I will be golden with just putting it out?
Maybe I should try to manually disable mISDN? (we seldom use it anyway.. only use it as a SIP-to-SIP pbx)


Administrator has disabled public posting
admin
useravatar
User Info

Re: Memory allocation problem

Hi Nunne,

If you don't use mISDN disable it before compilation.
Strange. My study in the past showed that SLUB is more stable in terms of memory fragmentation.
IP08 was used for the tests.

Best Regards
Dimitar


Administrator has disabled public posting

Board Info

Board Stats:   Total Users: 2587  Total Topics: 299  Total Polls: 1  Total Posts: 1727  Dormant
User Info:   Newest User :  user2553   Members Online: 0   Guests Online: 216
Online  There are no members online
Topic
New
Locked
Topic
New
Locked
Sticky
Active
New/Active
Sticky
Active
New/Active
New/Closed
New Sticky
Closed/Active
New/Locked
New Sticky
Locked/Active
Active/Sticky
Sticky/Locked
Sticky Active Locked
Active/Sticky
Sticky/Locked
Sticky/Active/Locked