General discussions

JRPassphrase Registration Control

TODO list for each hardware target can be found as sticky topic in the corresponding forum

Page: 1

Post tools
- Subscribe

nunne: Posts: 28

Memory allocation problem

Hi,

I have been seeing a memory allocation problem in the latest versions of switchfin.
I dont see this problem in earlier version, but then asterisk will fail on other stuff like dialing more than 3-4 panasonic sip phones at once

Luckily the latest version of switchfin will not crash when memory allocation fails (like it did some revisions ago!).

Here is the serial output from the shell:

Code:

asterisk: page allocation failure. order:6, mode:0xd0

Stack info:

 SP: [0x034bfdac] ��.

 Memory from 0x034bfda0 to 034c0000

034bfda0: 000000d0  034bfdac  00000000 [001db300] 00035dae  063fe5c0  00000000  000200d0 

034bfdc0: 063fe7a0  00000006  000000d0  00000001  00000042  001da8d4  00000000  034be008 

034bfde0: 00000010  00000050  00000040  034be000  034be000  01429780  000200d0  001db304 

034bfe00: 00000040  00000000  063fe5c0  00000000  00000040  00000010  00000000  001da8d4 

034bfe20: 00000000  00000000  00000000  001da8d4  03ae86e0  0003dd08  00029000  0277361c 

034bfe40: 0277361c  00000006  00000073  00000000  04000021  034bfe6c  00b2fd30  00000010 

034bfe60: 00000000  00000001  04000021  03ae86e0  0003e062  00000000  0277361c  00000000 

034bfe80: 00000003  00000073  00000000  04000021  02c9cf91  ffffffc0  00029000  00000004 

034bfea0: 04000021  00000000  034bfeb0  001cd210  00000004  034ba1b8  0003e3fc  00000000 

034bfec0: 00029000  05a61250  034be000  00000000  04000021  00000003  00000000  05a61250 

034bfee0: 00029000  00000003  04000021  00000000  ffa008be  0003e3b8  000000c0  00000000 

034bff00: ffffe000  00000000  00000000  05a61250  05a60b98  05a60b9c  04000021  00000000 

034bff20: 00000000  0418d2c8  00008000  00002000  00000000  034c0000  0418d2c8  0418d2c8 

034bff40: 041ac02e  ffa00e44  02003065  0419bb07  05be23b7  0419bb06  05be23b6  00000000 

034bff60: 00000000  00000000  00000000  00000000  00000000  00000000  7ffff000  000000c0 

034bff80: 00000137  00000000  00000000  00000000  00000000  0000005b  00001802  00000001 

034bffa0: ffffffc0  00000006  034b9f10  00000001  00000000  034ba1ac  034ba1b8  05a61250 

034bffc0: 0594e8d8  05a61250  00000000  05977718  000000c0  00000000  05a60b9c  00000000 

034bffe0: 00000000  04000021  00000003  00029000  00000000  00000000  000000c0  00000006 

Return addresses in stack:

Mem-Info:

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

active_anon:0 inactive_anon:0 isolated_anon:0

 active_file:6 inactive_file:4 isolated_file:0

 unevictable:90 dirty:2 writeback:0 unstable:0

 free:19676 slab_reclaimable:86 slab_unreclaimable:395

 mapped:0 shmem:0 pagetables:0 bounce:0

DMA free:78704kB min:1300kB low:1624kB high:1948kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:16kB unevictable:360kB isolated(anon):0kB o

lowmem_reserve[]: 0 0 0

DMA: 401*4kB 277*8kB 291*16kB 322*32kB 320*64kB 306*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 78684kB

100 total pagecache pages

26624 pages RAM

760 pages reserved

8 pages shared

6117 pages non-shared

Allocation of length 167936 from process 13592 (asterisk) failed

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

active_anon:0 inactive_anon:0 isolated_anon:0

 active_file:6 inactive_file:4 isolated_file:0

 unevictable:90 dirty:2 writeback:0 unstable:0

 free:19671 slab_reclaimable:86 slab_unreclaimable:400

 mapped:0 shmem:0 pagetables:0 bounce:0

DMA free:78668kB min:1300kB low:1624kB high:1948kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:16kB unevictable:360kB isolated(anon):0kB o

lowmem_reserve[]: 0 0 0

DMA: 401*4kB 277*8kB 290*16kB 322*32kB 320*64kB 306*128kB 1*256kB 0*512kB 0*1024k

100 total pagecache pages

and it spews out these errors. the call doesnt really seem to "fail".. but I have not verified it. I am currently stresstesting the system, so that is why I see this error after just an hour or so. (making about 10 calls/minute to different phones with a call file to /var/spool/asterisk/outgoing).

and in earlier versions i never really got anything usefull from my asterisk log. but now I did! probably because it didnt crash

Code:

[Mar 27 07:12:08] ERROR[13071]: /root/2012-03-26/switchfin/build_br4/asterisk-1.4.42/include/asterisk:369 _ast_calloc: Memory Allocation Failure in function ast_udptl_new_with_bindaddr at line 753 of udptl.c

Sounds like this might be the trouble? is it running out of "bigger" DMA-blocks? My kernel settings are default for the BR4-target. And it says its allowed to allocate 1MB of memory, and it fails on chunks of 160-210kb of memory. While there is 75-80mb of free memory (my BR4 board has 128mb memory instead of 64mb).

But is it a DMA allocation problem? that to many small chunks are reserved so my little "bigger" one fails? anyone have any input?

Or is there a problem in udptl.c? I dont see anything clear in it about memory / port allocation that I can change.

I will try it with trying to allow 2mb chunks of memory to be allocated instead of 1mb in kernel and see if that helps.

Administrator has disabled public posting

2 Top

Mar-28-12 11:11:00

admin Posts: 356	Re: Memory allocation problem Hi nunne, >asterisk: page allocation failure. order:6, mode:0xd0 >DMA: 4014kB 2778kB 29116kB 32232kB 32064kB 306128kB 1256kB 0512kB 01024kB >02048kB 04096kB 08192kB 016384kB 032768kB = 78684kB It suggests memory fragmentation. You have lot of free small blocks and cannot allocate 256kB one. mISDN is more prone to memory fragmentation. Are you using the default SLUB memory allocator in Switchfin? If not please try it. Best Regards Dimitar

	Administrator has disabled public posting

3 Top

Mar-28-12 14:14:24

nunne: Posts: 28

Re: Memory allocation problem

I have not changed anything of this in kernel. Where in the kernel config is this?

And is there an easy way to manually re-allocate the blocks?

Because in latests SVN it will still crash. I havent even built mISDN for the build Im using on the BR4 now just to try.

But after hammering 10+10 calls per minute for maybe 6-7h it will SEGV.
The system becomes unresponsive or at least asterisk crashes like mad.

Im thinking its the memory allocation thing that makes it crash, because after it starts SEGV i see this in the asterisk console:

Code:

[Mar 27 12:03:38] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:00] WARNING[856]: chan_sip.c:3622 retrans_pkt: Retransmission timeout reached on transmission 9c3d09ab-4c5bff160bf22dd136810080f0d179e5@192.168.3.35 for seqno 2 (Critical Response) -- See https://w[Mar 27 12:04:00] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:02] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:03] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:05] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:06] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:09] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:11] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:12] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:14] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

[Mar 27 12:04:15] WARNING[858]: pbx_spool.c:379 launch_service: Unable to create thread :( (returned error: 11)

and it just goes on like this "forever". This is in asterisk 1.8.. But it's the same in 1.4 (I tired 1.8 yesterday to see if it was the same behaviour ).

And in the shell i get a lot of the "memory allocation failure", where is also says that it's out of memory. And that it kills "asterisk process XXX", kills dropbear etc. etc.

So I guess here something weird happens.. Because killing dropbear leaves access to the pbx "hard"

Administrator has disabled public posting

4 Top

Mar-29-12 12:28:23

admin Posts: 356	Re: Memory allocation problem Hi Nunne, I still would not use Asterisk 1.8 in the production boxes. Do you have CONFIG_SLUB=y in your config.linux-2.6.x ? mISDN is compiled by default on BR4 Appliance. Best Regards Dimitar

	Administrator has disabled public posting

5 Top

Mar-29-12 17:08:29

nunne Posts: 28	Re: Memory allocation problem Hi! I actually choose SLOB instead of SLUB (found it in kernel config). And it actually produced better results! Maybe it's because Im on 128MB RAM instead of 64MB? It still produced memory leaks (this on 1.8), but it took a lot longer.. Almost 24h. But maybe my test is a bit unrealistic. Plus i use spooler-module.. it's disabled by default. So maybe it has memory leaks problems? I engage in 10 individual calls plus 1 group call of 10 phones per minute. That is 43200 "channels" in 24 hours. Maybe this is a to big peak for the box? and of course is unrealstic since only "smaller" customers use this box. What do you think? Maybe I will be golden with just putting it out? Maybe I should try to manually disable mISDN? (we seldom use it anyway.. only use it as a SIP-to-SIP pbx)

	Administrator has disabled public posting

6 Top

Mar-29-12 19:14:37

admin Posts: 356	Re: Memory allocation problem Hi Nunne, If you don't use mISDN disable it before compilation. Strange. My study in the past showed that SLUB is more stable in terms of memory fragmentation. IP08 was used for the tests. Best Regards Dimitar

	Administrator has disabled public posting

Page: 1

Index
» Developers
» General discussions
» Memory allocation problem

Most Active Users

admin Albi90 Gilles Chewi luckman212 ckleea switchfinreseller spiga

Board Info

Board Stats: Total Users: 2587 Total Topics: 299 Total Polls: 1 Total Posts: 1727 Dormant
User Info: Newest User : user2553 Members Online: 0 Guests Online: 258

Online There are no members online

Topic

New

Locked

Sticky

Active

New/Active

New/Locked

New Sticky

Locked/Active

Active/Sticky

Sticky/Locked

Sticky/Active/Locked