This forum uses cookies
This forum makes use of cookies to store your login information if you are registered, and your last visit if you are not. Cookies are small text documents stored on your computer; the cookies set by this forum can only be used on this website and pose no security risk. Cookies on this forum also track the specific topics you have read and when you last read them. Please confirm whether you accept or reject these cookies being set.

A cookie will be stored in your browser regardless of choice to prevent you being asked this question again. You will be able to change your cookie settings at any time using the link in the footer.

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Kernel hang with Armbian Bionic Linux 5.4.10-rockchip
#1
Hello TB community,
I try to use one of my TB with HomeAssistant on Armbian. Everything is installed and running, but after 1 or 2 days the board runs into a kernel hang.
Code:
_____ _       _             _                         _
|_   _(_)_ __ | | _____ _ __| |__   ___   __ _ _ __ __| |
 | | | | '_ \| |/ / _ \ '__| '_ \ / _ \ / _` | '__/ _` |
 | | | | | | |   <  __/ |  | |_) | (_) | (_| | | | (_| |
 |_| |_|_| |_|_|\_\___|_|  |_.__/ \___/ \__,_|_|  \__,_|
                                                       
Welcome to Armbian Bionic with Linux 5.4.10-rockchip

System load:   1.76 0.49 0.17   Up time:       0 min
Memory usage:  5 % of 2001MB IP:            192.168.1.58
CPU temp:      67°C          
Usage of /:    15% of 29G  

I'm a little bit lost. Someone here, who is able to see some reason from the backtrace I post here?

Code:
Jan 28 23:47:41 localhost kernel: [58823.203747] mmc_host mmc0: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
Jan 28 23:47:41 localhost kernel: [58823.250153] mmc_host mmc0: Bus speed (slot 0) = 148500000Hz (slot req 150000000Hz, actual 148500000HZ div = 0)
Jan 28 23:47:41 localhost kernel: [58823.660383] dwmmc_rockchip ff0c0000.dwmmc: Successfully tuned phase to 159
Jan 28 23:47:58 localhost kernel: [58841.607562] mmc_host mmc0: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
Jan 28 23:47:58 localhost kernel: [58841.651719] mmc_host mmc0: Bus speed (slot 0) = 148500000Hz (slot req 150000000Hz, actual 148500000HZ div = 0)
Jan 28 23:47:58 localhost kernel: [58842.064205] dwmmc_rockchip ff0c0000.dwmmc: Successfully tuned phase to 160
Jan 28 23:48:15 localhost kernel: [58858.566618] mmc_host mmc0: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
Jan 28 23:48:15 localhost kernel: [58858.609925] mmc_host mmc0: Bus speed (slot 0) = 148500000Hz (slot req 150000000Hz, actual 148500000HZ div = 0)
Jan 28 23:48:15 localhost kernel: [58859.020075] dwmmc_rockchip ff0c0000.dwmmc: Successfully tuned phase to 160
[59052.625321] Out of memory: Killed process 2443 (node) total-vm:2343840kB, anon-rss:1002360kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:2196kB oom_score_adj:0
Jan 28 23:51:28 localhost kernel: [59052.624730] node invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan 28 23:51:30 localhost kernel: [59052.624739] CPU: 0 PID: 7114 Comm: node Tainted: G         C        5.4.10-rockchip #19.11.8
Jan 28 23:51:30 localhost kernel: [59052.624742] Hardware name: Rockchip (Device Tree)
Jan 28 23:51:30 localhost kernel: [59052.624744] Backtrace:
Jan 28 23:51:30 localhost kernel: [59052.624758] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
Jan 28 23:51:30 localhost kernel: [59052.624762]  r7:600e0013 r6:c16f2140 r5:00000000 r4:c16f2140
Jan 28 23:51:30 localhost kernel: [59052.624769] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
Jan 28 23:51:30 localhost kernel: [59052.624777] [<c0eef804>] (dump_stack) from [<c02516e8>] (dump_header+0x64/0x214)
Jan 28 23:51:30 localhost kernel: [59052.624781]  r7:c11c4b58 r6:edd1fc80 r5:d0062d00 r4:ea501db0
Jan 28 23:51:30 localhost kernel: [59052.624786] [<c0251684>] (dump_header) from [<c02509bc>] (oom_kill_process+0x180/0x18c)
Jan 28 23:51:30 localhost kernel: [59052.624790]  r7:c11c4b58 r6:ea501db0 r5:d0063290 r4:d0062d00
Jan 28 23:51:30 localhost kernel: [59052.624796] [<c025083c>] (oom_kill_process) from [<c02513cc>] (out_of_memory+0x118/0x344)
Jan 28 23:51:30 localhost kernel: [59052.624800]  r7:c160a940 r6:ea501db0 r5:c1606788 r4:d0062d00
Jan 28 23:51:30 localhost kernel: [59052.624808] [<c02512b4>] (out_of_memory) from [<c029db14>] (__alloc_pages_nodemask+0x1184/0x13b4)
Jan 28 23:51:30 localhost kernel: [59052.624812]  r7:c16f41d0 r6:00000355 r5:00002aa4 r4:00000000
Jan 28 23:51:30 localhost kernel: [59052.624818] [<c029c990>] (__alloc_pages_nodemask) from [<c0282600>] (wp_page_copy+0x78/0x5c8)
Jan 28 23:51:30 localhost kernel: [59052.624823]  r10:edeb124c r9:ffffe000 r8:c1606788 r7:ce5cd8a0 r6:ce5cd8a0 r5:efd16278
Jan 28 23:51:30 localhost kernel: [59052.624826]  r4:ea501eb4
Jan 28 23:51:30 localhost kernel: [59052.624830] [<c0282588>] (wp_page_copy) from [<c02851f8>] (do_wp_page+0xb0/0x608)
Jan 28 23:51:30 localhost kernel: [59052.624835]  r10:edeb124c r9:ffffe000 r8:00000040 r7:00000055 r6:ce5cd8a0 r5:c1606788
Jan 28 23:51:30 localhost kernel: [59052.624837]  r4:ea501eb4
Jan 28 23:51:30 localhost kernel: [59052.624843] [<c0285148>] (do_wp_page) from [<c0286a6c>] (handle_mm_fault+0x6a8/0xc58)
Jan 28 23:51:30 localhost kernel: [59052.624847]  r9:ffffe000 r8:00000040 r7:00000055 r6:c1606788 r5:01caa000 r4:ce5cd8a0
Jan 28 23:51:30 localhost kernel: [59052.624854] [<c02863c4>] (handle_mm_fault) from [<c011695c>] (do_page_fault+0x12c/0x3e0)
Jan 28 23:51:30 localhost kernel: [59052.624858]  r10:edeb124c r9:ce5cd8a0 r8:0000080f r7:01caa624 r6:edeb1200 r5:d9258f00
Jan 28 23:51:30 localhost kernel: [59052.624861]  r4:ea501fb0
Jan 28 23:51:30 localhost kernel: [59052.624865] [<c0116830>] (do_page_fault) from [<c0116df8>] (do_DataAbort+0x4c/0xcc)
Jan 28 23:51:30 localhost kernel: [59052.624870]  r10:00000000 r9:bec5efe0 r8:ea501fb0 r7:01caa624 r6:c0116830 r5:0000080f
Jan 28 23:51:30 localhost kernel: [59052.624872]  r4:c160c74c
Jan 28 23:51:30 localhost kernel: [59052.624877] [<c0116dac>] (do_DataAbort) from [<c0101e20>] (__dabt_usr+0x40/0x60)
Jan 28 23:51:30 localhost kernel: [59052.624880] Exception stack(0xea501fb0 to 0xea501ff8)
Jan 28 23:51:30 localhost kernel: [59052.624884] 1fa0:                                     00000000 00000002 00000000 ffffffff
Jan 28 23:51:30 localhost kernel: [59052.624889] 1fc0: 01caa624 bec5efe0 00000001 bec5f000 b6daf000 bec5efe0 00000000 bec5efb4
Jan 28 23:51:30 localhost kernel: [59052.624893] 1fe0: 00000006 bec5efa8 b6dbfe43 00476028 000b0010 ffffffff
Jan 28 23:51:30 localhost kernel: [59052.624898]  r8:10c5387d r7:10c5387d r6:ffffffff r5:000b0010 r4:00476028
Jan 28 23:51:30 localhost kernel: [59052.624900] Mem-Info:
Jan 28 23:51:30 localhost kernel: [59052.624909] active_anon:203230 inactive_anon:67703 isolated_anon:64
Jan 28 23:51:30 localhost kernel: [59052.624909]  active_file:595 inactive_file:439 isolated_file:0
Jan 28 23:51:30 localhost kernel: [59052.624909]  unevictable:0 dirty:11 writeback:1 unstable:0
Jan 28 23:51:30 localhost kernel: [59052.624909]  slab_reclaimable:4459 slab_unreclaimable:7241
Jan 28 23:51:30 localhost kernel: [59052.624909]  mapped:1369 shmem:3338 pagetables:2844 bounce:0
Jan 28 23:51:30 localhost kernel: [59052.624909]  free:10995 free_pcp:4 free_cma:100
Jan 28 23:51:30 localhost kernel: [59052.624915] Node 0 active_anon:812920kB inactive_anon:270812kB active_file:2380kB inactive_file:1756kB unevictable:0kB isolated(anon):256kB isolated(file):0kB mapped:5476kB dirty:44kB writeback:4kB shmem:13352kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Jan 28 23:51:30 localhost kernel: [59052.624924] Normal free:42240kB min:3412kB low:4264kB high:5116kB active_anon:370548kB inactive_anon:15228kB active_file:764kB inactive_file:296kB unevictable:0kB writepending:8kB present:786432kB managed:738968kB mlocked:0kB kernel_stack:2000kB pagetables:11376kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:0kB
Jan 28 23:51:30 localhost kernel: [59052.624926] lowmem_reserve[]: 0 10240 10240
Jan 28 23:51:30 localhost kernel: [59052.624935] HighMem free:1740kB min:5816kB low:7328kB high:8840kB active_anon:442856kB inactive_anon:255488kB active_file:1736kB inactive_file:1636kB unevictable:0kB writepending:40kB present:1310720kB managed:1310720kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:400kB
Jan 28 23:51:30 localhost kernel: [59052.624937] lowmem_reserve[]: 0 0 0
Jan 28 23:51:30 localhost kernel: [59052.624942] Normal: 646*4kB (UME) 291*8kB (UME) 266*16kB (UME) 136*32kB (UME) 64*64kB (UME) 25*128kB (UME) 3*256kB (UM) 1*512kB (U) 2*1024kB (UM) 9*2048kB (UM) 0*4096kB = 42576kB
Jan 28 23:51:30 localhost kernel: [59052.624964] HighMem: 485*4kB (UMC) 35*8kB (UMC) 16*16kB (UMC) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2476kB
Jan 28 23:51:30 localhost kernel: [59052.624981] 4466 total pagecache pages
Jan 28 23:51:30 localhost kernel: [59052.624985] 3 pages in swap cache
Jan 28 23:51:30 localhost kernel: [59052.624988] Swap cache stats: add 307636, delete 307623, find 3269/40954
Jan 28 23:51:30 localhost kernel: [59052.624990] Free swap  = 0kB
Jan 28 23:51:30 localhost kernel: [59052.624992] Total swap = 1024840kB
Jan 28 23:51:30 localhost kernel: [59052.624994] 524288 pages RAM
Jan 28 23:51:30 localhost kernel: [59052.624996] 327680 pages HighMem/MovableOnly
Jan 28 23:51:30 localhost kernel: [59052.624998] 11866 pages reserved
Jan 28 23:51:30 localhost kernel: [59052.625000] 4096 pages cma reserved
Jan 28 23:51:30 localhost kernel: [59052.625002] Tasks state (memory values in pages):
Jan 28 23:51:30 localhost kernel: [59052.625004] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 28 23:51:30 localhost kernel: [59052.625014] [    319]     0   319     6740      554    24576      100             0 systemd-journal
Jan 28 23:51:30 localhost kernel: [59052.625019] [    340]     0   340      822        0    10240       38             0 blkmapd
Jan 28 23:51:30 localhost kernel: [59052.625023] [    349]     0   349     3596       11    16384      258         -1000 systemd-udevd
Jan 28 23:51:30 localhost kernel: [59052.625027] [    395]     0   395     1306       15    14336       79             0 rpcbind
Jan 28 23:51:30 localhost kernel: [59052.625032] [    397]     0   397      629        0    10240       43             0 rpc.idmapd
Jan 28 23:51:30 localhost kernel: [59052.625036] [    399]     0   399     1811        0    12288      768             0 haveged
Jan 28 23:51:30 localhost kernel: [59052.625040] [    402]   101   402     2051       20    16384       95             0 systemd-resolve
Jan 28 23:51:30 localhost kernel: [59052.625045] [    844]     0   844     9930      170    30720      131             0 accounts-daemon
Jan 28 23:51:30 localhost kernel: [59052.625067] [    846]   111   846     1282       54    14336       63             0 avahi-daemon
Jan 28 23:51:30 localhost kernel: [59052.625071] [    850]     0   850     3606        0    22528      224             0 cupsd
Jan 28 23:51:30 localhost kernel: [59052.625075] [    859]   102   859     6173      162    20480      164             0 rsyslogd
Jan 28 23:51:30 localhost kernel: [59052.625080] [    865]     0   865    14852       73    38912      247             0 udisksd
Jan 28 23:51:30 localhost kernel: [59052.625084] [    870]     0   870     1252       22    12288       38             0 cron
Jan 28 23:51:30 localhost kernel: [59052.625089] [    871]   103   871     1443      108    12288       69          -900 dbus-daemon
Jan 28 23:51:30 localhost kernel: [59052.625093] [    912]   111   912     1207        3    12288       66             0 avahi-daemon
Jan 28 23:51:30 localhost kernel: [59052.625098] [    955]     0   955     1986       50    18432       91             0 wpa_supplicant
Jan 28 23:51:30 localhost kernel: [59052.625102] [    957]     0   957    16012      324    49152      280             0 NetworkManager
Jan 28 23:51:30 localhost kernel: [59052.625107] [    964]     0   964     2024       24    14336      103             0 systemd-logind
Jan 28 23:51:30 localhost kernel: [59052.625111] [    965]     0   965     1205        1    14336       51             0 bash
Jan 28 23:51:30 localhost kernel: [59052.625115] [    983]     0   983     9566        0    30720      227             0 polkitd
Jan 28 23:51:30 localhost kernel: [59052.625120] [   1165]     0  1165     8553      143    26624       86             0 wiring-mqtt
Jan 28 23:51:30 localhost kernel: [59052.625124] [   1166]     0  1166     8568      143    24576       37             0 pwm16-mqtt
Jan 28 23:51:30 localhost kernel: [59052.625129] [   1167]     0  1167    36394      178    57344       88             0 cover-mqtt
Jan 28 23:51:30 localhost kernel: [59052.625133] [   1177]     0  1177     8833       95    26624       85             0 wdog-mqtt
Jan 28 23:51:30 localhost kernel: [59052.625137] [   1185]     0  1185     1105       25    14336      131             0 rpc.mountd
Jan 28 23:51:30 localhost kernel: [59052.625142] [   1204]   108  1204     3293       41    16384       66             0 chronyd
Jan 28 23:51:30 localhost kernel: [59052.625146] [   1209]     0  1209     2099        0    16384      131         -1000 sshd
Jan 28 23:51:30 localhost kernel: [59052.625150] [   1298]     0  1298    13252       22    26624      234             0 nginx
Jan 28 23:51:30 localhost kernel: [59052.625155] [   1299]    33  1299    13358       90    30720      295             0 nginx
Jan 28 23:51:30 localhost kernel: [59052.625159] [   1300]    33  1300    13290       31    26624      267             0 nginx
Jan 28 23:51:30 localhost kernel: [59052.625163] [   1301]    33  1301    13290       21    26624      294             0 nginx
Jan 28 23:51:30 localhost kernel: [59052.625167] [   1302]    33  1302    13290       31    26624      267             0 nginx
Jan 28 23:51:30 localhost kernel: [59052.625171] [   1345]     0  1345      357        0    10240       36             0 rtk_hciattach
Jan 28 23:51:30 localhost kernel: [59052.625176] [   1386]     0  1386      351        0     8192       15             0 rngd
Jan 28 23:51:30 localhost kernel: [59052.625180] [   1389]  1000  1389     2426       73    20480      149             0 systemd
Jan 28 23:51:30 localhost kernel: [59052.625185] [   1393]   114  1393     1513      177    14336       69             0 mosquitto
Jan 28 23:51:30 localhost kernel: [59052.625189] [   1394]  1000  1394     3015       11    16384      383             0 (sd-pam)
Jan 28 23:51:30 localhost kernel: [59052.625194] [   1582]  1000  1582     1346        0    12288       84             0 dbus-daemon
Jan 28 23:51:30 localhost kernel: [59052.625198] [   1590]     0  1590     1407        0    14336       82             0 bluetoothd
Jan 28 23:51:30 localhost kernel: [59052.625203] [   1618]  1000  1618     2674       26    20480      107             0 gconfd-2
Jan 28 23:51:30 localhost kernel: [59052.625206] [   1782]     0  1782     1579       36    16384      178             0 dhclient
Jan 28 23:51:30 localhost kernel: [59052.625211] [   1931]   999  1931    93187    14945   169984     2240             0 hass
Jan 28 23:51:30 localhost kernel: [59052.625215] [   1941]     0  1941     1015        0    12288       30             0 agetty
Jan 28 23:51:30 localhost kernel: [59052.625220] [   2287]  1000  2287   102323      156    61440      432             0 pulseaudio
Jan 28 23:51:30 localhost kernel: [59052.625225] [   2298]  1000  2298     3438        0    22528      158             0 gconf-helper
Jan 28 23:51:30 localhost kernel: [59052.625229] [   2302]     0  2302     1774        1    14336      119             0 login
Jan 28 23:51:30 localhost kernel: [59052.625233] [   2397]  1000  2397     1370        1    16384      237             0 bash
Jan 28 23:51:30 localhost kernel: [59052.625237] [   2415]     0  2415     1692        1    16384      102             0 sudo
Jan 28 23:51:30 localhost kernel: [59052.625242] [   2416]     0  2416      852        9    10240       12             0 tail
Jan 28 23:51:30 localhost kernel: [59052.625246] [   2421]  1000  2421    38106        1   167936     9337             0 node
Jan 28 23:51:30 localhost kernel: [59052.625251] [   2442]  1000  2442      368        0     8192       16             0 sh
Jan 28 23:51:30 localhost kernel: [59052.625255] [   2443]  1000  2443   585960   250590  2248704   235271             0 node
Jan 28 23:51:30 localhost kernel: [59052.625261] [   7114]  1000  7114   585960   250593  2207744   235271             0 node
Jan 28 23:51:30 localhost kernel: [59052.625264] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=node,pid=2443,uid=1000
Jan 28 23:51:30 localhost kernel: [59052.625321] Out of memory: Killed process 2443 (node) total-vm:2343840kB, anon-rss:1002360kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:2196kB oom_score_adj:0
Jan 28 23:51:30 localhost kernel: [59052.753075] oom_reaper: reaped process 2443 (node), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
tail: /var/log/kern.log: file truncated
Jan 29 14:14:55 localhost kernel: [110857.457880] mmc_host mmc0: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
Jan 29 14:14:55 localhost kernel: [110857.506153] mmc_host mmc0: Bus speed (slot 0) = 148500000Hz (slot req 150000000Hz, actual 148500000HZ div = 0)
Jan 29 14:14:55 localhost kernel: [110857.916042] dwmmc_rockchip ff0c0000.dwmmc: Successfully tuned phase to 159
[118052.915383] INFO: task kworker/0:0:11908 blocked for more than 120 seconds.
[118052.923478]       Tainted: G         C        5.4.10-rockchip #19.11.8
[118052.930945] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[118052.939846] kworker/0:0     D    0 11908      2 0x00000000
[118052.946279] Workqueue: events_freezable mmc_rescan
[118052.951895] Backtrace:
[118052.954976] [<c0f08ab0>] (__schedule) from [<c0f0936c>] (schedule+0x60/0xf4)
[118052.963285]  r10:ed52ac00 r9:ffffe000 r8:00000000 r7:00000000 r6:60070013 r5:c90d9680
[118052.972364]  r4:ffffe000
[118052.975457] [<c0f0930c>] (schedule) from [<c0b50bf8>] (__mmc_claim_host+0xd8/0x230)
[118052.984448]  r5:ed52ae68 r4:00000002
[118052.988752] [<c0b50b20>] (__mmc_claim_host) from [<c0b50d88>] (mmc_get_card+0x38/0x3c)
[118052.997952]  r10:00000000 r9:c16fbb10 r8:00000000 r7:eed97d00 r6:ed52ae68 r5:00000000
[118053.006966]  r4:edcef800
[118053.009983] [<c0b50d50>] (mmc_get_card) from [<c0b5aa44>] (mmc_sd_detect+0x24/0x7c)
[118053.018699]  r5:ed52ac00 r4:ed52ac00
[118053.022821] [<c0b5aa20>] (mmc_sd_detect) from [<c0b539a8>] (mmc_rescan+0xdc/0x38c)
[118053.031403]  r5:ed52ac00 r4:ed52aee4
[118053.035661] [<c0b538cc>] (mmc_rescan) from [<c014335c>] (process_one_work+0x224/0x5d8)
[118053.044879]  r9:c16fbb10 r8:00000000 r7:eed97d00 r6:eed94700 r5:ecffad80 r4:ed52aee4
[118053.053746] [<c0143138>] (process_one_work) from [<c0143770>] (worker_thread+0x60/0x588)
[118053.063169]  r10:eed94700 r9:c1604900 r8:eed94724 r7:00000008 r6:eed94700 r5:ecffad94
[118053.072139]  r4:ecffad80
[118053.075180] [<c0143710>] (worker_thread) from [<c014a1ac>] (kthread+0x170/0x174)
[118053.083749]  r10:d0339e68 r9:c0143710 r8:ecffad80 r7:cf3be000 r6:00000000 r5:d034ecc0
[118053.092810]  r4:ceef4380
[118053.095753] [<c014a03c>] (kthread) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
[118053.104057] Exception stack(0xcf3bffb0 to 0xcf3bfff8)
[118053.109817] ffa0:                                     00000000 00000000 00000000 00000000
[118053.119235] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[118053.128628] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
[118053.136178]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c014a03c
[118053.145130]  r4:d034ecc0
[118053.148071] NMI backtrace for cpu 1
[118053.152076] CPU: 1 PID: 33 Comm: khungtaskd Tainted: G         C        5.4.10-rockchip #19.11.8
[118053.161997] Hardware name: Rockchip (Device Tree)
[118053.167350] Backtrace:
[118053.170185] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
[118053.178747]  r7:60000113 r6:c16f2140 r5:00000000 r4:c16f2140
[118053.185172] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
[118053.193353] [<c0eef804>] (dump_stack) from [<c0ef6818>] (nmi_cpu_backtrace+0x88/0xbc)
[118053.202210]  r7:00000000 r6:00000001 r5:00000001 r4:00000000
[118053.208628] [<c0ef6790>] (nmi_cpu_backtrace) from [<c0ef69b0>] (nmi_trigger_cpumask_backtrace+0x164/0x1a4)
[118053.219522]  r5:c160a49c r4:c010f608
[118053.223618] [<c0ef684c>] (nmi_trigger_cpumask_backtrace) from [<c011125c>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[118053.235581]  r9:c1755d78 r8:01c1323a r7:c1604900 r6:c160ac80 r5:00007f0b r4:c97b4140
[118053.244340] [<c011123c>] (arch_trigger_cpumask_backtrace) from [<c01f697c>] (watchdog+0x2cc/0x49c)
[118053.254460] [<c01f66b0>] (watchdog) from [<c014a1ac>] (kthread+0x170/0x174)
[118053.262342]  r10:ee543e38 r9:c01f66b0 r8:00000000 r7:ee7f2000 r6:00000000 r5:ee7ca240
[118053.271198]  r4:ee67b180
[118053.274128] [<c014a03c>] (kthread) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
[118053.282302] Exception stack(0xee7f3fb0 to 0xee7f3ff8)
[118053.288044] 3fa0:                                     00000000 00000000 00000000 00000000
[118053.297287] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[118053.306531] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[118053.314026]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c014a03c
[118053.322879]  r4:ee7ca240
[118053.325859] Sending NMI from CPU 1 to CPUs 0,2-3:
[118053.331331] NMI backtrace for cpu 2
[118053.331334] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G         C        5.4.10-rockchip #19.11.8
[118053.331335] Hardware name: Rockchip (Device Tree)
[118053.331336] PC is at arch_cpu_idle+0x48/0x4c
[118053.331337] LR is at arch_cpu_idle+0x44/0x4c
[118053.331338] pc : [<c0109f2c>]    lr : [<c0109f28>]    psr: 60070113
[118053.331339] sp : ee577f60  ip : ee577f70  fp : ee577f6c
[118053.331340] r10: c1504df0  r9 : 00000000  r8 : 00000000
[118053.331342] r7 : 00000004  r6 : c16067f0  r5 : c16067a8  r4 : ffffe000
[118053.331343] r3 : c011e5e0  r2 : eedbd3b4  r1 : 543703b0  r0 : 00000000
[118053.331344] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[118053.331345] Control: 10c5387d  Table: 0ec5c06a  DAC: 00000051
[118053.331347] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G         C        5.4.10-rockchip #19.11.8
[118053.331348] Hardware name: Rockchip (Device Tree)
[118053.331349] Backtrace:
[118053.331351] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
[118053.331352]  r7:60070193 r6:c16f2140 r5:00000000 r4:c16f2140
[118053.331354] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
[118053.331355] [<c0eef804>] (dump_stack) from [<c010a1ec>] (show_regs+0x1c/0x20)
[118053.331356]  r7:00000000 r6:00000007 r5:00000002 r4:ee577f10
[118053.331358] [<c010a1d0>] (show_regs) from [<c0ef6848>] (nmi_cpu_backtrace+0xb8/0xbc)
[118053.331359] [<c0ef6790>] (nmi_cpu_backtrace) from [<c0110a84>] (handle_IPI+0x7c/0x438)
[118053.331360]  r5:c1505690 r4:c1505690
[118053.331361] [<c0110a08>] (handle_IPI) from [<c0102398>] (gic_handle_irq+0x9c/0xa0)
[118053.331363]  r10:c1504df0 r9:f0803000 r8:ee577f10 r7:f0802000 r6:f080200c r5:c168ad2c
[118053.331364]  r4:c1607054
[118053.331365] [<c01022fc>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0xa8)
[118053.331366] Exception stack(0xee577f10 to 0xee577f58)
[118053.331367] 7f00:                                     00000000 543703b0 eedbd3b4 c011e5e0
[118053.331369] 7f20: ffffe000 c16067a8 c16067f0 00000004 00000000 00000000 c1504df0 ee577f6c
[118053.331370] 7f40: ee577f70 ee577f60 c0109f28 c0109f2c 60070113 ffffffff
[118053.331372]  r9:ee576000 r8:00000000 r7:ee577f44 r6:ffffffff r5:60070113 r4:c0109f2c
[118053.331373] [<c0109ee4>] (arch_cpu_idle) from [<c0f0e66c>] (default_idle_call+0x38/0x3c)
[118053.331374] [<c0f0e634>] (default_idle_call) from [<c015c8c4>] (do_idle+0x214/0x2c0)
[118053.331376] [<c015c6b0>] (do_idle) from [<c015cc94>] (cpu_startup_entry+0x28/0x2c)
[118053.331377]  r10:00000000 r9:410fc0d1 r8:0000406a r7:c170da40 r6:10c0387d r5:00000002
[118053.331378]  r4:00000089
[118053.331379] [<c015cc6c>] (cpu_startup_entry) from [<c01100f4>] (secondary_start_kernel+0x160/0x16c)
[118053.331381] [<c010ff94>] (secondary_start_kernel) from [<001028cc>] (0x1028cc)
[118053.331382]  r5:00000051 r4:2e56c06a
[118053.331410] NMI backtrace for cpu 0
[118053.331412] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.4.10-rockchip #19.11.8
[118053.331414] Hardware name: Rockchip (Device Tree)
[118053.331414] PC is at arch_cpu_idle+0x48/0x4c
[118053.331416] LR is at arch_cpu_idle+0x44/0x4c
[118053.331417] pc : [<c0109f2c>]    lr : [<c0109f28>]    psr: 600f0113
[118053.331418] sp : c1601ef8  ip : c1601f08  fp : c1601f04
[118053.331419] r10: c1504df0  r9 : 00000000  r8 : 00000000
[118053.331420] r7 : 00000001  r6 : c16067f0  r5 : c16067a8  r4 : ffffe000
[118053.331421] r3 : c011e5e0  r2 : eed953b4  r1 : 59ee62c8  r0 : 00000000
[118053.331423] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[118053.331424] Control: 10c5387d  Table: 097ec06a  DAC: 00000051
[118053.331426] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.4.10-rockchip #19.11.8
[118053.331427] Hardware name: Rockchip (Device Tree)
[118053.331428] Backtrace:
[118053.331430] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
[118053.331431]  r7:600f0193 r6:c16f2140 r5:00000000 r4:c16f2140
[118053.331433] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
[118053.331434] [<c0eef804>] (dump_stack) from [<c010a1ec>] (show_regs+0x1c/0x20)
[118053.331435]  r7:00000000 r6:00000007 r5:00000000 r4:c1601ea8
[118053.331436] [<c010a1d0>] (show_regs) from [<c0ef6848>] (nmi_cpu_backtrace+0xb8/0xbc)
[118053.331438] [<c0ef6790>] (nmi_cpu_backtrace) from [<c0110a84>] (handle_IPI+0x7c/0x438)
[118053.331439]  r5:c1505690 r4:c1505690
[118053.331440] [<c0110a08>] (handle_IPI) from [<c0102398>] (gic_handle_irq+0x9c/0xa0)
[118053.331441]  r10:c1504df0 r9:f0803000 r8:c1601ea8 r7:f0802000 r6:f080200c r5:c168ad2c
[118053.331442]  r4:c1607054
[118053.331444] [<c01022fc>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0xa8)
[118053.331445] Exception stack(0xc1601ea8 to 0xc1601ef0)
[118053.331446] 1ea0:                   00000000 59ee62c8 eed953b4 c011e5e0 ffffe000 c16067a8
[118053.331448] 1ec0: c16067f0 00000001 00000000 00000000 c1504df0 c1601f04 c1601f08 c1601ef8
[118053.331449] 1ee0: c0109f28 c0109f2c 600f0113 ffffffff
[118053.331450]  r9:c1600000 r8:00000000 r7:c1601edc r6:ffffffff r5:600f0113 r4:c0109f2c
[118053.331451] [<c0109ee4>] (arch_cpu_idle) from [<c0f0e66c>] (default_idle_call+0x38/0x3c)
[118053.331453] [<c0f0e634>] (default_idle_call) from [<c015c8c4>] (do_idle+0x214/0x2c0)
[118053.331454] [<c015c6b0>] (do_idle) from [<c015cc94>] (cpu_startup_entry+0x28/0x2c)
[118053.331456]  r10:c148fe58 r9:00000001 r8:00000000 r7:c148fe58 r6:00000001 r5:00000002
[118053.331456]  r4:000000cd
[118053.331458] [<c015cc6c>] (cpu_startup_entry) from [<c0f07a28>] (rest_init+0xd0/0xd8)
[118053.331459] [<c0f07958>] (rest_init) from [<c1400b68>] (arch_call_rest_init+0x18/0x1c)
[118053.331460]  r5:c170d794 r4:c1606788
[118053.331462] [<c1400b50>] (arch_call_rest_init) from [<c14010b8>] (start_kernel+0x4d4/0x504)
[118053.331463] [<c1400be4>] (start_kernel) from [<00000000>] (0x0)
[118053.331482] NMI backtrace for cpu 3
[118053.331484] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G         C        5.4.10-rockchip #19.11.8
[118053.331485] Hardware name: Rockchip (Device Tree)
[118053.331486] PC is at arch_cpu_idle+0x48/0x4c
[118053.331487] LR is at arch_cpu_idle+0x44/0x4c
[118053.331489] pc : [<c0109f2c>]    lr : [<c0109f28>]    psr: 600f0113
[118053.331490] sp : ee579f60  ip : ee579f70  fp : ee579f6c
[118053.331491] r10: c1504df0  r9 : 00000000  r8 : 00000000
[118053.331492] r7 : 00000008  r6 : c16067f0  r5 : c16067a8  r4 : ffffe000
[118053.331493] r3 : c011e5e0  r2 : eedd13b4  r1 : 1e7af900  r0 : 00000000
[118053.331495] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[118053.331496] Control: 10c5387d  Table: 0e59406a  DAC: 00000051
[118053.331497] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G         C        5.4.10-rockchip #19.11.8
[118053.331498] Hardware name: Rockchip (Device Tree)
[118053.331499] Backtrace:
[118053.331502] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
[118053.331503]  r7:600f0193 r6:c16f2140 r5:00000000 r4:c16f2140
[118053.331504] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
[118053.331505] [<c0eef804>] (dump_stack) from [<c010a1ec>] (show_regs+0x1c/0x20)
[118053.331507]  r7:00000000 r6:00000007 r5:00000003 r4:ee579f10
[118053.331508] [<c010a1d0>] (show_regs) from [<c0ef6848>] (nmi_cpu_backtrace+0xb8/0xbc)
[118053.331510] [<c0ef6790>] (nmi_cpu_backtrace) from [<c0110a84>] (handle_IPI+0x7c/0x438)
[118053.331510]  r5:c1505690 r4:c1505690
[118053.331512] [<c0110a08>] (handle_IPI) from [<c0102398>] (gic_handle_irq+0x9c/0xa0)
[118053.331513]  r10:c1504df0 r9:f0803000 r8:ee579f10 r7:f0802000 r6:f080200c r5:c168ad2c
[118053.331514]  r4:c1607054
[118053.331515] [<c01022fc>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0xa8)
[118053.331517] Exception stack(0xee579f10 to 0xee579f58)
[118053.331518] 9f00:                                     00000000 1e7af900 eedd13b4 c011e5e0
[118053.331519] 9f20: ffffe000 c16067a8 c16067f0 00000008 00000000 00000000 c1504df0 ee579f6c
[118053.331521] 9f40: ee579f70 ee579f60 c0109f28 c0109f2c 600f0113 ffffffff
[118053.331522]  r9:ee578000 r8:00000000 r7:ee579f44 r6:ffffffff r5:600f0113 r4:c0109f2c
[118053.331523] [<c0109ee4>] (arch_cpu_idle) from [<c0f0e66c>] (default_idle_call+0x38/0x3c)
[118053.331525] [<c0f0e634>] (default_idle_call) from [<c015c8c4>] (do_idle+0x214/0x2c0)
[118053.331526] [<c015c6b0>] (do_idle) from [<c015cc94>] (cpu_startup_entry+0x28/0x2c)
[118053.331527]  r10:00000000 r9:410fc0d1 r8:0000406a r7:c170da40 r6:10c0387d r5:00000003
[118053.331528]  r4:00000089
[118053.331530] [<c015cc6c>] (cpu_startup_entry) from [<c01100f4>] (secondary_start_kernel+0x160/0x16c)
[118053.331531] [<c010ff94>] (secondary_start_kernel) from [<001028cc>] (0x1028cc)
[118053.331532]  r5:00000051 r4:2e56c06a
[118053.332241] Kernel panic - not syncing: hung_task: blocked tasks
[118054.204959] CPU: 1 PID: 33 Comm: khungtaskd Tainted: G         C        5.4.10-rockchip #19.11.8
[118054.214874] Hardware name: Rockchip (Device Tree)
[118054.220228] Backtrace:
[118054.223062] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
[118054.231625]  r7:60000193 r6:c16f2140 r5:00000000 r4:c16f2140
[118054.238050] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
[118054.246230] [<c0eef804>] (dump_stack) from [<c0124ee0>] (panic+0x12c/0x334)
[118054.254114]  r7:c1604900 r6:c11bfdd4 r5:00000000 r4:c170dbb8
[118054.260534] [<c0124db4>] (panic) from [<c01f6988>] (watchdog+0x2d8/0x49c)
[118054.268221]  r3:00000000 r2:00000000 r1:ee6d8780 r0:c11bfdd4
[118054.274642]  r7:c1604900
[118054.277569] [<c01f66b0>] (watchdog) from [<c014a1ac>] (kthread+0x170/0x174)
[118054.285451]  r10:ee543e38 r9:c01f66b0 r8:00000000 r7:ee7f2000 r6:00000000 r5:ee7ca240
[118054.294302]  r4:ee67b180
[118054.297229] [<c014a03c>] (kthread) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
[118054.305401] Exception stack(0xee7f3fb0 to 0xee7f3ff8)
[118054.311143] 3fa0:                                     00000000 00000000 00000000 00000000
[118054.320378] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[118054.329623] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[118054.337118]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c014a03c
[118054.345971]  r4:ee7ca240
[118054.348900] CPU0: stopping
[118054.352023] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.4.10-rockchip #19.11.8
[118054.361748] Hardware name: Rockchip (Device Tree)
[118054.367097] Backtrace:
[118054.369929] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
[118054.378490]  r7:600b0193 r6:c16f2140 r5:00000000 r4:c16f2140
[118054.384912] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
[118054.393084] [<c0eef804>] (dump_stack) from [<c0110e14>] (handle_IPI+0x40c/0x438)
[118054.401450]  r7:00000000 r6:00000004 r5:c16fb9e4 r4:c1505690
[118054.407873] [<c0110a08>] (handle_IPI) from [<c0102398>] (gic_handle_irq+0x9c/0xa0)
[118054.416425]  r10:c1504df0 r9:f0803000 r8:c1601ea8 r7:f0802000 r6:f080200c r5:c168ad2c
[118054.425275]  r4:c1607054
[118054.428198] [<c01022fc>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0xa8)
[118054.436659] Exception stack(0xc1601ea8 to 0xc1601ef0)
[118054.442400] 1ea0:                   00000000 59ee88bc eed953b4 c011e5e0 ffffe000 c16067a8
[118054.451641] 1ec0: c16067f0 00000001 00000000 00000000 c1504df0 c1601f04 c1601f08 c1601ef8
[118054.460880] 1ee0: c0109f28 c0109f2c 600b0113 ffffffff
[118054.466621]  r9:c1600000 r8:00000000 r7:c1601edc r6:ffffffff r5:600b0113 r4:c0109f2c
[118054.475378] [<c0109ee4>] (arch_cpu_idle) from [<c0f0e66c>] (default_idle_call+0x38/0x3c)
[118054.484523] [<c0f0e634>] (default_idle_call) from [<c015c8c4>] (do_idle+0x214/0x2c0)
[118054.493279] [<c015c6b0>] (do_idle) from [<c015cc94>] (cpu_startup_entry+0x28/0x2c)
[118054.501839]  r10:c148fe58 r9:00000001 r8:00000000 r7:c148fe58 r6:00000001 r5:00000002
[118054.510689]  r4:000000cd
[118054.513614] [<c015cc6c>] (cpu_startup_entry) from [<c0f07a28>] (rest_init+0xd0/0xd8)
[118054.522371] [<c0f07958>] (rest_init) from [<c1400b68>] (arch_call_rest_init+0x18/0x1c)
[118054.531318]  r5:c170d794 r4:c1606788
[118054.535409] [<c1400b50>] (arch_call_rest_init) from [<c14010b8>] (start_kernel+0x4d4/0x504)
[118054.544844] [<c1400be4>] (start_kernel) from [<00000000>] (0x0)
[118054.551555] CPU3: stopping
[118054.554676] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G         C        5.4.10-rockchip #19.11.8
[118054.564401] Hardware name: Rockchip (Device Tree)
[118054.569751] Backtrace:
[118054.572579] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
[118054.581140]  r7:600f0193 r6:c16f2140 r5:00000000 r4:c16f2140
[118054.587562] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
[118054.595735] [<c0eef804>] (dump_stack) from [<c0110e14>] (handle_IPI+0x40c/0x438)
[118054.604099]  r7:00000000 r6:00000004 r5:c16fb9e4 r4:c1505690
[118054.610522] [<c0110a08>] (handle_IPI) from [<c0102398>] (gic_handle_irq+0x9c/0xa0)
[118054.619083]  r10:c1504df0 r9:f0803000 r8:ee579f10 r7:f0802000 r6:f080200c r5:c168ad2c
[118054.627932]  r4:c1607054
[118054.630856] [<c01022fc>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0xa8)
[118054.639317] Exception stack(0xee579f10 to 0xee579f58)
[118054.645057] 9f00:                                     00000000 1e7afee8 eedd13b4 c011e5e0
[118054.654298] 9f20: ffffe000 c16067a8 c16067f0 00000008 00000000 00000000 c1504df0 ee579f6c
[118054.663538] 9f40: ee579f70 ee579f60 c0109f28 c0109f2c 600f0113 ffffffff
[118054.671028]  r9:ee578000 r8:00000000 r7:ee579f44 r6:ffffffff r5:600f0113 r4:c0109f2c
[118054.679784] [<c0109ee4>] (arch_cpu_idle) from [<c0f0e66c>] (default_idle_call+0x38/0x3c)
[118054.688919] [<c0f0e634>] (default_idle_call) from [<c015c8c4>] (do_idle+0x214/0x2c0)
[118054.697675] [<c015c6b0>] (do_idle) from [<c015cc94>] (cpu_startup_entry+0x28/0x2c)
[118054.706235]  r10:00000000 r9:410fc0d1 r8:0000406a r7:c170da40 r6:10c0387d r5:00000003
[118054.715085]  r4:00000089
[118054.718000] [<c015cc6c>] (cpu_startup_entry) from [<c01100f4>] (secondary_start_kernel+0x160/0x16c)
[118054.728203] [<c010ff94>] (secondary_start_kernel) from [<001028cc>] (0x1028cc)
[118054.736373]  r5:00000051 r4:2e56c06a
[118054.740461] CPU2: stopping
[118054.743581] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G         C        5.4.10-rockchip #19.11.8
[118054.753298] Hardware name: Rockchip (Device Tree)
[118054.758648] Backtrace:
[118054.761476] [<c010e738>] (dump_backtrace) from [<c010ead8>] (show_stack+0x20/0x24)
[118054.770027]  r7:60000193 r6:c16f2140 r5:00000000 r4:c16f2140
[118054.776448] [<c010eab8>] (show_stack) from [<c0eef890>] (dump_stack+0x8c/0xa0)
[118054.784620] [<c0eef804>] (dump_stack) from [<c0110e14>] (handle_IPI+0x40c/0x438)
[118054.792986]  r7:00000000 r6:00000004 r5:c16fb9e4 r4:c1505690
[118054.799409] [<c0110a08>] (handle_IPI) from [<c0102398>] (gic_handle_irq+0x9c/0xa0)
[118054.807970]  r10:c1504df0 r9:f0803000 r8:ee577f10 r7:f0802000 r6:f080200c r5:c168ad2c
[118054.816819]  r4:c1607054
[118054.819733] [<c01022fc>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0xa8)
[118054.828194] Exception stack(0xee577f10 to 0xee577f58)
[118054.833934] 7f00:                                     00000000 543739c4 eedbd3b4 c011e5e0
[118054.843175] 7f20: ffffe000 c16067a8 c16067f0 00000004 00000000 00000000 c1504df0 ee577f6c
[118054.852415] 7f40: ee577f70 ee577f60 c0109f28 c0109f2c 60000113 ffffffff
[118054.859906]  r9:ee576000 r8:00000000 r7:ee577f44 r6:ffffffff r5:60000113 r4:c0109f2c
[118054.868661] [<c0109ee4>] (arch_cpu_idle) from [<c0f0e66c>] (default_idle_call+0x38/0x3c)
[118054.877805] [<c0f0e634>] (default_idle_call) from [<c015c8c4>] (do_idle+0x214/0x2c0)
[118054.886560] [<c015c6b0>] (do_idle) from [<c015cc94>] (cpu_startup_entry+0x28/0x2c)
[118054.895120]  r10:00000000 r9:410fc0d1 r8:0000406a r7:c170da40 r6:10c0387d r5:00000002
[118054.903970]  r4:00000089
[118054.906894] [<c015cc6c>] (cpu_startup_entry) from [<c01100f4>] (secondary_start_kernel+0x160/0x16c)
[118054.917106] [<c010ff94>] (secondary_start_kernel) from [<001028cc>] (0x1028cc)
[118054.925277]  r5:00000051 r4:2e56c06a
[118054.929372] ---[ end Kernel panic - not syncing: hung_task: blocked tasks ]---
Reply
#2
My findings so far:
I saw that Almond-Server is dramatictly increasing the memory step by step. I assume that this at the end lets the kernel crash.
Code:
  1  [                                                                               0.0%]   Hostname: tb-hass
  2  [|||                                                                            2.9%]   Tasks: 55, 65 thr; 1 running
  3  [|                                                                              0.7%]   Load average: 0.19 0.43 0.59
  4  [|||                                                                            2.6%]   Uptime: 18:00:00
  Mem[|||||||||||||||||||||||||||||||||||||||||                                457M/1.95G]   CpuFreq1: 1.61 GHz
  Swp[||                                                                      20.5M/1001M]   CpuFreq2: 1.61 GHz
  Cpu Temp:   57 C                                                                           CpuFreq3: 1.61 GHz
                                                                                             CpuFreq4: 1.61 GHz

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
9025 joerg      20   0  668M  308M 28640 S  0.7 15.4  0:36.74 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9026 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:00.00 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9027 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:02.31 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9028 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:03.08 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9029 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:02.78 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9030 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:02.91 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9031 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:00.00 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9033 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:00.15 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9034 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:00.12 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9035 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:00.09 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9036 joerg      20   0  668M  308M 28640 S  0.0 15.4  0:00.10 /usr/bin/node --max-old-space-size=128 /srv/almond-server/main.js
9098 homeassis  20   0  366M 79316 12588 R  3.3  3.9 12:50.35 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9100 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:00.00 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9101 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:03.04 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9102 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:03.37 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9103 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:03.03 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9104 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:03.23 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9105 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:03.00 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9106 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:07.15 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9107 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:03.07 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9108 homeassis  20   0  366M 79316 12588 S  0.0  3.9  0:00.94 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
9109 homeassis  20   0  366M 79316 12588 S  0.0  3.9  5:01.43 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
When I restart the Almon-Server the total memory used is about 208M. Then it increases slowly until also the swap is filled.
I read that this sudo sysctl -w vm.dirty_ratio=10 and this sudo sysctl -w vm.dirty_background_ratio=5 and also this /usr/bin/node --max-old-space-size=128 should help. I have to wait to see if it really helps.
Reply
#3
After I saw your post I put my Tinkerboard under extreme stressing ...

and board remain rock stable. So it must be something in the application level that eats memory.
Armbian. Lightweight Debian Stretch or Ubuntu Bionic for Tinker Board.
Reply
#4
Yes, I think so. The Almond-Server has increased to 883M since the morning. For a workaround I will write a script the restarts the server daily to see if this lets the kernel survive more than 1 or 2 days.
Reply
#5
OK, so far it works if I restart the Almond-Server every 8h. For reference I let here the almond-server.service I wrote to start and restart automatically:
Code:
[Unit]
Description=Almond Server
After=homeassistant@homeassistant.service

[Service]
Type=simple
WorkingDirectory=/srv/almond-server
ExecStart=/usr/bin/node /srv/almond-server/main.js
WatchdogSec=28800
Restart=always
RestartSec=10
StandardOutput=journal

[Install]
WantedBy=default.target
The WatchdogSec=28800 and the Restart=always let the server restart after 8h.
I opened an issue on their github, but I haven't got a response so far.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)