10868 questions

12950 answers

20197 comments

25877 members

+1 vote
42 views 1 comments
ago by

We have a production environment with around 100 RUT955 devices. A couple of months ago we updated serveral devices to fw version RUT9_R_00.07.01.4. 

After about 50 days uptime we are getting high ping responses from those devices. Remote ssh is not available (connection refused or timeout). WebGUI login fails with "device busy". 

Device logs show out of memory problems with process "ports_eventsd" as the source:

[4268791.398774] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=port_eventsd,pid=2632,uid=0 
[4268791.407894] Out of memory: Killed process 2632 (port_eventsd) total-vm:67244kB, anon-rss:65532kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:80kB oom_score_adj:0 
[4268792.032643] oom_reaper: reaped process 2632 (port_eventsd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[4586670.162124] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=port_eventsd,pid=2632,uid=0
[4586670.171269] Out of memory: Killed process 2632 (port_eventsd) total-vm:71784kB, anon-rss:70076kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:88kB oom_score_adj:0
[4586670.965407] oom_reaper: reaped process 2632 (port_eventsd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
After the oom-kill device response time is back to normal. But before this we see 24-72 hrs with reduced performance and lost connectivity. 
Is this a fw related bug? Any workaround?
ago by

As far as we can see this is connected to firmware v7 and can be found on both RUT955 and RUT956 devices. The port_eventsd has increasing memory usage over time.

Firmware v7, RUT955:

root@Teltonika-RUT955:~# cat /etc/version && uptime && free && top -n 1 |grep eventsd

RUT9_R_00.07.01.4

 10:09:29 up 48 days, 16:13,  load average: 0.03, 0.04, 0.00

              total        used        free      shared  buff/cache   available

Mem:         124832       93832       21396         280        9604        1484

Swap:             0           0           0

 4599     1 root     S    66844  53%   0% /usr/bin/port_eventsd --suppress-topol

root@Teltonika-RUT955:~#

Firmware v7, RUT956:

root@Teltonika-RUT956:~# cat /etc/version && uptime && free && top -n 1 |grep eventsd

RUT9M_R_00.07.01.7

 12:08:34 up 32 days, 22:11,  load average: 0.25, 0.30, 0.34

              total        used        free      shared  buff/cache   available

Mem:         123268       73828       25900         220       23540       12988

Swap:             0           0           0

 2575     1 root     S    45924  37%   9% /usr/bin/port_eventsd --suppress-topol

root@Teltonika-RUT956:~#

Firmware v6, RUT955:

root@Teltonika-RUT955:~# cat /etc/version && uptime && free && top -n 1 |grep eventsd

RUT9XX_R_00.06.06.1

 10:10:48 up 27 days,  5:01,  load average: 0.27, 0.06, 0.02

              total        used        free      shared  buff/cache   available

Mem:         125984       25728       74764         492       25492       99040

Swap:             0           0           0

25175 25148 root     S     1536   1%   0% grep eventsd

root@Teltonika-RUT955:~#

1 Answer

0 votes
ago by
Hello,

Maybe there is a possibility to get a full troubleshoot file where these logs are visible or more of the logs are visible, that would allow me to create a better case for our RnD department to look more deeply into this.

Thank you