Question

Hello! We have RUT240 with FW version 07.02.7 and we are currently experiencing high RAM usage of ZeroTier plugin, which has lead to OOM killer killing it a few times, thus losing connection. It was happening very often, usually killed in a couple of minutes after running, after killing it a few times it seems to have calmed down.

Can you please provide an advice, or at least some suggested steps to lower RAM usage?

The RUT240 is running with WWAN (LTE) and WLAN and is in remote location. The only installed package on top of regular install is ZeroTier.

We can share additional logs if necessary. Some brief outputs I've copied to the message below.

Thanks a lot.
PV

top command:

Mem: 46844K used, 13032K free, 444K shrd, 744K buff, 4100K cached
CPU:  24% usr  28% sys   0% nic  41% idle   0% io   0% irq   5% sirq
Load average: 0.61 1.29 0.94 2/81 2204
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
20667     1 root     S    24168  40%  34% /usr/bin/zerotier-one -p9993 /var/lib/zerotier-one_zerotier

[email protected]:~# logread | grep oom_reaper
Tue Sep 20 13:28:41 2022 kern.info kernel: [ 5935.268370] oom_reaper: reaped process 410 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 13:29:05 2022 kern.info kernel: [ 5958.832060] oom_reaper: reaped process 642 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 13:40:16 2022 kern.info kernel: [ 6629.899611] oom_reaper: reaped process 5236 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 13:40:40 2022 kern.info kernel: [ 6655.025648] oom_reaper: reaped process 5503 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 13:41:12 2022 kern.info kernel: [ 6686.014712] oom_reaper: reaped process 5753 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 13:41:37 2022 kern.info kernel: [ 6711.025162] oom_reaper: reaped process 6019 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 13:57:15 2022 kern.info kernel: [ 7649.185483] oom_reaper: reaped process 6257 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 13:57:52 2022 kern.info kernel: [ 7685.437207] oom_reaper: reaped process 18497 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 14:00:47 2022 kern.info kernel: [ 7860.629810] oom_reaper: reaped process 19365 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 14:01:21 2022 kern.info kernel: [ 7894.971323] oom_reaper: reaped process 20099 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 14:01:58 2022 kern.info kernel: [ 7931.605451] oom_reaper: reaped process 20316 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Tue Sep 20 14:02:35 2022 kern.info kernel: [ 7970.162103] oom_reaper: reaped process 20480 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

# logread | grep -i oom
Tue Sep 20 14:02:35 2022 kern.err kernel: [ 7970.044366] Out of memory: Killed process 20480 (zerotier-one) total-vm:43108kB, anon-rss:16476kB, file-rss:8kB, shmem-rss:0kB, UID:0 pgtables:60kB oom_score_adj:0
Tue Sep 20 14:02:35 2022 kern.info kernel: [ 7970.162103] oom_reaper: reaped process 20480 (zerotier-one), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Answer 1

Hello,

Could you provide a troubleshoot file?

I will consult RnD about your issue.

Best regards,

Comments

I've uploaded the Troubleshoot file to the original message.

BR

---

Hello,

I would like you to download the firmware image from this link and upload this updated ZeroTier package in Services -> Package manager -> Upload section. See if your issue persists.

Best regards

---

Hello,

thank you for the files. I hope to get another testing device soon where I can upload the FW, currently I don't have a spare device at hand, the device we are troubleshooting is already in production and to workaround this issue we have used Wireguard VPN for now.

Just to understand the nature of the fix, it's not something that I could do with replacing maybe just the Zerotier package? That would somehow lower the risk for us.

BR,
Pavel

---

Hello,

The issue did not replicate during our internal testing, thus It would helpful if you could provide some more details. Could you describe your exact use case scenario? What was the topology of your set up? Did you make any other configurations of the device? Did you have remote HTTP(S), SSH access enabled before? Was the firmware RUT2_R_00.07.02.7 the initial version router shipped with or did you perform an update with Keep settings option enabled?

Best regards,

---

Hello,

I'll do my best not to skip any question:

• The use case is real-time video streaming over RTSP, continuous, throughout a tunnel to a remote location. (An IP camera is connected to RUT240.) AFAIK the issue was happening during large loads - mostly continuous upload of 2-5Mbps through the ZeroTier tunnel over LTE.
• The topology:
  • WAN -- WWAN (CGNAT, IPv4),
  • IPv4 LAN network with one connected IP camera, and WLAN enabled
  • A routed connection to our IPv4 ZeroTier and Wireguard network via WWAN. Router's private network is accessible remotely using both VPNs. NAT is disabled for VPN.
• Configurations:
  • Changed IPv4 subnet, set up DHCP static leases,
  • WLAN (one AP with WPA2-PSK - AES),
  • SMS commands - both calls and SMS,
  • VoLTE (first to Yes - it worked, yay, then back to Auto),
  • one Wireguard tunnel with one peer,
  • one ZeroTier configuration,
  • firewall configuration
• Both HTTP(S) and SSH access was enabled and is enabled, only from LAN or VPN.
• Version: We performed an update from RUT2XX_R_00.01.14.5 to RUT2_R_00.07.02.7 with "Keep settings" enabled.

BR,
Pavel