May I know what kind of IP address is your RUTX11 able to get from your home network? Do you have a Public IP address assigned to it? If the Public IP address is given to your main router where the RUTX11 is connected to your main router the best thing to do is to connect it like this:
Main Router (ISP Router with Public IP address) -----> LAN Port ---Connected---> WAN port of RUTX11 ----- LAN devices..
What you need to do is to port-forwarding from the Main router to the RUTX11 WAN IP address and another port-forwarding to RUTX11 to your LAN device.