Hi,
Over IP mode is for reaching out to your RS485 device via the IP that is given to your device on WAN.
So to successfully use it in your use case you must choose the server option and set baud rate, flow control and other settings depending on your RS485 device, not the router.
Once you've done everything, you should be able to access the RS485 device through the Public IP of the RUT955 modem.
EB.