First, if you are trying to connect over WAN/Internet you would have to note that you will either need a VPN or public IP at both ends of connection.
Second, you need to use different ports (and I would offer something form high ports like 6500 or 7000 or something alike, not port like 80 that is used by HTTP). when configuring use one port for one router Server configuration and another router Client configuration. e.g.:
Router 1: Server - 6500 port / Client - 7000 port.
Router 2: Server - 7000 port / Client - 6500 port.
Lastly, do not forget to add interface through which communication will commence (you can find that under Serial configuration)
If ports match and there is no mistakes in server address both devices connected through serial ports should be able to send and receive information from each other.