Don’t set Jetty’s <stopPort> to the same <port> where Jetty is running or you might run into some nasty-hard-to-debug problem.
Now a full explanation:
Imagine the following scenario,
Here we have to web services, A and B, both running on the same machine on Jetty webserver and one of them is listening for petitions on port 9090 (configured via de maven-jetty-plugin). Web service A has to request some data to B and process its reply. Nothing unusual.
So A issue a request to http://localhost:9090/<path> expecting an XML response as a result of doing it. But it gets nothing, nada, nichts.
Well, at this moment the first thing that came through my mind was that I did something wrong while coding or configurating any of the web services. So I checked everything, but it seemed right. My next move was to create a fake request with curl just to see what happened.
What happened misleaded me even more: web service B was responding. WTF!!
So I went one level down. I ran tcpdump. And what I got? I discovered that the main difference between the requests issued by web service A and curl was the ip version they used: web service A was using ipv4 and curl ipv6.
To force both requests (from the web service and curl) to use ipv4, I run curl against the web service B but now instead of localhost I used 127.0.0.1. And the response was empty, just like if the request came from web service A.
So I thought that all of my troubles had to do with the ip stack beeing used. And I wasted some hours
trying to figure out what was wrong with it.
As I was really desperate with all this, I download Jetty sources to debug it on eclipse. I wanted to know
what was happening with my requests. And here was were light came to me by chance. While debugging Jetty,
an incomming connection from internet arraived to my <public_ip>:9090 (I guess it was someone doing some kind of portscanning) and got stuck on my breakpoints…
Mmmm, wait a moment…
Yep, internet addresses are ipv4 only (for most of us), so this ruled out any kind of problem with the ipv4 stack or shit. Now I began to see the light. This was some mess with the adresses.
Just to check I did an alias in my loopback interface to 127.0.0.2 and run again the request with curl to that ip. This time being successful. Ooook.
So, whats wrong with 127.0.0.1:9090?
13:29:46 ~ $ netstat -f inet -an Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 127.0.0.1.9090 *.* LISTEN tcp46 0 0 *.9090 *.* LISTEN ...
13:49:39 ~ $ lsof -i :9090 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 239 madtrick 94u IPv6 0x08c10f80 0t0 TCP *:websm (LISTEN) java 239 madtrick 95u IPv6 0x08c11460 0t0 TCP [::127.0.0.1]:websm (LISTEN)
I have two sockets listening on port 9090 and both belong to the same process (as reported by lsof), my Jetty Server. But requests to 127.0.0.1:9090 are lost meanwhile requests to <any_other_ip>:9090 are processed by the server.
Lets try something, lets change the <stopPort> to 9091 and see what happens…
Request from web service A, OK.
Request from curl, OK.
Run netstat again,
13:59:50 ~ $ netstat -f inet -an Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 127.0.0.1.9091 *.* LISTEN tcp46 0 0 *.9090 *.* LISTEN ...
And lsof for ports 9090 and 9091,
14:00:48 ~ $ lsof -i :9090 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 301 madtrick 93u IPv6 0x08c10f80 0t0 TCP *:websm (LISTEN)
14:00:55 ~ $ lsof -i :9091 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 301 madtrick 94u IPv6 0x08c11460 0t0 TCP [::127.0.0.1]:xmltec-xmlmail (LISTEN)
So you see, Jetty opens another socket which listens on 127.0.0.1:<stopPort> (lets call it jetty stop daemon) and being that an explicit address it has precedence over the catch all 0.0.0.0:9090 (where Jetty is listening). That’s why my requests to 127.0.0.1:9090 returned nothing.
This is what was happening before,
And this is whats happens now,