def Tenako.blog () end

For the impatient readers:

Don’t set Jetty’s <stopPort> to the same <port> where Jetty is running or you might run into some nasty-hard-to-debug problem.

Now a full explanation:

Imagine the following scenario,

Here we have to web services, A and B, both running on the same machine on Jetty webserver and one of them is listening for petitions on port 9090 (configured via de maven-jetty-plugin). Web service A has to request some data to B and process its reply. Nothing unusual.

So A issue a request to http://localhost:9090/<path> expecting an XML response as a result of doing it. But it gets nothing, nada, nichts.

Well, at this moment the first thing that came through my mind was that I did something wrong while coding or configurating any of the web services. So I checked everything, but it seemed right. My next move was to create a fake request with curl just to see what happened.

What happened misleaded me even more: web service B was responding. WTF!!

So I went one level down. I ran tcpdump. And what I got? I discovered that the main difference between the requests issued by web service A and curl was the ip version they used: web service A was using ipv4 and curl ipv6.

To force both requests (from the web service and curl) to use ipv4, I run curl against the web service B but now instead of localhost I used 127.0.0.1. And the response was empty, just like if the request came from web service A.

So I thought that all of my troubles had to do with the ip stack beeing used. And I wasted some hours
trying to figure out what was wrong with it.
As I was really desperate with all this, I download Jetty sources to debug it on eclipse. I wanted to know
what was happening with my requests. And here was were light came to me by chance. While debugging Jetty,
an incomming connection from  internet arraived to my <public_ip>:9090  (I guess it was someone doing some kind of portscanning) and got stuck on my breakpoints…

Mmmm, wait a moment…

Yep, internet addresses are ipv4 only (for most of us), so this ruled out any kind of problem with the ipv4 stack or shit. Now I began to see the light. This was some mess with the adresses.

Just to check I did an alias in my loopback interface to 127.0.0.2 and run again the request with curl to that ip. This time being successful. Ooook.

So, whats wrong with 127.0.0.1:9090?

Netstat output,

13:29:46 ~ $ netstat -f inet -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4       0      0  127.0.0.1.9090         *.*                    LISTEN
tcp46      0      0  *.9090                 *.*                    LISTEN
...

Lsof output,

13:49:39 ~ $ lsof -i :9090
COMMAND PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
java    239 madtrick   94u  IPv6 0x08c10f80      0t0  TCP *:websm (LISTEN)
java    239 madtrick   95u  IPv6 0x08c11460      0t0  TCP [::127.0.0.1]:websm (LISTEN)

I have two sockets listening on port 9090 and both belong to the same process (as reported by lsof), my Jetty Server.  But requests to 127.0.0.1:9090 are lost meanwhile requests to <any_other_ip>:9090 are processed by the server.

Lets try something, lets change the <stopPort> to 9091 and see what happens…
Request from web service A, OK.
Request from curl, OK.

Run netstat again,

13:59:50 ~ $ netstat -f inet -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4       0      0  127.0.0.1.9091         *.*                    LISTEN
tcp46      0      0  *.9090                 *.*                    LISTEN
...

And lsof for ports 9090 and 9091,

14:00:48 ~ $ lsof -i :9090
COMMAND PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
java    301 madtrick   93u  IPv6 0x08c10f80      0t0  TCP *:websm (LISTEN)

14:00:55 ~ $ lsof -i :9091
COMMAND PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
java    301 madtrick   94u  IPv6 0x08c11460      0t0  TCP [::127.0.0.1]:xmltec-xmlmail (LISTEN)

So you see, Jetty opens another socket which listens on 127.0.0.1:<stopPort>  (lets call it jetty stop daemon) and being that an explicit address it has precedence over the catch all 0.0.0.0:9090 (where Jetty is listening). That’s why my requests to 127.0.0.1:9090 returned nothing.

This is what was happening before,

And this is whats happens now,