Java Networking and Proxies setting by options and checking etc

1) Introduction

In today's networking environments, particularly corporate ones,application developers have to deal with proxies almost as often assystem administrators. In some cases the application should use thesystem default settings, in other cases it will we want to have avery tight control over what goes through which proxy, and,somewhere in the middle, most applications will be happy todelegate the decision to their users by providing them with a GUIto set the proxy settings, as is the case in most browsers.

In any case, a development platform, like Java, should providemechanisms to deal with these proxies that are both powerful andflexible. Unfortunately, until recently, the Java platform wasn'tvery flexible in that department. But all that changed in J2SE 5.0as new API have been introduced to address this shortcoming, andthe purpose of this paper is to provide an in-depth explanation ofall these APIs and mechanisms, the old ones, which are still valid,as well as the new ones.

4.50 (50 votes)

Downloadsourcecode- 3 KB

2) System Properties

Up until J2SE 1.4 system properties were the only way to setproxy servers within the Java networking API for any of theprotocol handlers. To make matters more complicated, the names ofthese properties have changed from one release to another and someof them are now obsolete even if they are still supported forcompatibility's sake.

The major limitation of using system properties is that they arean “all or nothing” switch. Meaning that once a proxyhas been set for a particular protocol, it will affect allconnections for that protocol. It's a VM wide behavior.

There are 2 main ways to set system properties:

• As a command line option when invoking the VM
• Using the System.setProperty(String, String)method, assuming, of course that you have permission to do so.

Now, let's take a look, protocol by protocol, at the propertiesyou can use to set proxies. All proxies are defined by a host nameand a port number. The later is optional as, if it is notspecified, a standard default port will be used.

2.1) HTTP

There are 3 properties you can set to specify the proxy thatwill be used by the http protocol handler:

• http.proxyHost: the host name of the proxyserver
• http.proxyPort: the port number, the default valuebeing 80.
• http.nonProxyHosts: a list of hosts that should bereached directly, bypassing the proxy. This is a list of patternsseparated by '|'. The patterns may start or end with a '*' forwildcards. Any host matching one of these patterns will be reachedthrough a direct connection instead of through a proxy.

Let's look at a few examples assuming we're trying to executethe main method of the GetURL class:

$ java -Dhttp.proxyHost=webcache.mydomain.com GetURL

All http connections will go through the proxy server atwebcache.mydomain.com listening on port 80 (we didn'tspecify any port, so the default one is used).

$ java -Dhttp.proxyHost=webcache.mydomain.com -Dhttp.proxyPort=8080
-Dhttp.noProxyHosts=”localhost|host.mydomain.com” GetURL

In that second example, the proxy server will still be atwebcache.mydomain.com, but this time listening on port8080. Also, the proxy won't be used when connecting to eitherlocalhost orhost.mydonain.com.

As mentioned earlier, these settings affect all http connectionsduring the entire lifetime of the VM invoked with these options.However it is possible, using the System.setProperty() method, tohave a slightly more dynamic behavior.

Here is a code excerpt showing how this can be done:

//Set the http proxy to webcache.mydomain.com:8080

System.setProperty("http.proxyHost", "webcache.mydomain.com");
System.setPropery("http.proxyPort", "8080");

// Next connection will be through proxy.
URL url = new URL("http://java.sun.com/");
InputStream in = url.openStream();

// Now, let's 'unset' the proxy.
System.setProperty("http.proxyHost", null);

// From now on http connections will be done directly.

Now,this works reasonably well, even if a bit cumbersome, but itcan get tricky if your application is multi-threaded. Remember,system properties are “VM wide” settings, so allthreads are affected. Which means that the code in one threadcould, as a side effect, render the code in an other threadinoperative.

2.2) HTTPS

The https (http over SSL) protocol handler has its own set ofproperties:

• htttps.proxyHost
• https.proxyPort

As you probably guessed these work in the exact same manner astheir http counterparts, so we won't go into much detail except tomention that the default port number, this time, is 443 and thatfor the "non proxy hosts" list, the HTTPS protocol handler will usethe same as the http handler (i.e.http.nonProxyHosts).

2.3) FTP

Settings for the FTP protocol handler follow the same rules asfor http, the only difference is that each property name is nowprefixed with 'ftp.' instead of'http.'

Therefore the system properties are:

• ftp.proxHost
• ftp.proxyPort
• ftp.nonProxyHosts

Note that, this time, there is a separate property for the "nonproxy hosts" list. Also, as for http, the default port number valueis 80. It should be noted that when going through a proxy, the FTPprotocol handler will actually use HTTP to issue commands to theproxy server, which explains why this is the same default portnumber.

Let's examine a quick example:

$ java -Dhttp.proxyHost=webcache.mydomain.com
-Dhttp.proxyPort=8080 -Dftp.proxyHost=webcache.mydomain.com -Dftp.proxyPort=8080 GetURL

Here, both the HTTP and the FTP protocol handlers will use thesame proxy server at webcache.mydomain.com:8080.

2.4) SOCKS

The SOCKS protocol, as defined in RFC 1928, provides a frameworkfor client server applications to safely traverse a firewall bothat the TCP and UDP level. In that sense it is a lot more genericthan higher level proxies (like HTTP or FTP specific proxies). J2SE5.0 provides SOCKS support for client TCP sockets.

There are 2 system properties related to SOCKS:

• socksProxyHost for the host name of the SOCKSproxy server
• socksProxyPort for the port number, the defaultvalue being 1080

Note that there is no dot ('.') after the prefix this time. Thisis for historical reasons and to ensure backward compatibility.Once a SOCKS proxy is specified in this manner, all TCP connectionswill be attempted through the proxy.

Example:

$ java -DsocksProxyHost=socks.mydomain.com GetURL

Here, during the execution of the code, every outgoing TCPsocket will go through the SOCKS proxy server atsocks.mydomain.com:1080.

Now, what happens when both a SOCKS proxy and a HTTP proxy aredefined? Well the rule is that settings for higher level protocols,like HTTP or FTP, take precedence over SOCKS settings. So, in thatparticular case, when establishing a HTTP connection, the SOCKSproxy settings will be ignored and the HTTP proxy will becontacted. Let's look at an example:

$ java -Dhttp.proxyHost=webcache.mydomain.com -Dhttp.proxyPort=8080
-DsocksProxyHost=socks.mydomain.com GetURL

Here, an http URL will go throughwebcache.mydomain.com:8080 because the http settingstake precedence. But what about an ftp URL? Since no specific proxysettings were assigned for FTP, and since FTP is on top of TCP,then FTP connections will be attempted through the SOCKS proxyserver at socks.mydomsain.com:1080. If an FTP proxyhad been specified, then that proxy would have been usedinstead.

3) Proxy class

As we have seen, the system properties are powerful, but notflexible. The "all or nothing" behavior was justly deemed toosevere a limitation by most developers. That's why it was decidedto introduce a new, more flexible, API in J2SE 5.0 so that it wouldbe possible to have connection based proxy settings.

The core of this new API is the Proxy class which represents aproxy definition, typically a type (http, socks) and a socketaddress. There are, as of J2SE 5.0, 3 possible types:

• DIRECT which represents a direct connection, orabsence of proxy.
• HTTP which represents a proxy using the HTTPprotocol.
• SOCKS which represents proxy using either SOCKS v4or v5.

So, in order to create an HTTP proxy object you would call:

SocketAddress addr = new
InetSocketAddress("webcache.mydomain.com", 8080);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);

Remember, this new proxy object represents a proxydefinition, nothing more. How do we use such an object? AnewopenConnection() method has been added to the URLclass and takes a Proxy as an argument, it works the same way asopenConnection() with no arguments, except it forcesthe connection to be established through the specified proxy,ignoring all other settings, including the system propertiesmentioned above.

So completing the previous example, we can now add:

URL url = new URL("http://java.sun.com/");
URConnection conn = url.openConnection(proxy);

Simple, isn't it?

The same mechanism can be used to specify that a particular URLhas to be reached directly, because it's on the intranet forexample. That's where the DIRECT type comes into play. But, youdon't need to create a proxy instance with the DIRECT type, all youhave to do is use the NO_PROXY static member:

URL url2 = new URL("http://infos.mydomain.com/");
URLConnection conn2 = url2.openConnection(Proxy.NO_PROXY);

Now, this guarantees you that this specific URL will beretrieved though a direct connection bypassing any other proxysettings, which can be convenient.

Note that you can force a URLConnection to go through a SOCKSproxy as well:

SocketAddress addr = new InetSocketAddress("socks.mydomain.com", 1080);
Proxy proxy = new Proxy(Proxy.Type.SOCKS, addr);
URL url = new URL("ftp://ftp.gnu.org/README");
URLConnection conn = url.openConnection(proxy);

That particular FTP connection will be attempted though thespecified SOCKS proxy. As you can see, it's prettystraightforward.

Last, but not least, you can also specify a proxy for individualTCP sockets by using the newly introduced socket constructor:

SocketAddress addr = new InetSocketAddress("socks.mydomain.com", 1080);
Proxy proxy = new Proxy(Proxy.Type.SOCKS, addr);
Socket socket = new Socket(proxy);
InetSocketAddress dest = new InetSocketAddress("server.foo.com", 1234);
socket.connect(dest);

Here the socket will try to connect to its destination address(server.foo.com:1234) through the specified SOCKS proxy.

As for URLs, the same mechanism can be used to ensure that adirect (i.e. not through any proxy) connection should be attemptedno matter what the global settings are:

Socket socket = new Socket(Proxy.NO_PROXY);
socket.connect(new InetAddress("localhost", 1234));

Note that this new constructor, as of J2SE 5.0, accepts only 2types of proxy: SOCKS or DIRECT (i.e. the NO_PROXY instance).

4) ProxySelector

As you can see, with J2SE 5.0, the developer gains quite a bitof control and flexibility when it comes to proxies. Still, thereare situations where one would like to decide which proxy to usedynamically, for instance to do some load balancing betweenproxies, or depending on the destination, in which case the APIdescribed so far would be quite cumbersome. That's where theProxySelector comes into play.

In a nutshell the ProxySelector is a piece of code that willtell the protocol handlers which proxy to use, if any, for anygiven URL. For example, consider the following code:

URL url = new URL("http://java.sun.com/index.html");
URLConnection conn = url.openConnection();
InputStream in = conn.getInputStream();

At that point the HTTP protocol handler is invoked and it willquery the proxySelector. The dialog might go something likethat:

Handler: Hey dude, I'm trying to reachjava.sun.com, should I use a proxy?
ProxySelector: Which protocol do you intend to use?
Handler: http, of course!
ProxySelector: On the default port?
Handler: Let me check.... Yes, default port.
ProxySelector: I see. Then you shall usewebcache.mydomain.com on port 8080 as a proxy.
Handler: Thanks. <pause> Dude,webcache.mydomain.com:8080 doesn't seem to be responding! Any otheroption?
ProxySelector: Dang! OK, try webcache2.mydomain.com, on port8080 as well.
Handler: Sure. Seems to be working. Thanks.
ProxySelector: No sweat. Bye.

Of course I'm embellishing a bit, but you get the idea.

The best thing about the ProxySelector is that it is plugable!Which means that if you have needs that are not covered by thedefault one, you can write a replacement for it and plug it in!

So what is a ProxySelector? Let's take a look at the classdefinition:

public abstract class ProxySelector {
        public static ProxySelector getDefault();
        public static void setDefault(ProxySelector ps);
        public abstract List<Proxy> select(URI uri);
        public abstract void connectFailed(URI uri,
                SocketAddress sa, IOException ioe);
}

As we can see, ProxySelector is an abstract class with 2 staticmethods to set, or get, the default implementation, and 2 instancemethods that will be used by the protocol handlers to determinewhich proxy to use or to notify that a proxy seems to beunreachable. If you want to provide your own ProxySelector, all youhave to do is extend this class, provide an implementation forthese 2 instance methods then call ProxySelector.setDefault()passing an instance of your new class as an argument. At this pointthe protocol handlers, like http or ftp, will query the newProxySelector when trying to decide what proxy to use.

Before we see in details how to write such a ProxySelector,let's talk about the default one. J2SE 5.0 provides a defaultimplementation which enforces backward compatibility. In otherterms, the default ProxySelector will check the system propertiesdescribed earlier to determine which proxy to use. However, thereis a new, optional feature: On recent Windows systems and on Gnome2.x platforms it is possible to tell the default ProxySelector touse the system proxy settings (both recent versions of Windows andGnome 2.x let you set proxies globally through their userinterface). If the system propertyjava.net.useSystemProxies is set to true (by defaultit is set to false for compatibility sake), then the defaultProxySelector will try to use these settings. You can set thatsystem property on the command line, or you can edit the JREinstallation filelib/net.properties, that way youhave to change it only once on a given system.

Now let's examine how to write, and install, a newProxySelector.

Here is what we want to achieve: We're pretty happy with thedefault ProxySelector behavior, except when it comes to http andhttps. On our network we have more than one possible proxy forthese protocols and we we'd like our application to try them insequence (i.e.: if the 1^st one doesn't respond, then trythe second one and so on). Even more, if one of them fails too manytime, we'll remove it from the list in order to optimize things abit.

All we need to do is subclassjava.net.ProxySelector and provide implementations forboth theselect() andconnectFailed()methods.

The select() method is called by the protocolhandlers before trying to connect to a destination. The argumentpassed is a URI describing the resource (protocol, host and portnumber). The method will then return a List of Proxies. Forinstance the following code:

URL url = new URL("http://java.sun.com/index.html");
InputStream in = url.openStream();

will trigger the following pseudo-call in the protocolhandler:

List<Proxy> l = ProxySelector.getDefault().select(new URI("http://java.sun.com/"));

In our implementation, all we'll have to do is check that theprotocol from the URI is indeed http (or https), in which case wewill return the list of proxies, otherwise we just delegate to thedefault one. To do that, we'll need, in the constructor, to store areference to the old default, because ours will become thedefault.

So it is starting to look like this:

public class MyProxySelector extends ProxySelector {
        ProxySelector defsel = null;
        MyProxySelector(ProxySelector def) {
                defsel = def;
        }
        
        public java.util.List<Proxy> select(URI uri) {
                if (uri == null) {
                        throw new IllegalArgumentException("URI can't be null.");
                }
                String protocol = uri.getScheme();
                if ("http".equalsIgnoreCase(protocol) ||
                        "https".equalsIgnoreCase(protocol)) {
                        ArrayList<Proxy> l = new ArrayList<Proxy>();
                        // Populate the ArrayList with proxies
                        return l;
                }
                if (defsel != null) {
                        return defsel.select(uri);
                } else {
                        ArrayList<Proxy> l = new ArrayList<Proxy>();
                        l.add(Proxy.NO_PROXY);
                        return l;
                }
        }
}

First note the constructor that keeps a reference to the olddefault selector. Second, notice the check for illegal argument inthe select() method in order to respect the specifications.Finally, notice how the code defers to the old default, if therewas one, when necessary. Of course, in this example, I didn'tdetail how to populate the ArrayList, as it not of particularinterest, but the complete code is available in the appendix ifyou're curious.

As it is, the class is incomplete since we didn't provide animplementation for theconnectFailed() method. That'sour very next step.

The connectFailed() method is called by theprotocol handler whenever it failed to connect to one of theproxies returned by theselect() method. 3 argumentsare passed: the URI the handler was trying to reach, which shouldbe the one used whenselect() was called, theSocketAddress of the proxy that the handler was tryingto contact and the IOException that was thrown when trying toconnect to the proxy. With that information, we'll just do thefollowing: If the proxy is in our list, and it failed 3 times ormore, we'll just remove it from our list, making sure it won't beused again in the future. So the code is now:

public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
        if (uri == null || sa == null || ioe == null) {
                throw new IllegalArgumentException("Arguments can't be null.");
        }
        InnerProxy p = proxies.get(sa); 
        if (p != null) {
                if (p.failed() >= 3)
                        proxies.remove(sa);
        } else {
                if (defsel != null)
                        defsel.connectFailed(uri, sa, ioe);
        }
}

Pretty straightforward isn't it. Again we have to check thevalidity of the arguments (specifications again). The only thing wedo take into account here is the SocketAddress, if it's one of theproxies in our list, then we do deal with it, otherwise we defer,again, to the default selector.

Now that our implementation is, mostly, complete, all we have todo in the application is to register it and we're done:

public static void main(String[] args) {
        MyProxySelector ps = new MyProxySelector(ProxySelector.getDefault());
        ProxySelector.setDefault(ps);
        // rest of the application
}

Of course, I simplified things a bit for the sake of clarity, inparticular you've probably noticed I didn't do much Exceptioncatching, but I'm confident you can fill in the blanks.

It should be noted that both Java Plugin and Java Webstart doreplace the default ProxySelector with a custom one to integratebetter with the underlying platform or container (like the webbrowser). So keep in mind, when dealing with ProxySelector, thatthe default one is typically specific to the underlying platformand to the JVM implementation. That's why it is a good idea, whenproviding a custom one, to keep a reference to the older one, aswe've done in the above example, and use it when necessary.

5) Conclusion

As we have now established J2SE 5.0 provides quite a number ofways to deal with proxies. From the very simple (using the systemproxy settings) to the very flexible (changing the ProxySelector,albeit for experienced developers only), including the perconnection selection courtesy of the Proxy class.

Appendix

Here is the full source of the ProxySelector we developed inthis paper. Keep in mind that this was written for educationalpurposes only, and was therefore kept pretty simple on purpose.

import java.net.*;
import java.util.List;
import java.util.ArrayList;
import java.util.HashMap;
import java.io.IOException;

public class MyProxySelector extends ProxySelector {
        // Keep a reference on the previous default
    ProxySelector defsel = null;
        
        /*
         * Inner class representing a Proxy and a few extra data
         */
        class InnerProxy {
        Proxy proxy;
                SocketAddress addr;
                // How many times did we fail to reach this proxy?
                int failedCount = 0;
                
                InnerProxy(InetSocketAddress a) {
                        addr = a;
                        proxy = new Proxy(Proxy.Type.HTTP, a);
                }
                
                SocketAddress address() {
                        return addr;
                }
                
                Proxy toProxy() {
                        return proxy;
                }
                
                int failed() {
                        return ++failedCount;
                }
        }
        
        /*
         * A list of proxies, indexed by their address.
         */
        HashMap<SocketAddress, InnerProxy> proxies = new HashMap<SocketAddress, InnerProxy>();

        MyProxySelector(ProxySelector def) {
          // Save the previous default
          defsel = def;
          
          // Populate the HashMap (List of proxies)
          InnerProxy i = new InnerProxy(new InetSocketAddress("webcache1.mydomain.com", 8080));
          proxies.put(i.address(), i);
          i = new InnerProxy(new InetSocketAddress("webcache2.mydomain.com", 8080));
          proxies.put(i.address(), i);
          i = new InnerProxy(new InetSocketAddress("webcache3.mydomain.com", 8080));
          proxies.put(i.address(), i);
          }
          
          /*
           * This is the method that the handlers will call.
           * Returns a List of proxy.
           */
          public java.util.List<Proxy> select(URI uri) {
                // Let's stick to the specs. 
                if (uri == null) {
                        throw new IllegalArgumentException("URI can't be null.");
                }
                
                /*
                 * If it's a http (or https) URL, then we use our own
                 * list.
                 */
                String protocol = uri.getScheme();
                if ("http".equalsIgnoreCase(protocol) ||
                        "https".equalsIgnoreCase(protocol)) {
                        ArrayList<Proxy> l = new ArrayList<Proxy>();
                        for (InnerProxy p : proxies.values()) {
                          l.add(p.toProxy());
                        }
                        return l;
                }
                
                /*
                 * Not HTTP or HTTPS (could be SOCKS or FTP)
                 * defer to the default selector.
                 */
                if (defsel != null) {
                        return defsel.select(uri);
                } else {
                        ArrayList<Proxy> l = new ArrayList<Proxy>();
                        l.add(Proxy.NO_PROXY);
                        return l;
                }
        }
        
        /*
         * Method called by the handlers when it failed to connect
         * to one of the proxies returned by select().
         */
        public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
                // Let's stick to the specs again.
                if (uri == null || sa == null || ioe == null) {
                        throw new IllegalArgumentException("Arguments can't be null.");
                }
                
                /*
                 * Let's lookup for the proxy 
                 */
                InnerProxy p = proxies.get(sa); 
                        if (p != null) {
                                /*
                                 * It's one of ours, if it failed more than 3 times
                                 * let's remove it from the list.
                                 */
                                if (p.failed() >= 3)
                                        proxies.remove(sa);
                        } else {
                                /*
                                 * Not one of ours, let's delegate to the default.
                                 */
                                if (defsel != null)
                                  defsel.connectFailed(uri, sa, ioe);
                        }
     }
}