Commons-net Class Library

 Through Jakarta, Apache has released a hoard of Java related technologies, probably the most popular being the Tomcat Servlet engine. Jakarta does however, have a collection of libraries and tools: one of the newest being the Commons/Net library. If you have ever implemented the various Internet protocols, such as FTP, Telnet, SMTP, POP3, and NNTP, what you might have noticed is that the error checking and intricate details of each protocol seem to be more trouble than they are worth. Many times I implemented a small subset of one of the protocols to satisfy my specific project requirements, but I never had the time to fully develop a proper implementation of any of the protocols.

Now thanks to the Commons/Net library, you no longer need to trouble yourself with the details of the individual protocols but rather share in the benefit they bring to your applications. The Commons/Net library provides support for the following Internet protocols:

• Finger
• Whois
• TFTP
• Telnet
• POP3
• FTP
• NNTP
• SMTP
• And some miscellaneous protocols such as Time and Echo as well as BSD R command support

The goal of the Commons/Net library is to "make the global functionality of a protocol accessible when possible, but also provide access to the fundamental protocols where applicable so that the programmer may construct his own custom implementations." With this said, your current knowledge of the specific protocols will help you understand the APIs, and if you are not that familiar with the protocols, the libraries mirror the APIs so you soon will be.

This week I will discuss FTP and over the next few articles I will fill in some of the others.

File Transfer Protocol

Transferring files from one machine to another is a fundamental task in today's distributed environment. Most techies know all of the commands to connect to a remote FTP server, find a file or files, setup the transfer type, and grab the files your want. A typical FTP session might look as follows:

 

C:\> ftp 192.168.1.99
Connected to 192.168.1.99.
220 WGB01 FTP server (Version wu-2.6.2-5) ready.
User (192.168.1.99:(none)): myusername
331 Password required for myusername.
Password:
230 User myusername logged in.
ftp> cd /apps/mydir
ftp> ls 
200 PORT command successful.
150 Opening ASCII mode data connection for directory listing. 
total 6160
-rw-r--r--  1 501   100    3147032 Mar 31 13:37 myfile.pdf
226 Transfer complete.
ftp: 101 bytes received in 0.58Seconds 0.17Kbytes/sec.
ftp> lcd c:\download
ftp> bin
ftp> get myfile.pdf

 

 

The steps outlined in this session are as follows:

1. Connect to the server

2. Logon with a username and password

3. Change directory to "/apps/mydir" (cd)

4. Get a listing the files in that directory (ls)

5. Change the local directory to "c:\download" (lcd)

6. Set the transfer mode to binary (bin)

7. Retrieve the file "myfile.pdf" from the server (get)

Each of these steps require an action from the FTP client and the FTP server and then error checking at every stage. The bottom line is that it is a lot of work. Now consider the code in listing 1.

Listing 1. getDataFiles() method

...
import org.apache.commons.net.ftp.*;
...

  public static void getDataFiles( String server,
                   String username,
                   String password,
                   String folder,
                   String destinationFolder,
                   Calendar start,
                   Calendar end )
  {
    try
    {
      // Connect and logon to FTP Server
      FTPClient ftp = new FTPClient();
      ftp.connect( server );
      ftp.login( username, password );
      System.out.println("Connected to " + 
           server + ".");
      System.out.print(ftp.getReplyString());

      // List the files in the directory
      ftp.changeWorkingDirectory( folder );
      FTPFile[] files = ftp.listFiles();
      System.out.println( "Number of files in dir: " + files.length );
      DateFormat df = DateFormat.getDateInstance( DateFormat.SHORT );
      for( int i=0; i<files.length; i++ )
      {
        Date fileDate = files[ i ].getTimestamp().getTime();
        if( fileDate.compareTo( start.getTime() ) >= 0 &&
          fileDate.compareTo( end.getTime() ) <= 0 )
        {
          // Download a file from the FTP Server
          System.out.print( df.format( files[ i ].getTimestamp().getTime() ) );
          System.out.println( "\t" + files[ i ].getName() );
          File file = new File( destinationFolder + 
               File.separator + files[ i ].getName() );
          FileOutputStream fos = new FileOutputStream( file ); 
          ftp.retrieveFile( files[ i ].getName(), fos );
          fos.close();
          file.setLastModified( fileDate.getTime() );
        }
      }

      // Logout from the FTP Server and disconnect
      ftp.logout();
      ftp.disconnect();

    }
    catch( Exception e )
    {
      e.printStackTrace();
    }
  }

 

 

To compile listing 1, you will need to download the Commons/Net library from:

http://jakarta.apache.org/commons/net/index.html

And then setup your CLASSPATH to include the following file distributed in that download:

 

commons-net-1.0.0.jar

 

 

The getDataFiles()connects to an FTP server using a provided username and password, changes to a specified directory, and downloads all files in a date range to a specified local directory. All of this is accomplished through two of the Commons/Net classes:

• FTPClient: used to connect and logon to a server, navigate the directory structure, list files, upload and download files, and log out and disconnect from the server

• FTPFile: used to discover file properties

The FTPClient class is the high-level front-end to connecting to FTP servers. It provides the following functionality (this list is a subset of the methods provided):

• connect(String server) - connect to a server

• disconnect() - disconnect from a server

• login(String username, String password) - login to an FTP server

• logout() - logout from the FTP server

• changeWorkingDirectory(String pathname) - change the current working directory on the FTP server

• FTPFile[] listFiles() - list the files in the current working directory on the FTP server

• String[] listNames() - list the filenames in the current working directory on the FTP server

• String[] listNames(String pathname) - list the filenames in the specified directory on the FTP server

• makeDirectory(String pathname) - create the new specified directory

• rename(String from, String to) - rename a file on the FTP server

• storeFile(String remoteName, InputStream local) - upload the file from the specified java.io.InputStream to the remoteName in the current working directory

• retrieveFile(java.lang.String remoteName, OutputStream local) - downloads the specified remoteName file from the current working directory to the specified java.io.OutputStream

Obtaining a file listing provides an array of FTPFiles. The FTPFile class provides the following functionality:

• String getName() - returns the name of the file

• String getGroup() - returns the group that this file belongs to

• String getUser() - returns the user that owns the file

• long getSize() - returns size of the file

• int getType() - returns the type of the file: DIRECTORY_TYPE, FILE_TYPE, SYMBOLIC_LINK_TYPE, UNKNOWN_TYPE

• boolean isDirectory() - returns true if this file is a directory

• boolean isFile() - returns true if this file is a file (not a directory)

• boolean isSymbolicLink() - returns true if this file is a symbolic link

• Calendar getTimestamp() - returns the timestamp that the file was last modified

Listing 1 shows a function that in a very small number of lines of code accomplishes an incredible number of tasks. The total time to write and test this code was about 30 minutes, but if you had to focus on the details of the FTP protocol, then that estimate could easily be increased tenfold. The FTPClient and FTPFile classes allow you to focus on your business function rather than the underlying details of the transfer mechanism.

Telnet

While most non-technical people have never heard of telnet, most software developers, and especially those working on distributed software projects, are very familiar with using telnet. But you might not be familiar with what Telnet really is and how did it originated.

History

Since 1969, as different Internet protocols (such as HTTP, FTP, SMTP, and Telnet) have been developed, the mechanism for distributing them to the world has been through the production of a "Request for Comments", or RFC document. Each RFC document is assigned a number upon submission; the Telnet specification is also known as RFC 854. The collection of RFC documents is available in multiple locations throughout the web, but one specific one that I like is:

 

http://www.rfc-editor.org

 

 

In 1983, the Telnet specification was created and defined as follows:

"The purpose of the TELNET Protocol is to provide a fairly general, bi-directional, eight-bit byte oriented communications facility. Its primary goal is to allow a standard method of interfacing terminal devices and terminal-oriented processes to each other. It is envisioned that the protocol may also be used for terminal-terminal communication ("linking") and process-process communication (distributed computation)."

The Telnet protocol and implementation is the mechanism by which a TCP (Transmission Control Protocol) connection is established between two hosts: it essentially allows you to connect to a remote host and use it as though you were sitting directly in front of its terminal. The Telnet protocol (and supporting applications: server and client) takes care of the overhead required to maintain the connection and handle your requests. Most operating systems ship with an implementation of a Telnet client: in Windows and Unix variations it is available through the telnet command. One interesting side note: if you are ever testing an Internet protocol and want to understand the "guts" of how it works, the best mechanism to do so is to open a telnet session to the desired server of the specific port that it is listening on. A great example of this is understanding the HTTP protocol: open a telnet session to www.informit.com on port 80 (HTTP port) and then you can interact with the HTTP Server directly. For example:

 

c:\> telnet http://www.informit.com 80
GET / HTTP/1.0
(blank line)

 

The HTTP protocol is beyond the scope of this article, but for the purposes of this example: the "GET" command requests a page from the server, the "/" asks the server for the root document, and "HTTP/1.0" specifies that we are using the 1.0 version of the HTTP specification. HTTP requests are terminated by a blank line followed by a carriage return. If you run this example you will see the raw HTML returned for InformIT.com's home page.

Telnet and Commons/NET

The Commons/NET libraries provide simple support for Telnet through the org.apache.commons.net.telnet package and specifically through the TelnetClient class in that package. This class is fairly rudimentary and the procedure for using it is as follows:

1. Call its connect() method to connect to the server

2. Call its getInputStream() and getOutputStream() methods to establish a mechanism through which to communicate with the server

3. When you are finished, call its disconnect() method (and do not call the close() methods of the streams)

The framework for a Telnet session may look as follows:

 

TelnetClient telnet = new TelnetClient();
telnet.connect( "192.168.1.99", 23 );
InputStream in = telnet.getInputStream();
PrintStream out = new PrintStream( telnet.getOutputStream() );
...
telnet.close();

 

By default Telnet servers listen on port 23; there are standard ports that you can expect to be the same on all computers, such as FTP is on port 21, SMTP is on port 25, POP3 is on port 119, and HTTP is on port 80. The connect() statement connects to the machine at IP address "192.168.1.99" on port 23. Next it gets a standard java.io.InputStream object: we do not bother getting a buffered input stream as you will soon see that we will have to look at our Telnet server one character at a time. Next it gets a java.io.OutputStream from the TelnetClient but wraps it inside a java.io.PrintStream; PrintStream is probably very familiar to you though the System.out class: it has methods such as print() and println() that are much easier for us to work with.

The interesting part of the Telnet session, and all of the work, comes in those three little dots that I innocently placed before the close() statement. Because the purpose of Telnet is to establish and maintain the connection between your client and the server and it is the server's responsibility to expose its functionality, the Telnet protocol is out of the specific command-by-command equation. This means that the burden of understanding the server you are connected to and how to manipulate it falls on you.

A real-life task that I used the TelnetClient class to accomplish is one that allows me to sleep at night: monitor a web site and if it goes down then restart it. Monitoring it is easy: open an HTTP connection to the web server and see if you get back what you expect. Restarting the server is a little more difficult: telnet to the server, navigate to the script directory, and then execute the restart script. The server that I am connecting to is a Linux system, but the sample code should work on most Unix based systems. Listing 1 shows the code for the TelnetExample.java file.

Listing 1. TelnetExample.java

package com.informit.commons;

import org.apache.commons.net.telnet.*;
import java.io.*;

public class TelnetSample
{
  private TelnetClient telnet = new TelnetClient();
  private InputStream in;
  private PrintStream out;
  private char prompt = '$';

  public TelnetSample( String server, String user, String password ) {
   try {
	 // Connect to the specified server
	 telnet.connect( server, 23 );
	 
	 // Get input and output stream references
	 in = telnet.getInputStream();
	 out = new PrintStream( telnet.getOutputStream() );

	 // Log the user on
	 readUntil( "login: " );
	 write( user );
	 readUntil( "Password: " );
	 write( password );

	 // Advance to a prompt
	 readUntil( prompt + " " );
   }
   catch( Exception e ) {
	 e.printStackTrace();
   }
  }

  public void su( String password ) {
    try {
      write( "su" );
      readUntil( "Password: " );
      write( password );
      prompt = '#';
      readUntil( prompt + " " );
    }
    catch( Exception e ) {
      e.printStackTrace();
    }
  }

  public String readUntil( String pattern ) {
   try {
	 char lastChar = pattern.charAt( pattern.length() - 1 );
	 StringBuffer sb = new StringBuffer();
	 boolean found = false;
	 char ch = ( char )in.read();
	 while( true ) {
	  System.out.print( ch );
	  sb.append( ch );
	  if( ch == lastChar ) {
	    if( sb.toString().endsWith( pattern ) ) {
		 return sb.toString();
	    }
	  }
	  ch = ( char )in.read();
	 }
   }
   catch( Exception e ) {
	 e.printStackTrace();
   }
   return null;
  }

  public void write( String value ) {
   try {
	 out.println( value );
	 out.flush();
	 System.out.println( value );
   }
   catch( Exception e ) {
	 e.printStackTrace();
   }
  }

  public String sendCommand( String command ) {
   try {
	 write( command );
	 return readUntil( prompt + " " );
   }
   catch( Exception e ) {
	 e.printStackTrace();
   }
   return null;
  }

  public void disconnect() {
   try {
	 telnet.disconnect();
   }
   catch( Exception e ) {
	 e.printStackTrace();
   }
  }

  public static void main( String[] args ) {
   try {
	 TelnetSample telnet = new TelnetSample( "192.168.1.99", 
                         "username", 
                         "password" );
	 telnet.sendCommand( "cd /mydir/mysubdir" );
     telnet.su( "root-password" );
	 telnet.sendCommand( "./restart.sh" );
	 telnet.disconnect();
   }
   catch( Exception e ) {
	 e.printStackTrace();
   }
  }

 

 

A key observation of listing 1 is that the TelnetClient does not know anything about what you are doing: it helps you establish the connection and then it leaves you to read from and write to the server. Thus the process involves understanding what the server will send you and in what order: you have to program in all of the logic to facilitate what you are trying to accomplish. Let's look at a typical telnet session (launch the telnet utility that comes with your operating system and connect to a telnet server:

Red Hat Linux release 7.3 (Valhalla)
Kernel 2.4.18-3smp on an i686
login: myusername
Password: mypassword (not echoed)
Last login: Mon Jun 9 11:49:28 from 192.168.1.1
[username@SERVERNAME directory]$ cd /apps
[username@SERVERNAME apps]$ ls -l
total 111136
-rwxrwxrwx  1 root   root   25099003 May 19 2002 j2sdk-1_3_1_03-linux-i386-rpm.bin
-rw-r--r--  1 root   root   25307714 Feb 20 2002 jdk-1.3.1_03.i386.rpm
[username@SERVERNAME apps]$

 

 

The Telnet server first displayed its configuration information (identifying it as a RedHat 7.3 server) and then prompted me for to login. It terminated its prompt to me with "login: ". I then entered my username and pressed the carriage return and then it prompted me for my password with the prompt "Password: ". (Note the spaces after the colon)

It then told me the last time I logged in and from where and then placed me in my home directory. All of this was terminated by a prompt for me to do something, terminated by "$ ". I changed directories to "/apps" (followed by carriage return) and then it prompted me for another command (again "$ "). I asked for a long directory listing ("ls –l" + carriage return) which it gave me and then prompted for another command.

So for this server I know that the sequence of events is:

1. Display information and then prompt for a login with "login: "

2. Prompt for a password with "Password: "

3. Display results of the last action and prompt me for a new command with "$ "

So from a programmatic perspective, we are going to have to read character-by-character until we receive one of these key events. In listing 1 I added a method called readUntil(String) that will read character-by-character from the Telnet server until it matches the String passed to it; for a production project, you might want to put error handling in this method to set a timeout to abort if you do not find the String – as it is right now it will keep reading forever.

The first thing that the main() method does is construct a TelnetSample instance. The constructor of the TelnetSample class connects to the specified Telnet server on port 23, obtains input and output streams, and then calls the readUntil() method to read characters until the server prompts for a login. Next it calls the write() method to send the user's username. The write() method calls the PrintStream's println() method followed by a call to flush(). Most buffered output streams/writers will maintain a buffer that gets written to on each write or print and then burst the information to the server all at once: this is a good practice when sending a lot of data, but when you only need to send a small amount you have to explicitly tell the stream/writer to flush to buffer to the server.

At this point the server has our username and if it is valid (no error checking in this sample code) it will next prompt for our password. The constructor next calls readUntil("Password: ") and subsequently writes our password to the server. The final step is to read until we receive a prompt ("$ "). I externalized the prompt "$" to a character variable because later in the program we are going to need to switch users and become root and whenever you are root, the prompt changes from "$" to "#".

I created a method called sendCommand(String) that writes the command to the server and then reads until it finds a prompt; it returns the contents of what it read. After logging on, the main() method sends the command "cd /mydir/mysubdir" to change the current working directory. It next executes the su(String) method: this method uses the Unix su command to change users to the super-user (or root.) The su command prompts for a password, so the su() method executes the su command, reads until it receives a "Password: " prompt, writes the specified password, and then changes the prompt character from "$" to "#" and reads until it finds that prompt.

The final step is to execute the restart.sh script from the current directory: "./restart.sh" and call disconnect().

Summary

Telnet is a very powerful protocol and because of its vast uses, the protocol itself focuses on the connection and not what the connection is used for. Thus using it in your programs is more difficult than using the Commons/Net FTP class. I hope that this introduction has inspired you to develop this further and I hope that these classes can help you sleep better at night too!

SMTP

An important function that the Commons/Net libraries provide is a strong implementation of the SMTP protocol. The Simple Mail Transfer Protocol (SMTP) is the protocol by which e-mail is transferred from one party to another. It is a very simple protocol to learn but starts getting complex when you start attaching files; luckily, the Commons/Net implementation takes care of all the details for you.

History

The first draft of the SMTP protocol was released under RFC 821, but the latest draft is RFC 2821; this document obsoletes RFC 821, RFC 974, and updates RFC 1123. As previously mentioned, you can view the contents of these RFCs at the following URL:

http://www.rfc-editor.org

The protocol consists of several commands, but the following are the most common:

• HELO/EHLO: Used as a greeting to initiate an e-mail conversation

• MAIL: Command "MAIL FROM: ..." denotes the sender of the e-mail

• RCPT: Recipient command "RCPT TO: ..." denotes who the e-mail is being sent to

• DATA: Marks the beginning of the body of the e-mail message; the data is terminated by a line with nothing but a period '.' on it

• QUIT: Terminates this session

• HELP: Displays these plus the other commands

As I mentioned earlier, the best way to test a protocol is to open a Telnet session to the server and start typing. In this case, you need an SMTP server, which may require to you check with your Internet Service Provider. Open a Telnet session to the SMTP server on port 25.

A quick configuration note: you may need to turn your local echo on to see what you are typing. In Windows Telnet this is accomplished by typing the following at the Telnet command prompt:

set LOCAL_ECHO

 

For example:

Microsoft Telnet> open mail.myserver.com 25
220 blabla.mailserver.com ESMTP Sendmail 8.11.6/8.11.6; Mon, 
16 Jun 2003 23:08: 49 -0500
helo myserver.com
250 blabla.mailserver.com Hello ip11-11-11-11.hostname.com 
[11.11.11.111], pleased to meet you
mail from: me@home.com
250 2.1.0 me@home.com... Sender ok
rcpt to: shaines@myserver.com
250 2.1.5 shaines@myserver.com... Recipient ok
data
354 Enter mail, end with "." on a line by itself
Subject: test
This is a test

.
250 2.0.0 h5H49FX28364 Message accepted for delivery
quit
221 2.0.0 blabla.mailserver.com closing connection

Connection to host lost.

All the statements that I typed are bolded above.

Depending on your SMTP server's security settings (and the company that hosts it), you may or may not have to send an additional argument to the HELO command specifying the server that you are going to send mail to. After saying "hello" to the server, it was nice enough to tell me that it was "pleased to meet" me and identified my IP address and host name. I told it that I am sending mail from "me@home.com" and to "shaines@myserver.com". I started the message using the DATA command (it is not case sensitive) and the first thing that I sent it was the subject "Subject: test". There is no protocol-specific mechanism for setting the title; it just must come before the body of your data. It is, however, identified by the prepended "Subject:". Some of the simple options you can send in your message header that will appear in the recipient's e-mail message are:

• "From:" - The sender of this e-mail

• "To:" – The recipient of this e-mail

• "Cc: cc1, cc2, ..." – The list of people carbon copied on the e-mail

• "Subject:" – The subject of this e-mail

After typing the body of the message I end with a period "on a line by itself" as the server asked for. And then I terminated the connection with the QUIT command.

SMTP and Commons/NET

Following suit with the other Commons/NET classes we have used, an SMTPClient class is provided as our front-end to talking to an SMTP server. The order of events to use this class are as follows:

1. Connect to the server using the connect() method

2. Login to the server, specifying a fully qualified hostname if required, using the login() method (HELO)

3. Set the sender of the e-mail using the setSender() method (MAIL FROM)

4. Add one or more recipients using the addRecipient() method (RCPT TO)

5. Obtain a java.io.Writer class to send the data of your e-mail message with using the sendMessageData() method (DATA)

6. Construct a header for your message (you can use the SimpleSMTPHeader class to help you or you can write the aforementioned header elements yourself)

7. Send the body of your message

8. Close your Writer

9. Logout from the SMTP server using the logout() method (QUIT)

10. Disconnect from the SMTP server using the disconnect() method

Listing 1 puts all of this together into an example. Two words of caution when compiling this example:

1. Make sure that you have downloaded the Commons/NET classes from Jakarta's Web Site

2. Make sure that you are using version 1.4 or later of the JDK; if you are not then you cannot use the SimpleSMTPHeader class (I found out the hard way while compiling on my 1.3 system)

Listing 1. SMTPSample.java

 

package com.informit.commons;

import org.apache.commons.net.smtp.*;
import java.io.*;

public class SMTPSample
{
  public static void main( String[] args )
  {
    try
    {
      // Connect to the SMTP Server
      SMTPClient client = new SMTPClient();
      client.connect( "mail.myserver.com", 25 );
      int reply = client.getReplyCode();
      if( !SMTPReply.isPositiveCompletion( reply ) )
      {
        client.disconnect();
        System.err.println("SMTP server refused connection.");
        System.exit(1);
      }

      // Login
      client.login( "myserver.com" );

      // Set the sender and recipient(s)
      client.setSender( "me@home.com" );
      client.addRecipient( "shaines@myserver.com" );

      // Use the SimpleSMTPHeader class to build the header
      PrintWriter writer = new PrintWriter( client.sendMessageData() );
      SimpleSMTPHeader header = 
        new SimpleSMTPHeader( "me@home.com", 
                   "shaines@myserver.com", 
                   "My Subject"); 
      header.addCC( "steve@myserver.com" );
      header.addHeaderField( "Organization", "Test E-mail Company" );

      // Write the header to the SMTP Server
      writer.write( header.toString() );

      // Write the body of the message
      writer.write( "This is a test..." );

      // Close the writer 
      writer.close();
      if( !client.completePendingCommand() ) // failure
        System.exit( 1 ); 

      // Logout from the e-mail server (QUIT)
      client.logout();

      // Close the connection
      client.disconnect();
    }
    catch( Exception e )
    {
      e.printStackTrace();
    }
  }
}

 

 

The Commons/NET SMTP classes are all found in the org.apache.commons.net.smtp package. Thus, we import those classes in our example. Listing 1 follows the aforementioned ten steps in order and should be self-explanatory.

Worth noting however, is that we check the return status of the connection by calling the SMTPReply class's isPositiveCompletion() method. This method returns true if the operation was successful, false otherwise. The designers of the Commons/NET SMTP class recommend testing the result of the connection attempt and note that the other methods actually return a boolean value that we would check as a result of every call in a more robust implementation.

The SimpleSMTPHeader class constructor accepts the following parameters:

• From: e-mail address of the sender

• To: e-mail address of the recipient

• Subject: the subject of the e-mail

It also provides the addCC() method that you can call multiple times to add more names to the CC list in the recipient's e-mail (it doesn't actually send those users e-mail messages). It provides an addHeaderField() method to add arbitrary name/value header pairs to send to the recipient. It is worth noting that the actual recipient and sender are specified earlier and the header is used solely to reveal this information to the recipient. Some of the information is duplicated and sent through the header for purely informational purposes to the recipient's e-mail program.

Summary

The Commons/NET classes provide the basic implementation for sending standard e-mail messages to an SMTP server. The implementation is very user friendly.

Plus, there are some shortcut methods to send quick e-mail messages (take a look at the sendSimpleMessage() methods.) The Commons/NET SMTP classes do not yet address the more advanced features of SMTP, such as attaching binary files and performing encryption and signing messages, but give them time.

NNTP

The Network News Transfer Protocol (NNTP) is popularly used across the world to exchange information, ideas, and even files. There are hundreds of News Servers, many of which you can publicly subscribe to. But, chances are, your Internet Service Provider (ISP) has a News Server containing at least a subset of the newsgroups will interest you. If you are not already familiar with News Servers, contact your ISP, find out the address of their server, and jump into a newsgroup such as comp.lang.java. Start reading the messages. Or, even better, if you have a problem you can't solve, post it to the group.

History

NNTP was first proposed through a Request For Comments (RFC) document in 1986 and was given the RFC number 977. Again, you can reference this at many locations on the Internet - one such location is:

http://www.rfc-editor.org

The RFC 977 document defines NNTP as follows:

"NNTP specifies a protocol for the distribution, inquiry, retrieval, and posting of news articles using a reliable stream-based transmission of news among the ARPA-Internet community. NNTP is designed so that news articles are stored in a central database allowing a subscriber to select only those items he wishes to read. Indexing, cross-referencing, and expiration of aged messages are also provided. This RFC suggests a proposed protocol for the ARPA-Internet community, and requests discussion and suggestions for improvements."

You can think of a news server as a set of components:

• The news server (or NNTP server) itself

• A set of newsgroups (or topics) that the news server maintains

• A set of articles that belong to a specific newsgroup

Figure 1 shows this graphically.

Figure 1.

 

Once you connect to a news server, you can obtain a list of the newsgroups it contains, then connect to a newsgroup and look at the list of articles inside the newsgroup.

As with all of the Internet Protocols we have been studying, NNTP specifies how we are to communicate with an NNTP server. So let's look specifically at the common commands in the NNTP protocol as we outline the steps required to find a newsgroup, an article, and read it. (As previously mentioned, the best way to understand any protocol is to use your operating system's Telnet client to connect to the server and type the commands by hand)

1. Connect to the news server; typically news servers are listening on port 119

2. Enter the LIST command to list the available newsgroups

3. Use the GROUP groupname command to select one of the newsgroups to connect to

4. Use the STAT command to choose an article to read

5. Use the HEAD command to read the article's header information (subject, poster, date, etc.)

6. Use the BODY command to read the body of the article

7. Use the NEXT command to start reading the next article

8. Once you are done, issue the QUIT command

So let's put this together in an example. (Bolded text represents the text that I typed in myself and non-bolded text represents the server's response.)

 

Microsoft Telnet> open newsgroups.myserver.com 119
200 NNTP server ready (posting ok)
LIST
215 list of newsgroups follows
...
comp.lang.java 10101 8888 y
... 
GROUP comp.lang.java
211 1201 8888 10101 comp.lang.java selected
STAT 8888
223 4 <3b8d7d03@newsgroups.myserver.com> article retrieved - request text separately
HEAD
221 4 <3b8d7d03@newsgroups.myserver.com> article retrieved - head follows
Return-Path: <news@newsgroups.myserver.com>
To: someone@somewhere.com
From: "A Java Programmer" <someoneelse@somewhere.com>
Sender: "A Java Programmer" <someoneelse@somewhere.com>
Reply-To: "A Java Programmer" <someoneelse@somewhere.com>
Subject: I need help!
Newsgroups: comp.lang.java
X-User-Info: x.x.x.x
X-Original-NNTP-Posting-Host: x.x.x.x
Date: 29 Aug 2001 16:08:18 -0800
Lines: 8
XPident: dr
Status: RO
NNTP-Posting-Host: x.x.x.x
X-Original-NNTP-Posting-Host: x.x.x.x
Message-ID: <3b8d7d03@newsgroups.myserver.com>
X-Trace: 29 Aug 2001 16:38:43 -0800, x.x.x.x
Organization: My Company, Inc.
XPident: dr
Path: newsgroups.myserver.com!newsgroups2.myserver.com!newsgroups.myserver.com!x.x.x.x
Xref: newsgroups.myserver.com comp.lang.java:8888
.
BODY
222 4 <3b8d7d03@newsgroups.myserver.com> article retrieved - body follows

I need some help learning Java...

Thank you

A Java Programmer


.
NEXT
223 5 <6_3b8d7d03@newsgroups.myserver.com> article retrieved - request text separately
...
QUIT
205 closing connection - goodbye!


Connection to host lost.

 

 

In this example we connect to the hypothetical News Server newsgroups.myserver.com on port 119. We execute the LIST command to retrieve a list of all newsgroups in the News Server; note that the parameters following the newsgroup name are latest article number, the first article number, and the posting privileges, respectively. Therefore the following line:

 

comp.lang.java 10101 8888 y

 

 

Shows that the comp.lang.java newsgroup has articles ranging from a low of 8888 to a high of 10101 and posting is permitted ('y' stands for yes.)

Next we choose the comp.lang.java group by issuing the following command:

 

GROUP comp.lang.java

 

 

And the server returns the following:

211 1201 8888 10101 comp.lang.java selected

 

 

This shows us that the command was executed successfully (any return status that starts with a 2 is a success and we received 211) and that there are 1201 new articles starting with article number 8888 and ending with article 10101 (note that although articles are increasing in number as they are posted, some articles may have been deleted: 10101 – 8888 = 1213, but we only have 1201 articles in the newsgroup.)

We start our reading at the first article (number 8888) by issuing the following command:

 

STAT 8888

 

 

And then read the header by issuing the following command:

 

HEAD

 

 

Which returns a list of key value pairs separated by colons. This includes the subject of the message, the sender, who to reply to, the date of the posting, the newsgroups that the article was posting to, and some routing information to see where the article originated from and the network path it took to reach this server.

We issue the BODY command to read the body of the message and then we can proceed to the next article. We use NEXT command to advance to the next article in place of issuing the STAT 8889 command to manually choose the next article number because we do not know which article numbers represent the missing 12 articles.

Finally we terminate our session by issuing the QUIT command.

After this exercise, you can see that the basics of the NNTP protocol are not that intimidating.

NNTP and Commons/NET

As with the other Commons/NET classes we have been studying, our main high-level interface to NNTP servers is the NNTPClient class. All of the NNTP classes are in the org.apache.commons.net.nntp package with NNTPClient being the conduit we will use. The following outlines the steps you will perform with the NNTP classes to parallel the previous example:

1. Create an NNTPClient class

2. Call its connect() to connect to the NNTP server

3. Call its listNewsgroups() to get a list of newsgroups supported by this server; returns an array of NewsgroupInfo instances, one per news group; similar to the LIST command

4. Once you have selected a newsgroup from the list, you can call the NNTPClient's selectNewsgroup(String) method; similar to the GROUP command

5. Call the NNTPClient's getReplyString() method to discover the first article number, last article number, and article count

6. Call the NNTPClient's retrieveArticleHeader(int articleNumber) method to retrieve a Reader that can read the header; similar to the HEAD command

7. Call its retrieveArticleBody(int articleNumber) method to retrieve a Reader that can read the body of the message; similar to the BODY command

8. Construct an instance of the ArticlePointer class that holds the current article number in it and call the NNTPClient's selectNextArticle(ArticlePointer) method; you can call this method repeatedly until is returns false, indicating that there are not any more articles to read; similar to the NEXT command

9. Logout from the Newsgroup; similar to the QUIT command

10. Close the connection

Listing 1 is a complete example that does the following:

1. Connects to a news server

2. Lists the newsgroups in the server

3. Connects to the "comp.lang.java" group

4. Lists the subjects of all the articles in the group

5. Reads the body of the first article

Listing 1. NNTPSample.java

 

package com.informit.commons;

import org.apache.commons.net.nntp.*;

import java.util.*;
import java.io.*;

public class NNTPSample
{
  public static String getArticleSubject( Reader r ) {
    StringBuffer sb = new StringBuffer();
    try {
      BufferedReader br = new BufferedReader( r );
      String line = br.readLine();
      while( line != null ) {
        if( line.indexOf( "Subject:" ) != -1 ) {
          return line.substring( line.indexOf( ':' ) + 1 );
        }
        line = br.readLine();
      }
    }
    catch( Exception e ) {
      e.printStackTrace();
    }
    return null;
  }

  public static String getArticleBody( Reader r ) {
    StringBuffer sb = new StringBuffer();
    try {
      BufferedReader br = new BufferedReader( r );
      String line = br.readLine();
      while( line != null ) {
        sb.append( line + "\n" );
        line = br.readLine();
      }
      return sb.toString();
    }
    catch( Exception e ) {
      e.printStackTrace();
    }
    return null;
  }

  public static ArticlePointer getFirstArticle( String replyString ) {
    StringTokenizer st = new StringTokenizer( replyString );
    int status = Integer.parseInt( st.nextToken() );
    if( status == 211 ) {
      int articleCount = Integer.parseInt( st.nextToken() );
      int firstArticle = Integer.parseInt( st.nextToken() );
      int lastArticle = Integer.parseInt( st.nextToken() );
      String groupName = st.nextToken();

      System.out.println( "\tGroup Name:  " + groupName );
      System.out.println( "\tFirst Article: " + firstArticle );
      System.out.println( "\tLast Article: " + lastArticle );
      System.out.println( "\tArticle Count: " + articleCount );

      ArticlePointer ap = new ArticlePointer();
      ap.articleNumber = firstArticle;
      return ap;
    }
    return null;
  }

  public static void main( String[] args ) {
    try {
      System.out.println( "Starting..." );
      NNTPClient client = new NNTPClient();
      System.out.println( "Connecting..." );
      client.connect( "newsgroups.myserver.com", 119 );
      System.out.println( "Connected to newsgroups.myserver.com, " +
            "allowed to post = " + client.isAllowedToPost() );

      // List all of the Newsgroups
      NewsgroupInfo[] groups = client.listNewsgroups();
      for( int i=0; i<groups.length; i++ ) {
        System.out.println( groups[ i ].getNewsgroup() + 
                  " (" + 
                  groups[ i ].getArticleCount() + 
                  "), range = " +
                  groups[ i ].getFirstArticle() +
                  " to " +
                  groups[ i ].getLastArticle() +
                  ", Posting = " +
                  groups[ i ].getPostingPermission() );
      }

      // Connect to a Newsgroup and list its messages
      boolean status = client.selectNewsgroup( "comp.lang.java" );
      ArticlePointer ap = getFirstArticle( client.getReplyString() );
      int firstArticle = ap.articleNumber;
      System.out.println( ap.articleNumber + ": " + 
       getArticleSubject( client.retrieveArticleHeader( 
       ap.articleNumber ) ) );
      while( client.selectNextArticle( ap ) ) {
        System.out.println( ap.articleNumber + ": " + 
          getArticleSubject( client.retrieveArticleHeader( 
          ap.articleNumber ) ) );
      }

      // Display the body of the last message in the group
      String body = getArticleBody( 
         client.retrieveArticleBody( firstArticle ) );
      System.out.println( 
         "Article Body for first article in group: \n" + body );

      client.logout();
      client.disconnect();
    }
    catch( Exception e ) {
      e.printStackTrace();
    }
  }
}

 

 

Listing 1 is fairly straightforward, with the only introduced complication being support methods that parse incoming text through the java.io.Reader. The getFirstArticle() method parses and displays the group name, first article number, last article number, and article count, and then returns a new ArticlePointer instance set to point to the first article in the group. The getArticleSubject() method reads characters line-by-line until it finds one that starts with "Subject:" and then returns the remainder of that line. The getArticleBody() method simply reads the incoming Reader until it the stream is complete (signified by the BufferedReader returning null) and then returns the value back as a String.

Summary

The NNTP protocol has widespread application across the Internet and was one of the first forums for sharing ideas and messages around the world. The protocol itself is not complicated once you examine it and the Commons/NET classes provide a simple front-end interface to navigating it.