If you're looking for a tutorial to configure an SMTP server, you shouldn't be looking for JavaMail. Simply look for a tutorial on your server of choice (Kerio, for example ... or Exim, SendMail, Apache James, Postfix) or ask on Serverfault. Any SMTP-compliant server will play nicely with JavaMail.
Alternatively, you may even use any "standard" mail provider's infrastructure. For example, I use a Google Apps account along with Google's SMTP infrastructure to send mail from our Java applications. Using a Gmail account is a good starting point anyway if you don't want to setup your own SMTP server in order to simply testdrive JavaMail.
As a last option, you might even lookup the MX Records for a domain and deliver your mails directly to the SMTP server of the recipient. There are some common gotchas to workaround tough.
As a last point, you'll have to look into how to avoid that your mails be filtered as spam - which is a huge topic itself. Here it helps to rely on standard providers that will deal with some of the issues you might encounter when hosting your own server.
Btw: Regarding the error message you posted: the SMTP server is denying relaying of messages. This is if your SMTP server (thinks that it) is running on example.com and you're sending as bob@example.net to alice@example.org, you're asking the SMTP server to act as a relay. This was common practice several years ago, until it was - you guessed it - abused by spammers. Since those days, postmasters are encouraged to deny relaying. You have two choices: authenticate before sending mail or send to accounts hosted at your server only (i.e. on example.com, e.g. alice@example.com).
Edit:
Here is some code to get you started with authenticationg (works with Gmail accounts but should do for your own server as well)
private Session createSmtpSession() {
final Properties props = new Properties();
props.setProperty("mail.smtp.host", "smtp.gmail.com");
props.setProperty("mail.smtp.auth", "true");
props.setProperty("mail.smtp.port", "" + 587);
props.setProperty("mail.smtp.starttls.enable", "true");
// props.setProperty("mail.debug", "true");
return Session.getDefaultInstance(props, new javax.mail.Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("john.doe@gmail.com", "mypassword");
}
});
}