It is not officially possible prior to Windows XP SP2 and Windows 2003 SP1. see NWMF_HTMLDIALOG in MSDN documentation for more information.
However, you may "set" the "showModalDialog" property of the window object to a custom function in order to override the default behavior after the Document Object Modal (DOM) is available. The DOM is available between the DocumentComplete event and the BeforeNavigate2 event of the top level browser object.
window.showModalDialog = functionName
here the signature of functionName is
function functionName(sURL, vArguments, sFeatures)
for more information about Dynamic Object Composition , see http://www.microsoft.com/mind/1099/dynamicobject/dynamicobject.asp