Post-Build script to fix MSI issues in Vista for VS 2005 Setup and Deployment Projects

Problem: The setup project (MSI) output generated out of the VS 2005 IDE fails to run properly on Vista. Specifically it has two glaring, known issues:

1. The Deferred Custom Action is NOT marked with the NO_IMPERSONATE bit which results in the custom actions being run under the Stardard User mode.
2. If InstallException class is used in managed code custom action from the System.Configuration.Install namespace, the package throws an ugly "The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2869" instead of the user friendly error message defined in the exception class.

Reason:

1. The problem with the NO_IMPERSONATE bit is that the VS 2005 IDE has very limited support for creating Vista Compatible MSIs and as such "forgets" to mark the custom actions (both, the ones created out of the CustomActionsEditor as well as the managed code Installer class custom actions) with the No_Impersonate value of 0x00000800. As a result, when these custom actions execute in Vista, they will be under the Standard User mode, and any actions which tamper with the registry of invoke external application which need admin credentials start breaking.
2. The InstallException problem is a combination of the new architecture of Windows Installer 4.0 which is packaged with Visa and the 'limited'-ness of VS 2005 while creating MSI. Windows Installer 4.0 expects an entry in th error table for all error codes. The InstallException class throws an error with the code 1001, but this error code is not present in the MSI's error table due to which we get the ugly error! The funny part about this entire goof is that only Windows Installer 4.0 behaves like this with the previous version showing a proper error message to the user!

Solution:
The solution to the above two problems is to include the below javascript (which I have culled out of the various sources in Internet and have made it more robust) in the Post-Build event of the VS 2005 or in the MSBUILD task. 

---

// PostBuildVistaFix.js <msi-file> 
// Performs a post-build fixup of an msi to change all deferred custom actions to NoImpersonate and adds an entry in the error table
// Constant values from Windows Installer

var msiOpenDatabaseModeTransact = 1;
var msiViewModifyInsert = 1
var msiViewModifyUpdate = 2
var msiViewModifyAssign = 3
var msiViewModifyReplace = 4
var msiViewModifyDelete = 6

var msidbCustomActionTypeInScript = 0x00000400;
var msidbCustomActionTypeNoImpersonate = 0x00000800;

if (WScript.Arguments.Length != 1)
{
    WScript.StdErr.WriteLine(WScript.ScriptName + " filename.msi");
    WScript.Quit(1);
}

var filespec = WScript.Arguments(0);
var installer = WScript.CreateObject("WindowsInstaller.Installer");
var database = installer.OpenDatabase(filespec, msiOpenDatabaseModeTransact);

var sql
var view
var record
var errorRecordFound = false;

try
{
// first set the NO_IMPERSONATE bit for all the custom actionss
sql = "SELECT `Action`, `Type`, `Source`, `Target` FROM `CustomAction`";
view = database.OpenView(sql);
view.Execute();
record = view.Fetch();

while (record)
{
    if (record.IntegerData(2) & msidbCustomActionTypeInScript)
    {
        record.IntegerData(2) = record.IntegerData(2) | msidbCustomActionTypeNoImpersonate;
        view.Modify(msiViewModifyReplace, record);
    }
    record = view.Fetch();
}
// close the view.
view.Close();

//check whether a row exists in the error table, if it doesn't create it

sql = "SELECT `Error`, `Message` from `Error` where `Error`=1001" ;
view = database.OpenView(sql);
view.Execute();
record = view.Fetch();

if(record)
{
    errorRecordFound = true;
}
view.Close();

if( !errorRecordFound )
{
    // create a new view to insert an error row in the error table so 
    // that an error message is displayed in vista, instead of package error!

    sql = "INSERT INTO `Error` (`Error`, `Message`) VALUES (1001, '[2]')";
    view = database.OpenView(sql);
    view.Execute();
    view.Close();
}

database.Commit();
}
catch(e)
{
    WScript.StdErr.WriteLine(e);
    WScript.Quit(1);
}

---

Published Wednesday, April 25, 2007 7:53 AM by  nikhiln 

Filed under: Vista, Setup and Deployment, VS 2005, Build, MSI

Notes:

1. Add the following code in the PostBuildEvent property of setup project

cscript.exe "$(ProjectDir)PostBuildVistaFix.js" "$(BuiltOuputPath)"

2.Copy the PostBuildVistaFix.js to the release or debug folder of setup project.