Debugging Windows Installer Custom Actions
General
First, you must know the name of the custom action you want to debug. In this example we'll call it MyCustomAction.
Set the MsiBreak environment variable (user or system) to the name of the custom action. You can do this easily by running setx from the command-line:
C:\Windows\System32>setx MsiBreak MyCustomAction
SUCCESS: Specified value was saved.
C:\Windows\System32>
Run your installer.
You should get now get a message box prompt from Windows Installer giving you the name of the process to attach to for debugging:
To debug your custom action, attach your debugger to process <pid> and press OK

At this point, you can use Visual Studio or another debugger such as WinDBG to attach to the specified process.