You can modify system image. Pre-installed applications are located at /system/app/ folder. You can add new application to this folder or remove pre-existing applications from that folder.
Applications from this folder are not removable (if device is not rooted).
As a fast example for customizing Android emulator:
- Go to Android SDK folder, .../platforms/android-9/images/
- Note system.img file, it will be modified (you should back it up)
- Extract system.img using unyaffs
- Customize files inside extracted folder
- Pack new modified system using mkyaffs2image (this tool is built when Android source is built).
- Replace old system.img with new customized system.img
- Create emulator for Android 2.3, launch it and verify that your changes took effect.
This mostly an overview of process, and customization of a real device is much trickier.