DDKBUILD - Integrating the Windows DDK with Visual Studio .Net

 

What's New The latest released version is 3.13. This version adds quiet mode - the script attempts to reduce all output to stdout to the minimum, simplifying the use of ddkbuild in automated build procedures. Credit goes to Beverly Brown at Mercury Computer for the quiet mode implementation. The Vista DDK (build 5270) is supported. WDF (KMDF1.0) is released and 3.13 supports that as well. DDK's going back to the Windows 2000 DDK continue to be supported. In addition a lot of bugs have been fixed. Bug fixes were suggested by Daniel Germann, Thomas Schimming, David Craig, Norman Diamond and probably other people I forgot about. Version  3.12. This version added support for the drvfast version of prefast builds and support for CUV builds. The following people have provided enhancements to the 3.12 version of ddkbuild: Hannes Sternerson, Spiro Trikaliotis. Version 3.11 added support for WDF (Windows Driver Framework) builds. Version 3.9 added support for  the Longhorn DDK (LDK Beta version 4051) .NET XP W2K DDKs. NT4 platform support has been dropped from version 3.9, although you should be able to get the NT4 DDK to work on a W2K or later platform. Version 3.8 added support for the .NET,  XP, W2K and NT4 DDKs. Continued support for W2K and XP DDKs is problematic. Multiple DDK Support This has been a feature for a while, but has not been adequately documented. The idea is to allow one to build visual studio projects with multiple configurations, one for each targeted major release (and it would appear that .Net server qualifies as a major release as it has its own DDK.) On my development system I have the  W2K, XP, Longhorn Beta, W2K3 and W2K3SP1 DDKs installed, and I have visual studio projects that I  can build for all of  these NT DDK versions and their related platforms, simply by selecting the appropriate project configuration. Ddkbuild uses environment variables and command line arguments to choose the correct DDK and OS platform. Ddkbuild has only a limited idea about where the DDK you want to use is installed. Once upon a time the DDK (and that is singular,) was installed wherever the BASEDIR environment variable said it was installed. That was a bit limiting, in that one system (actually the limit is probably one user on a system,) could only use one flavor of DDK. Starting with the W2K DDK, the use of BASEDIR was abandoned, and instead the location of the DDK root directory was computed dynamically each time a DDK shell was invoked. If no target DDK is specified DDKBUILD gives up and displays usage information. To use other DDKs you need to do two things: set a specific environment variable to the location of the DDK root directory set a corresponding command line flag when you invoke ddkbuild For the W2K DDK the environment variable is W2KBASE, and the flag is -W2K. For the XP DDK the environment variable is XPBASE, and the flag is -XP. For the Windows 2003 Server  and W2K3SP1 DDK the environment variable is WNETBASE, and the flag is -WNET. For the LDK (Longhorn Beta build 4051) the environment variable is WLHBASE and the flag is -WLH For WDF builds, the environment variable WDF_DDK must be set to the root directory of the supported DDK for WDF builds. As of the release date for version 3.12 of ddkbuild, this was the 3790.1289 W2K3 SP1 Beta DDK. You must of course setup WDF builds as documented in the WDF documentation. For example you also have to set the WDF_ROOT environment variable correctly. Variations on a theme Starting with the XP DDK, the DDK build environment supports multiple target OS platforms, (actually there is support for 64bit W2K platforms in the W2K DDK, but like who is using that?) This has introduced variations on the above command line flags (the environment variable remains the same,) to reflect the various targets that the selected DDK can build. So, -XPW2K uses the XP DDK to build a W2K target, -WNETXP uses the .NET DDK to build an XP target, etc. Support for multiple DDKs is getting a bit cumbersome, particularly as the XP DDK supports building W2K and 64bit targets and the .NET ddk supports building W2K, XP and 64bit targets.  At some point I'll have to start limiting the variations here. Other Stuff If you are not running the standard NT shell, like for example if you are running that thing on windos/me9x, or NT4, please do not bother me with your complaints. Sorry. Introduction A long, long time ago in a far away place I started building NT operating system components. Much to my horror I discovered that rather than a nice gui'd set of tools, the standard kernel build environment looked a bit like this:  Figure 1. I don't know, I'm sure it is all subjective, but the word "fugly" comes to mind.  The standard NT kernel build environment reminds one of the ultra-sophisticated (and I suppose oh so retro-trendy these days, freakin penguin heads everywhere,) pre-X, pre-Emacs Unix development environment of vi and make, except of course that notepad is far worse than vi. I really love that NT 'shell' (and I use the term loosely) and in fact the idea of groveling through the scrolled-away output of build, or those lovely build log files using notepad, searching for error messages, combined with whatever code editor floats your boat (as in you need three applications with no clue of what the others are doing,) just thrills me. I'm sure you feel the same way. I have a few simple requirements when it comes to software development: Editor/Compiler integration. Source code browsing (as in Unix tags.) That isn't a very big list, is it?  In addition, I don't want my development environment to do anything that might even possibly compromise the actual output of the process. In other words, I want to be absolutely (or at least very highly) certain that whatever tools I use are producing valid results. So faced with what to me was the unacceptable NT DDK way of doing things, I set out to develop my own tools that would integrate the VisualStudio IDE with the NT DDK, getting me most of what I find useful in the IDE without compromising at all the integrity of the build process. Thus was born ddkbuild.bat. DdkBuild DdkBuld has gone through three incarnations. It was born at Sequoia Systems, in the midst of a rather painful corporate collapse. It traveled from Sequoia to OSR with its creator, and there became the standard VisualStudio/Build integration tool for OSR and, rumor has it, quite a few other places as well. A deviant version of ddkbuild is available at OSR. I've never used it, but I've heard it works. Currently Ddkbuild supports the W2K3SP1 and earlier DDKs back to the Windows 200 DDK. It also supports WDF builds, prefast builds, the drvfast variation of prefast builds and CUV builds. What is ddkbuild? It is an NT command shell script. A bat file.  What does ddkbuild do? It does a lot of things, but mostly it does exactly what that crufty checked or free build environment ddk window does, that is it sets up the correct environment for executing build, and then in invokes build. Huh? What's the point? Oh, yes well ddkbuild does things in such a way that it is really easy to set up a VisualStudio project that will, through the vehicle of ddkbuild, integrate with the NT build process. Oh, fine but I'll just use that SrcToDsp thing from Numega. Ok, that's a really neat tool. It does have its limits though. I'm still looking for the DspToSrc utility :-) Remember that what I want is a process that uses the standard NT build tools and integrates with VisualStudio. Ddkbuild does not preclude using the DDK shell windows for building your projects, in fact it encourages it. Get On With It Already Ok, so here is the step-by-step algorithm for moving an existing NT DDK project into the IDE with DdkBuild. We will also discuss very briefly how to create a new build project in the IDE. Through some sort of miracle I've actually updated this discussion to be relative to Visual Studio .Net 2003! Isn't that exciting? But first a word from our sponsor: folks please remember that what drives the creation of a NT DDK Build based software project is the Sources file that describes the project components. The process I'm describing here works with the DDK build process it does not replace it. Let's take an existing W2K3 driver, in fact we will use the one pictured way up as the fugly figure 1, the isousb driver. 1. Start your engines Fire up VisualStudio .Net 2003. From the File menu select New Blank Solution, which ought to bring you to this screen: Figure 2. New Project Dialog (less fugly) Note that we have chosen to create a Visual C++ Project using a Makefile Project Template. Also note that we have named the project isousb, that the location for the project is E:/test, and that the project will be created at E:/test/isousb. Note also that in the command line window shown earlier, E:/test/isousb is the directory we used for building the isousb ddk component from the W2k3 ddk command shell window. That directory contains a DIRS file and subdirectories named exe and sys containing auser mode test program sources and the driver sources. Our solution will build both the user mode and kernel mode components. 2. Wizards. Click OK to create the new solution. This brings up the next panel - the Makefile Application Wizard. We will set up preliminary Debug and Release settings for our project using this panel by clicking on the Application Settings element of the panel. Figure 3 What Would We Do Without Wizards? Note a couple of things here. Ddkbuild.bat has to be found through the PATH environment variable. The environment variable WNETBASE has to be set to the path to the installed location of the W2K3 DDK on this system. We have specified build and rebuild command lines for ddkbuild. We have no bothered with a clean command line. Finally, the Output field is meaningless. 3. Projects Click on the Finish button and Visual Studio sets up our initial view of the project. At this point we can build the project, and if we have done everything correctly it will build. Lets take a look. Figure 4 Like a Project, only fuglier. We selected "Rebuild Solution" from the Build menu. Our Visual Studio external makefile project used ddkbuild to invoke the W2K3 ddk build utility, and rebuilt both the driver and the test program using the DDK and its built in compilation tools. Note that oddly enough there are no source files, include files or resources configured for our project. The project simply invokes ddkbuild. Ddkbuild in turn invokes the W2K3 Build utility, the Build utility uses the DIRS file in the current directory (E:/test/isousb in this case) to build the ddk components in the exe and sys directories. The exe and sys directories each have a SOURCES file that Build uses to compile these components into an application and a driver. Lets make our solution look more like what we are actually building. First, add the DIRS file to the isousb project. Then create project folders for exe and sys and put Source Files, Header Files, and Resource Files folders in each of these folders. Now add the source files, include files, and resource files for the application and the driver to their appropriate folders. Also add files like the SOURCES files to their appropriate locations. Here is what the project looks like after we've added all of its files to it. Figure 5 A project fulfilled. 4. This Project's got class Notice that this is the Solution Explorer view of the project. The Class View is also enabled. If you click on the Class View tab you will see something like this: Figure 6 Neat, but no you cannot use C# in a driver (yet.) 5. Symbolism All of the data structures and functions defined in the project are browsable from the  Class View. Note also that you can use Visual Studios Find Symbol function to locate the definitions of variables, data structures, and functions defined in the isousb project. For example if you select IsoUsb_DbgPrint in the source file window depicted above and then invoke the Find Symbol function (Alt F12 or the toolbar or the Edit menu.) the Find Symbol dialog pops up, followed by the Find Symbol Results window: Figure 7 Sometimes a cigar is just a cigar Double clicking on the highlighted line will in fact navigate to the isousb.h file and the definition of IsoUsb_DbgPrint. Nice. Now expand the symbol results for IsoUsb_DbgPrint so we can find all the references to it: Figure 8 OK so these captions don't make sense, what can I say? Hmmm.... what gives? Oh, we need to enable source browsing. So it seems we just need to enable 'Build Browse Information' and we should be all set. Not quite. We have to digress a bit here. External Makfile projects in visual studio don't really support bscmake, worse the DDK build tools do not include bscmake. We are going to have to commit a bit of hackery here. As you have access to Visual Studio .Net you have access to a copy of bscmake.exe. That is good. Now you have to locate that copy somewhere that the DDK can pick it up from, as in it has to be in your path. To be sure, open a DDK shell window and examine the PATH variable from that window's perspective. Now comes the further complication: we have to set the SOURCES file up to build a bsc file for our project. So in each SOURCES file (there are two, one for the driver, and one for the test application) add the following lines: # # setup visualC++ source browsing # BROWSER_INFO=1 BSCMAKE_FLAGS=$(BSCMAKE_FLAGS) -n Now rebuild the solution. If you examine the Output Window from the build you should see something like this: Figure 9 Convincing Build to Do bscmake. Note that ddkbuild claims that it is building browse files. Excellent. Lets go find all the references to IsoUsb_DbgPrint. Figure 10 the fugly parade continues. What? Ok. There is one more step. For external makefile projects you have to add the bsc file to the project manually. Use the Add Existing Items dialog by right clicking on the isousb project and add the two bsc files, one from the test application and one from the driver, to the project. Now try Find Symbols on IsoUsb_DbgPrint: Figure 11 Finally Symbol browsing is now enabled for the project. Note that the bsc files are in the output directory for each of the isousb components. For example the driver's bsc file is located in isousb/sys/objchk_wnet_x86/i386/isousb.bsc. 6. Fixing up the Release build The wizard setup Release and Debug builds, however the settings for both builds are identical. We need to modify the Release build to create a free version of our driver and test program. Right click on the isousb project in the Solution Explorer view and select Properties. This brings up the isousb Property Pages dialog. Select the Release Configuration. It should look like this: Figure 12 The Release Build before going on the driver fast diet plan. You have to change the ddkbuild buildtype value from checked to free. It should look like this: Figure 13 The Release Build after just 12 weeks on the driver fast diet plan. Guess what? That is basically all there is. We've put the isousb project into Visual Studio 2003.Net. We can build it in the IDE, using the DDK. We have source browsing via bscmake. We have Class View and Solution Explorer working for us. But you do have to be aware that this is a DDK project, that its build settings are specified from the SOURCES file not from any options we might set in Visual Studio. For example, if you add a new source file to the driver project, you have to add it to the SOURCES file or it will not get compiled. 7. Lets use Prefast Prefast is a pretty neat tool. So lets create a prefast configuration for our isousb project. From the Build menu select Configuration Manager. This will pop up the Configuration Manager dialog box, which looks like this: Figure 14 Gonna Prefast this Project From the Active Solution Configuration drop down list select <New...> which brings you to yet another dialog: Figure 15 This Caption Intentionally Left Blank Make your New Solution Configuration look just like this one and click OK and the close the Configuration Manager dialog as well. Now right click on the isousb project in the Solution Explorer view and select Properties to bring up the Property Pages dialog. We have to edit the settings for the new Prefast configuration to add -prefast to the ddkbuild command line. In addition to adding -prefast, we are settng the -cZ flag for the build command line as we want to do a rebuild anytime we do a prefast build. When you are done the Propery Pages dialog should look like this: Figure 16 Ready to Prefast Click OK and then build the prefast configuration. The Prefast View window should pop up and look like this: Figure 17 Postfast? Is that just ducky or what? Note that if you have the W2K3 SP1 DDK you can use the -drvfast option to the latest release of ddkbuild and that will invoke prefast with the driver presets. Note also that the latest version of ddkbuild supports CUV through the -cuv option, so you can set up a CUV configuration as well. Appendix - Project Management If you add or remove source files you have to modify both the VisualStudio project and the Sources file. Consequently if you are going to create a new project, you have to remember to create a Sources file, a makefile, and anything else you might need. VisualStudio will not do these steps for you as it doesn't know anything at all about how to build a device driver or what build.exe is and what its requirements are. We've found that the best way to manage large projects is to have a regular old Makefile for the top of the project build tree. This makefile can do things like build multiple configurations (checked/free/w2k/xp,) build releases, documentation sets, installation packages, etc. Our projects are all under a 'overnight build' system that uses this top level makefile to drive the process. Here is a sample makefile to illustrate what I mean: # build all components all: free check doc # clean all components clean: clean src # clean and then build all allclean: clean all free: ddkFree ddkXpFree vsFree check: ddkCheck ddkXpCheck vsCheck ddkFree: ddkbuild -W2K free . w2k @-if exist buildfre.err type buildfre.err ddkCheck: ddkbuild -W2K checked . w2k @-if exist buildchk.err type buildchk.err ddkXpCheck: ddkbuild -XP checked . xp @-if exist buildchk.err type buildchk.err ddkXpFree: ddkbuild -XP free . xp @-if exist buildfre.err type buildfre.err vsFree: msdev sample.dsw /MAKE "user - RELEASE" vsCheck: msdev sample.dsw /MAKE "user - DEBUG" doc: doxygen Ddkbuild Command Line Usage Download Latest Ddkbuild.bat