Portable Delphi 2007 for Win32 - Part I

[EDIT] Even though you can do all listed here, there are a few minor 'gotchas' that you should be aware of, so browse through the 2nd part before following this to the letter, especially if you use the command line rather than a graphical tool to delete the folders. I'm currently typing Part II into Windows Live Writer, so check back in a couple hours if it's not online yet when you're reading this... Unfortunately I ran into an internet problem and then changed my mind and decided to split the work and use WinWord to create a PDF file to upload with the whole start to finish process and only keep the "blah-blah" on the post so that you can more easily follow the process without waving through a ton of blah-blah. But real-life has a way of messing up one's plans! I have been sick for a few days (an annoying flue that kept me in bed and hardly willing to touch a computer). I'm feeling better now and will finish work on this as promised, but it most likely won't happen before the next weekend as I have a lot of work stuff to catch up during the week.[/EDIT]

Ever wanted to carry your development environment with you on a flash memory stick? Well, using Thinstall you can, albeit it comes with a hefty price tag of 5.000 USD... Maybe now that VMWare has bought the company they'll wise up and release a cheaper personal version! I would like one, and have e-mailed them already stating that. If you would too, don't forget to bring it to their attention! (Edit: Thinstall is now known as VMWare Project NorthStar)

There seem to be some cheaper alternatives, and I even tried Xenocode's Virtual Application Studio (VAS for short) but, even though it has a nice visual interface, it has a few fatal flaws. I couldn't find a way to have multiple entry points and even something as simple as selecting the main entry point is painful with an application that has several thousand files as they use a simple non-typing lookup and they display ALL files to be selected. And if your app (or app combo) somehow has the same exe name in two different folders, then you can't tell them apart in that combo-box... But the main problem against VAS was it failing a simple test: virtualizing ActiveState's Perl 5.8.8... That's something that Thinstall did without a glitch...

This is a two-part article, but don't worry: this time both parts are already written before publishing the first one! But I also like to tease a bit, so Part II will only come out in a couple days! That and because I took my notes on a text file while doing things and have to get them into WLW for proper formatting! :)

Thinstalling (TM) Delphi for Win32 2007 (TM) Professional

Those reading my blog will know that I'm a bit anal about files and disk space in my computer: I'm a software developer (among other things), so I like to be in control! It is me and not the machine nor the OS or any given application that should be in control of what goes on on my computer...

In light of that, I have a plethora of small and some not so small utilities that I have in a "ready-to-run" state on a different partition, so that, after a fresh install, I can run those immediately. I also use Delphi (currently the "2007 for Win32" version) and if you couple all that with the lousy installer and/or lousy install script that Delphi ships with you can understand why I have targeted it a few times before.

I also hate the way my system gets noticeably slower as I add programs. That's no big surprise: the big junkyard of Windows, usually known as Registry, easily starts to hold a gazillion of entries slowing it down. Add Windows Installer to the mix (and very few major-named products these days use an installer that is not based on MSI technology) and you can see where all this goes...

MSI technology is one of those nice-in-theory concepts with all it's self-healing that allows a "regular Joe" to keep running, say, Microsoft Office, even after he blissfully deletes some required folders because upon running an office application it will detect that some files are missing and will re-install them. Now, this is bad news for those like me that like to tweak things. Something as simple as deleting some readme's in 30 different languages will trigger the self-healing process and next time you run your app, they'll be back in place... Also, messing up with the MSI files left behind on your disk will usually result in bad news, from missing application icons (why on earth don't they use the proper .EXEs for the shortcut icons?), to more serious problems with self-healing kicking in and asking for original media to put things back there and you not being able to run the app until you please the installer...

So, I decided to try a new angle and this time I decided to experiment with an old tool but one which I only recently got in contact with: Thinstall. In a short sentence, it's VMWare for single applications, that is, it's a virtualization system that allows you to run applications without installing them first. Just double click a "simple" executable file and it will run as if you had installed it on your machine.

So, what are the pluses and minuses of Thinstalling applications?

+ No more re-install whenever you switch machines or re-format your C: drive or reset it to a "working" ghost image;
- You do need to re-activate when you switch machines; (that's an area where those using pirated and cracked versions have an edge over us, licensed users, but that's the price to pay when companies don't trust their customers and with MSFT setting the way, other companies keep following like lambs...)
+ You can run Delphi 2007 for Win32 from a 1 GB USB flash drive even on a machine with NO .Net installed and from a restricted guest account;
+ You can run it from a network share with no concurrency problems;
- Updating a "Thinstalled" app requires a bit more work (more on that on Part II);
- By default, a thinstalled program will have R/W access to "My Documents" and "Desktop" only: every other file/folder accessed, if changed, will be modified in a local copy on a "sandbox"; you can however configure per folder/drive which ones will be writeable or not. Or you can configure your thinstalled app to run in "merged" mode, but that can be tricky if the host computer has a similar version of the app installed.
+ Thankfully, network and removable drives are used R/W by default so if your files are not on a local disk, you don't need to make further changes to modify them.
- You can't Thinstall apps that install device drivers, such as PalmOne apps or a Virtual CD/DVD emulator or even some Anti-Virus.
+ You can Thinstall applications that install and depend on services as those will be virtualized and started as well when you run your app.

In my current setup, using a 4 GB flash drive, I can fit pretty much everything I need, being 1 GB for a ghost image of Win XP SP3 RC2 with nothing but Win XP + AV + , another one for Delphi 2007 and still have 2 GB left for other thinstalled applications and some data files. This allows me to be back in a working condition within 10 minutes, even if something seriously messed up my hard disk big-time...

So, how does one get to Thinstall Delphi 2007 for Win32?

After you get hold of a copy (you can get a trial from Thinstall's website), you do the following:

0 - Start with a CLEAN machine: I recently re-installed my C: partition with Windows XP SP2 and, after changing some settings, I proceeded to make a ghost of that partition to a bootable 2 GB USB flash disk (only uses less than 600 MB after a bit of tweaking). Using a CLEAN machine is important as is choosing the "oldest" one, that is, if you plan on running it on Windows 2000 and XP, don't use XP for the CLEAN machine as it may already have some files that the 2000 will not and those won't be installed and thus won't be captured.

0b- It's also VERY important to do it all in one go and refrain from using non-related programs, such as surfing the net or doing downloads or even cleaning up your temporary folders: whatever you do may get added to the thinstall project and needlessly clutter it.

1. Run "Setup Capture" and do the pre-install capture.
   
2. Install Delphi (including .net runtime and SDK); as for the shortcuts location, I used the option for "All Users" so it would not clutter my relocated "My Documents".
   
3. Launch and activate (if you want the install to be pre-activated, e.g., you're the only one using it); if you don't activate, when you run it you'll have 30 days to do it later, although I believe, but haven't tested, that those 30 days start counting from the day you INSTALLED it and not from the date you first run the Thinstalled version.
   
4. Configure any settings and/or install components and/or install updates.
   
5. Run "Setup Capture" and do the post-install capture.
   
6. Select the "shortcuts" to create and main package name. When you Thinstall an application, you create a single executable file but you can have alternative "entry points" into that file to run other applications that are part of the same capture. For instance, you can have an independent shortcut for running the Help Documentation without first launching Delphi. These are dependent of the "main" package where all the files/registry settings are stored. On my system, and for some reason (GUID conflict?), it suggests to store the capture in "ATI - Software Uninstall Utility". I opted instead to name the capture folder "Delphi 2007".
   
7. Wait a few minutes while it copies over 31.000 files using 3+ GB for my Professional SKU full install.
   
8. On my system, it failed to copy "C:\Documents and Settings\LocalService\NTUSER.DAT" and "C:\Documents and Settings\NetworkService\NTUSER.DAT": just ignore as those should not be copied anyway... This may be a symptom of me not following step 0b! :)
   
9. IMMEDIATELY SAVE this captured installation, that is, save the "Delphi 2007" sub-folder inside the Captures folder: this will be needed later if you want to upgrade your installation. I copied it to my external hard-drive and named it "Delphi 2007 (Full)" to distinguish from a copy that I'll make later after the tweaking is done.
   
10. Now we can start tweaking the thinstallation!
    

Tweaking the Thinstallation

You can do the remaining process on any machine: all you need is access to the thinstall folder and respective Capture sub-folder where you stored your "Delphi 2007" capture. I opted to use the same machine as I will quickly restore it to a clean state by booting from a flash drive and running ghost from there.

1. Go to your "Delphi 2007" capture folder and open up "Package.ini";
   
2. Uncomment (remove the starting ";") the "CompressionType=Fast" line and comment the other one (";CompressionType=None"); this will instruct Thinstall to use a quick compression when creating this application. Whenever in doubt, check Thinstall's online help file for available options.
   
3. Edit the "Sandbox=" line and use a descriptive name. (I used SandboxName=Delphi 2007). Whenever you run a Thinstalled application, any changes to registry and files that are not in the "merged" folders/keys will be saved in a local sandbox. By default, that is in your current user's Application Data\Thinstall\<sandboxname> folder, so, in a typical Win XP install, with a user named Joe and the suggested sandbox name above, this will be "C:\Documents and Settings\Joe\Application Data\Thinstall\Delphi 2007". It is possible to configure the application to clear this folder on exit, but you can also do a more interesting thing: if you create a "Thinstall" folder in the same place where you have your Thinstalled application, that folder will be used to store the Sandbox. This is particularly useful to do when running an application from an USB flash drive as in this way, not only your settings are not stored on the local computer, they will "travel" alongside the application on your flash drive.
   3b- I like to keep my apps in the same drive, a partition of my HDD and a copy of everything on a 4 GB flash drive. In both cases, a neat way to keep all your settings (read: sandboxes) nicely grouped in one place is to add the following line next to the previous one:
   SandboxPath=LocalSandbox\..\..\..\..\.Settings
   This way, and as long as your apps are not stored more than 3 levels deep, all the sandboxes will be created under the .Settings folder in the root of the drive where the applications are. So you can have something like:
   X:\Multimedia\Image\Image Editing App 1.exe
   X:\Multimedia\Image\Image Editing App 2.exe
   X:\Multimedia\Audio\Audio Editing App 1.exe
   X:\Dev\RAD Studio 2007\Delphi 2007.exe
   and, assuming the Sandboxes use the same name as the executables, you'll have the sandboxes created like this:
   X:\.Settings\Image Editing App 1\
   X:\.Settings\Image Editing App 2\
   X:\.Settings\Audio Editing App 1\
   X:\.Settings\Delphi 2007\
   
4. At this point, I did my first "build" which is as simple as invoking the BUILD.BAT file: 1 hour and 22 minutes later (on a 3 year old laptop!), I had a 2.000.782.941 bytes "Delphi 2007.dll" file along with a few .exe entry points for a total of 1.86 GB. Of course, this includes at least the huge 600 MB file with the .NET framework SDK install files. We'll get rid of those soon!
   
   4b- Don't worry about two things: the app won't show the proper icon (we'll use a workaround for an explorer bug soon), and it may take a LOAD of time launching (we'll get to that fix soon too!). If you want to test it out quickly, just launch something like the Command Prompt or the Documentation.
   
5. Thinstall captures are very easy to process: 3 "registry change" files are created containing all the registry changes to HKLM, HKCU and HKU; several "virtual" folders are created such as "%desktop%" which equals the current user's desktop folder and some %drive_x% folders for each drive where changes where made other than in one of the "standard" folders, that is, if the program only makes changes to, say, "C:\Program Files\CodeGear...", you will see no %drive_c% folder created, but if you change the default install to, say, "C:\Delphi2007", then you'll see a %drive_c%" folder with whatever was created/modified there.
   
6. We can safely get rid of uninstall files(*) as there's no point in uninstalling stuff from a Thinstalled app: even though the changes would be recorded in the local sandbox, and it would appear uninstalled, you'd save nothing by doing so and by deleting the sandbox, it will revert back to the "default" state. Thinstall also already ignores changes to some self-healing registry keys, so you can remove stuff without being prompted with a "re-install".
   (*) As we'll see in Part II, there are some exceptions, with Delphi being one of them!
   
7. With that in place, let's delete (you DID make that full copy already, didn't you?!) the following folders/files:
   - %ProgramFilesDir%\Microsoft Visual Studio 8\Microsoft .NET Framework 2.0 SDK - ENU (569,8 MB)
   - %SystemRoot%\Microsoft.NET\Framework\v2.0.50727\Microsoft .NET Framework 2.0 (76,4 MB)
   - %SystemRoot%\Microsoft.NET\Framework\v2.0.50727\Microsoft Visual J# 2.0 Redistributable Package (15,6 MB)
   - %SystemRoot%\Installer (93,9 MB - this was unrelated to Delphi 2007 and probably because I somehow ran something else: I've had this capture on "hold" for a couple days...)
   - %Common AppData%\{6AF0EFC6-B937-4704-A430-319EB93F4C12} - Delete all sub-folders but leave the files as they're needed for the "Check for Updates" option. (423 MB) This one will be in %Local AppData% instead of %Common AppData% if you used the "Just for me" shortcuts option when installing.
   - %Common AppData%\{135F413C-658B-4923-B950-5F2B786BC8DA} (341 MB - Help update setup files)
   - %Profile% (nothing useful)
   - %Common Programs% and %Desktop% - These hold the shortcuts as created by the installer, but we'll use a different set
   - HKEY_USERS.txt (on my system, there's only a handful entries and none Delphi related, probably for me breaking step 0b!)
   
8. Now let's open up Package.ini and fix a few more things:
   - Fix the broken Icon for "Delphi 2007.exe". Find the section [Delphi 2007.exe] and duplicate it. Now do a find/replace (on the whole text file) finding "Delphi 2007.exe" and replacing with "Delphi 2007.dll". Next, change one of those two sections to read [Delphi 2007.exe]. You'll need a few more changes so just check that they look like this:
   
   [Delphi 2007.exe]
   Shortcut=Delphi 2007.dll
   Source=%ProgramFilesDir%\CodeGear\RAD Studio\5.0\bin\bds.exe
   CommandLine="%ProgramFilesDir%\CodeGear\RAD Studio\5.0\bin\bds.exe" -pDelphi
   Shortcuts=%Programs%\CodeGear RAD Studio
   
   [Delphi 2007.dll]
   ReadOnlyData=bin\Package.ro.tvr
   Source=%ProgramFilesDir%\CodeGear\RAD Studio\5.0\bin\boreditu.dll
   
   The "problem" you fixed is that Explorer will try loading the whole .exe file in memory to show the icon, and with an .exe several hundred MBs it will fail. We trick it by storing the "bulk" of the information in a .dll or .bin or .dat or whatever non-exe file and then adding a smaller shortcut to Delphi 2007.exe, as per the Thinstall help section on this topic.
   
   - You can cleanup quite a few more entries in here, or just leave them with Disabled=1 so they won't have shortcuts created alongside with the main executable. I left the following entry points enabled:
   Check for Updates, Delphi 2007, RAD Studio Command Prompt, RAD Studio Documentation, Rave Reports.
   
9. You may want to find and delete all the "Icon=" lines as they force some applications to use an icon stored in an MSI instead of the one in the EXE and deleting the Install cache will get rid of the icons.
   [EDIT]In order to keep the RAD Studio Command Prompt and the Documentation icons, move those two *.ICO files elsewhere or copy them from the "full" version if you have already deleted them from step 7 above and update the Icon= line accordingly. You can, for instance, name them RAD Studio Command Prompt.ico and Documentation.ico and place them straight in the project root directory so that it's clear what they're used for.
   [/EDIT]

1. Also, while you're at it, check the lines with "Shortcuts=" (notice that it's PLURAL Shortcuts with an S at the end: not to be mistaken with the similar but different meaning option Shortcut!).
   In my install, I set all the shortcuts into a Thin\genre\app type of folder, for instance, I use Shortcuts=%Programs%\Thin\Dev\Delphi 2007, so that all shortcuts created (for the sections that don't have Disabled=1), will be stored in a Thin -> Dev -> "Delphi 2007" folder. (More on that on Part II).
   
2. The speed problem seemed to go away as soon as the "main" executable file dropped in size: didn't test to see if it was related to being over 2 GB of uncompressed data or to having a few huge files in there, but it works ok in a 1.7 GB uncompressed version as well as in this 1 GB compressed one...
   
3. Now you can re-build by simply invoking the batch again. On the same machine as above, and because Thinstall caches the files from the previous compilation, it took only 10 minutes to generate the brand new 1.032.472.412 bytes file. This means that, together with the few extra shortcuts or alternative entry points, it now uses up only 984 MB, so it perfectly fits on a 1 GB flash drive. (And we'll trim that further in Part II).

That's all for now. In Part II, we'll tweak a lot more by removing the .net related help content. I will also explain how to update your Thinstalled "Delphi 2007.exe" file so that you can incorporate patches, add/remove/update some components or simply modify default settings. Finally, I'll show you how easy it is to have the proper file type associations so that you can simply double-click an *.dpr file and open up your Thinstalled Delphi. Until then, get your own (30 day trial of Thinstall)* and start playing!

* EDIT: Thanks to Xsintill for pointing out the VMWare NorthStar beta (which is essential the full Thinstall project) but has two things better than the Thinstall trial: 1) lasts until 1st of June 2008 (roughly 70 days from now); 2) The whole registration process for the trial is automated and less restrictive in the choice of e-mails. So go grab that beta and start playing...

More Disk Space Tweaking...

UPDATE 1: DO NOT TRY THIS! Even though everything is working ok on my system, I've just noticed a shortcut icon problem that I believe is related to the order I've done something. I'll have to test it a bit more and will update this post where needed to prevent that from happening but it may take a couple days before I update the post!

Update 2: Apparently it's Windows Installer that is to blame. Not only it completely ignores the junction, it goes even further: deletes the junction (causing all contents to be lost!) and creates a new INSTALLER folder in your Windows subdirectory. So, this particular folder CANNOT be moved or the next time you use msi installer (which is just about any installer or patcher these days), you'll LOSE all that was transferred.

It's always "fun" to see Microsoft NOT playing by their own book!

Or, how to keep your C: drive mean and lean and still don't throw away those lengthy patches and cached install files...
This is not entirely related to Delphi as the techniques used here are more general and thus you can extend this to your own liking. It is however triggered by the fact that I hate to see wasted space on my C: drive, which, as I've told you before, Delphi's current installer likes to do! :)
In short, you'll learn how to relocate some huge folders to another partition or even another hard-disk. This is especially useful for temporary or cache folders as those serve no useful purpose 99,99% of the time but still use up disk space.

Theory

The whole process revolves around Windows XP and NTFS formatted partitions: you can create junctions, which are links to folders or whole disks and access those as if they were native folders on another partition. For instance, you could relocate C:\Documents and Settings\All Users\Application Data to D:\MyData and whenever a program tries to access the first folder it is in fact accessing physical data from the MyData folder in drive D:

To ease up the process, I created a small batch file in 4NT (sorry, but I don't do CMD in years and don't feel like going back!). 4NT has a nice little command MKLNK (and also a newer one MKLINK which is *NOT* what we'll use, so just keep it in mind that there are two very SIMILAR but DISTINCT commands!). The MKLNK command allows you to take advantage of mount points and create/delete them. In the end of this post, I'll list the whole batch file I used. As you can see, using ALIASes and a few 4NT variables and functions makes the whole batch a lot better than if I were to try it out using CMD only!)

Now, I'm sure that there is a MKLNK-alike utility or two around, even though I haven't searched for any, so feel free to ignore the batch and try doing the commands on your own, although you'd probably be doing yourself a big favor by actually trying out 4NT! ;)

Limitations

As mentioned in the batch file itself, which you can download from here, you need to be aware of the following:

• Non-English versions of Windows XP may use different names for Application Data and Local Settings;
• You have to edit the SET %RootFolder= line to set your own root folder. I used T:\Patches but you can use anything else (and even modify the batch to spread things around multiple folders instead of sub-folders of this one)
• This batch file is a first version. In order to keep things simple, it does NOT check for available free space on the target folder but it does warn at start that it doesn't do that and that you SHOULD make sure you have enough or else you risk being left with some stuff moved and other not quite so... I do believe, but have NOT tested, that it should be able to resume from where it left if you run out of free space and then run the batch again, but, I repeat, I have NOT tested that!

Let's get busy!

So, let's start digging into the batch file:



You should edit line 5 and, if running on a non-English OS, change line 9 and comment lines 15 to 17.

Line 20 defines an alias to test for the existence of a folder name, create it if not found (creating parent folders if needed) and then changes to that folder. It's not exactly needed here as it's only used in one place, but I copy/pasted it from my aliases file.

Line 23 defines another alias to echo a blank line, then echo the passed parameters slightly indented.

The %+ symbol equals "whatever command separator is configured on this system". 4NT allows several commands to be executed on a single line, which is useful, among other places, in aliases.



Now we get to the main alias, the one doing most of the work in line 30.
Here we test if the 1st parameter is already a junction: if that's the case, then the "TRUENAME" (which is a 4NT function that returns the actual full path and "sees" through junctions, substs, net shares and symlinks) and we compare that to the "FULL" name, that is, the name after expanding any % variables and processing any ..\ or drive aliases. If they're equal, then no junction is in place.
Here's an example to make it clearer:
Imagine that:

• %windir% equals C:\WINDOWS
• %windir%\$hf_mig$ has been redirected to T:\Patches\$hf_mig$
• %1 equals %windir%\system32\..\$hf_mig$ (this is intentionally complicated to illustrate)
  

ECHO %@TRUENAME[%1] would return T:\Patches\$hf_mig$
ECHO %@FULL[%1] would return C:\Windows\$hf_mig$

If the two match (which is false in this case), then no junction or subst is in place. Seeing that you can only SUBST a whole drive letter and not a folder, it is assumed by the batch that if it fails you already have a junction in place and you get a warning (the ELSE part). If the two strings match, then no junction is in place so it MOVEs the source to the destination, showing only /Totals instead of all files moved, moving /Hidden files, not complaining on /Errors, and moving whole /Subdirectories. Finally, the junction is created to "fake" the system into believing that the whole folder exists but it now points to a different drive.

 

Line 36 shows a message box prompting the user to acknowledge that the batch will NOT check for free space and asking to proceed.

Lines 39-41 stop some services that interfere with moving some of these folders. If you add to the list below, you should first use an utility like HANDLE from SysInternals to check if some program or service is keeping that folder or anything inside open and add the service to the list and maybe even use PSKILL to terminate the application if it's not a service. (Edit: on checking SysInternals website for the links to add here, I noticed a Junction utility that you can use if you don't/can't/won't use 4NT!)

Line 43 creates the root folder if needed.

This is the "bulk" of the work, relocating several folders from my system to the cache folder. Pretty self-explanatory (ignore the color bug in some of the lines: that's UltraEdit not syntax highlighting BTM files properly!)

Finally, I also disable (/CACHESIZE=0) and remove existing entries (/PURGECACHE) in the dreaded %windir%\dllcache folder. You may want to remove or comment out this line if you rather keep half a gigabyte of duplicate dlls on your disk.

Lines 69-75 remove ALIASes and SETs used by this batch. Not really needed, but I like to clean up after myself! :)

And that's all for now. There are more folders that can be safely added to this list, and, on my system, this saved me 2.5 GB worth of C: space. EDIT: Even not counting Installer, I still save 1.6 GB worth of data...

End Notes

If you're thinking of doing this to, say, the user profile directory, you'll need to do it from another account and not have logged that account in the current work session. But I would advise you NOT to do it! I once did it, and even in a more "crude" way requiring messing around with the registry, but then I found that doing it saves a LOT of trash! So, upon restoring a working C: image, I'd have my profile all set with all the trash accumulated from all the programs installed/removed/changed.

You can download the batch file from here. (Yes, I know I've linked it above, but it's easier to find it here if you come back later!)

Quick tip for compiling/testing DLLs

Now that Delphi 2007 is using MSBuild, you can take advantage of the Pre-Build events to automate a common headache: killing that task that is using your DLL before you build it!

I'm currently developing an Outlook add-in so I found myself repeatedly quitting Outlook or getting an error because Delphi can't generate the DLL if I don't close Outlook, so, to solve that, and since, for some weird reason, I got used to always build instead of compiling, I just added the following command to this project's Pre-Build Events:

taskkill /F /IM outlook.exe

DIR >NUL

And that's all there is to it. You can also launch outlook after the build so as to immediately test your changes by adding the proper command to the Post-Build events, but I didn't add that as sometimes I do a few builds just to check syntax/compiler errors while I'm still working on something and I don't want to have outlook popup everytime...

Also, you can try without /F (force) and if your outlook or dll host can terminate gracefully in a speedy way you won't need to force termination, but in my PC and with Outlook it doesn't quit that fast, so I need the /F...

EDIT: The DIR >NUL command is just so that, should you not currently have outlook running, the error from taskkill won't stop your build process! :)

BTW: you can achieve something similar in BDS 2006 (and maybe 2005: haven't tested on this one), if you read my blog post about adding pre/post build events to BDS 2006...