Editing Binary Resources with VS

The need occasionally arises to modify binary resources without re-compilation. Say you want to change the manifest-dependencies of a dll you don’t have the source to.  Or you wish to bump up the version of an executable without actually working on it, exactly as, ahem, a good friend of mine sometimes does.

A quick search will get you tons of free and commercial dedicated tools for the task.  I accidentally learnt that you already have such a tool. It’s called Visual Studio.

Just 0pen your binary (exe, dll, ocx etc.) as a regular file from the menu (you can’t drag-n-drop it in). All the file resources are there on the screen, for you to abuse.

Duplicate Volume Serial Numbers

We recently released a product version, with yearly licenses attached to the machine’s Volume Serial Number.  Now it is called a ‘serial number’, and it seems as meaningless and as random as a UID (mine is 34EE-10A0), so it must be a UID. Right?

Well, not quite. This ID characterizes a volume, not a disk. If you have a partitioned disk, just type at a command prompt  ‘dir c:’ and ‘dir d:’ (or whatever) and watch your partitions’ different VSNs. As the link teaches, the VSN data is part of the partition’s extended boot sector, and is no more then a hash of the partition-creation date & time (i.e., disk formatting date & time).  So, it’s not technically unique – if any two disks are formatted (or partitions created) at the exact same time, they’d have identical VSN. Also – since its only 4 Bytes, the chances of a random hash-duplication are very real.  Just for the sports, if it’s evenly distributed and the world has, say, 1 billion computers,  the chances of duplicate-free distribution of VSN is around 0.187^(1 billion). So there are out there in fact quite a few duplicate VSNs.  But hey – unless you’re Microsoft, such global-scale stuff really shouldn’t trouble you. I mean, c’mon – say you have – what, 1000 clients? 10,000?  make it a hundred-thousand clients. You should never worry about the chance of a duplicate VSN. Now should you?

The real and sad answer, as I recently discovered, is that if you have two clients who use an identical computer model (at least by Dell, but probably true for all other major vendors), the chance of them having identical VSN is exactly ONE.

Dell do not separately format and install every hard drive of the kajillion they deploy. They make some master copy, then deep-copy it around (as us home users do with Acronis, Norton Ghost or whatever). As noted, the VSN is part of the data on the disk, and so is copied as well.

We tried to confirm this officialy with Dell, so far without success. The issue has very sparse web presence too, hence – this post. Hope it helps someone.

Memory Fragmentation Trouble

We recently had some weird issues that turned out to emanate from a failure to allocate a large consecutive chunk of heap memory.  (It was an exceptional pain to nail the cause there – maybe more on that in a future post).  The desired allocation was to be  ~400M, and since machines today ship more-or-less-by-default with 2G-4G RAM, there shouldn’t be a real justification for such allocations to fail.  Or should there?

First of all, regardless of your available physical RAM, your real memory playground size is 2G – the bottom half of your process’ address space, its user-mode portion.  Yes, I’m well aware of the /3GB boot.ini switch, and trust me – you don’t want to go there in a 3D application. I was badly burnt there already.  PAE/AWE have downright hostile API sets too – you’d just have to do with 2G.

The real issue here is memory fragmentation.

An obvious solution would be migrating to Win64, and forgetting about fragmentation issues for the near century. Sadly, this was not a feasible option for us: we have a legacy stash of in-house 32-bit custom hardware drivers, and migrating those would be the absolute last resort.

Happily, a  surprisingly short online research gave quite a few constructive 32-bit directions. Here are some.

1. Low Fragmentation Heap is a nice built in feature, on by default since Vista.  you should apply LFH to the CRT heap, retrieved by _get_heap_handle (just try the sample code). Even better – try applying to all process heaps.   There should be no reason not to apply this to all projects, except (screeeeeeeeeeeeech..) it seems the magic doesn’t work on standard debug builds.  Which, well, err, makes it kinda useless.
2. HeapDecommitFreeBlockThreshold is a magical registry key that is advertised to make a noticeable difference. It does so by causing the heap to hold on to small allocations just a bit longer. Such increase of the HeapManager jurisdiction can potentially prevent page ‘theft’ for non-heap usage, thereby reducing some fragmentation factors.
3. Typically a lot of fragmentation (at the 100Megs scale) is caused by sparse mapping of binary images to the process address space, at load time.
   In simpler English, say your process uses forty 1-Meg dll, and maps them to memory in regular 50Meg intervals.  They now sparsely occupy just 40Megs of your available 2G, leaving no consecutive memory chunk larger than 49M!
   To counter that, first map your virtual address usage. Until recently you’d have to use either vadump or direct code instrumentation, but since this summer you have the incredible (as always) SysInternals tool VMMap. When you spot some dll’s that are just teasingly smiling at you from the middle of your address space, use editbin.exe to ruthlessly rebase them away.
4. Pre-designate a large heap (say 500M) at link time, thus giving the heap a head start in the race for consecutive pages.

I decided to try the steps in order of increasing effort, and am overjoyed to say (2) & (4) sufficed. We now successfully allocate 400M chunks.

We did peek into the process with VMMap, though, and it did surface some interesting finds. For one, babylon translator, installed on all our development machines, has the HUTZPA to inject captlib.dll into the very middle of our precious address space.

My hunch says rebasing could indeed hold the highest impact. We may have to try that too eventually – I hope to post with some findings.