Debugging Memory Leaks, part 3.5: Hacks with Hooks

Alan rightfully comments that setting a conditional breakpoint at _heap_alloc_dbg  significantly slows down the application.

If run time is an issue for you even in debug builds and re-compilation is not an issue, here’s an alternative trick: use an allocation hook that detects a given allocation size. Within the hook itself you can set a break and detect the stack yourself, or dump the stack via a tracepoint.  Somewhere early in the application you’d need to assign it with _CrtSetAllocHook.

The hook can be as simple as:

int MyAllocHook( int allocType, void *userData, size_t size, int  blockType, long requestNumber,
const unsigned char *filename, int lineNumber)
{
if(size==68)
DebugBreak();
return 1;
}

And the assignment as simple as:

BOOL MyApp::InitInstance()
{
_CrtSetAllocHook(MyAllocHook);

…

To dump the call stack to the output window using the debugger, set a tracepoint in a line that runs only when your requested size is detected:

(Alternatively, set a regular breakpoint, then right click it at the breakpoints window and select ‘when hit…’).

Then, type $CALLSTACK at the message line:

Allocation hooks are often used in much heavier debugging facilities – usually involving procedurally walking call stacks (more on that some day). This is quite an overkill for a hack such as this, and the debugger supplies convenient alternatives anyway.

Debugging Memory Leaks, Part 3: Breaking on Allocations of Given Size

When battling memory leaks, you often start from the output of _CrtMemDumpAllObjectsSince or _CrtDumpMemoryLeaks (called for you if you use MFC) – something similar to:

C:\myfile.cpp(20): {130} normal block at 0x00780E80, 68 bytes long.
Data: <      > CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD
Object dump complete.

I’ve mentioned a way to break at the 130th allocation by setting _crtBreakAlloc to 130 (either by _CrtSetBreakAlloc or from the debugger). Unfortunately this works only if the allocation number is consistent across runs – and this is not the case when the code contains threaded regions (with allocations) up to the leaking allocation.

As an alternative, here’s another trick that works very well for me: the allocated sizes are virtually always consistent across runs. They are not guaranteed to be unique, of course, but you can often notice at least one seemingly non-random requested size – notice the bold 68 at the dump above. Once you do, all that remains is to set a breakpoint at _heap_alloc_dbg, and set a condition on it:

nSize == RequestedAllocSizeToTrack .

A glimpse at the code might help:

There is no direct CRT API for this, but the hassle is minimal and I prefer anyway to maximize the use of the debugger and avoid recompilation.

I hoped to be able to set the breakpoint from the breakpoints window using the context operator, but I can’t locate _heap_alloc_dbg in msvcrXXd.dll exports.  Any advice would be appreciated!

You can try similar tricks with _malloc_dbg (which is visibly exported), but that would probably work only if you’re positive that’s where your allocation are channeled through (and not, e.g., _aligned_malloc_dbg).

Checking Memory Corruption with _CrtCheckMemory – From the Debugger

There are several ways to locate memory corruptions, but probably the easiest is to spread _CrtCheckMemorys around and pin the culprit (hopefully) in a binary search scheme. This requires recompilation at each search stage and thus can get tedious quickly.

Enter the forgotten, mis-documented, context operator. Turns out there’s a surprising variety of system/CRT operations you can invoke using it, from the debugger, while at a breakpoint (or the debugee is otherwise halted).  Since _CrtCheckMem is implemented in msvcrtXXXd.dll (XXX being VC version), the syntax to invoke it, e.g. in VS2005, is -

{,,msvcr80d.dll}_CrtCheckMemory()

Just type it at a watch window, immediate, or even a QuickWatch – anywhere the expression evaluator can find it.

In my case it quickly turned out I miscalculated the number of vertices in a mesh, and wrote beyond the vertex buffer limit. Last breakpoint before the corruption:

And immediately after:

You can probably pull similar stunts with many other CRT debug routines - just not those that directly allocate/deallocate memory.

Would have been nice if VS was smart enough to lookup symbols at least in all loaded module exports, but turns out it’s not yet the case, as of VS2010:

Oh well. Maybe some day.

/d1reportAllClassLayout – Dumping Object Memory Layout

Roman just posted a nice investigation he did, mostly using the g++ switch -fdump-class-hierarchy – which dumps to stdout a graphical representation of generated classes layout.

VC++ has no official similar switch, but deep inside  its undocumented pile of goodies lies the /d1reportAllClassLayout compiler switch, which achieves identical results.   It has made its first (AFAIK) public appearance here, and is mentioned in an official MS support reply and sporadically in other places. An individual object layout dump is available too, as -/d1reportSingleClassLayout.

To see it in action, compile the following with /d1reportAllClassLayout:

class A
{
  int m_a, m_b;
  virtual void cookie() {}
  virtual void monster() {}
}

class B : public A
{
  double m_c;
  virtual void cookie() {};
}

And observe the output (excerpt):

class A size(12):
+—
0 | {vfptr}
4 | m_a
8 | m_b
+—
A::$vftable@:
| &A_meta
|  0
0 | &A::cookie
1 | &A::monster
A::cookie this adjustor: 0
A::monster this adjustor: 0
class B size(24):
+—
| +— (base class A)
0 | | {vfptr}
4 | | m_a
8 | | m_b
| +—
| <alignment member> (size=4)
16 | m_c
+—
B::$vftable@:
| &B_meta
|  0
0 | &B::cookie
1 | &A::monster
B::cookie this adjustor: 0

As the VC team blog post demonstrated, this can be an irreplaceable tool in tracking down obj-compatibility problems – which can go uncaught by the compiler. It is also a great learning tool, and since /d1reportAllClassLayout dumps all compiled objects, including CRT internals – this it is also a unique reversing tool, already used for some deep spelunking into VC++ internals.