Stepping Over Unwanted Functions

There’s an undocumented VS setting that lets you specify functions that would not be stepped into when you click F11.  This can be save quite a few clicks in situations of the form -

InterestingFunct(BoringFunc1(), BoringFunc2(),...,BoringFunc99);

Where pressing F11 would repeatedly step you into constructors, accessors, SmartPointer management routines, overloaded operators, you name it.

Now, it used to be that to achieve that you used the [ExecutionControl] section in autoexp.dat, and seems it is widely believed to still hold. However, since 2003(!) the home of these setting is the registry. Since VS2005, it’s the key-

HKLM\Software\Microsoft\VisualStudio\8.0\NativeDE\StepOver

Andy Pennell tells the full story, along with syntax (which I won’t rehash),  examples and some history – all in all, a great read.

Debugging a Process Launch

While this is hardly a hidden setting, I found myself spending too much time to find it twice already – the msdn page describing it appears low on search results, since it uses very different terminology.  So,  if only for self reference, there goes:

You can easily debug executables without source (or even symbols) in VS – just attach to the running process. You can then step through instructions and watch memory and registers as you would in regular debugging.  However, I had some process-launch issues, with a process I didn’t have the source to – so I had no running process to attach to..

Apparently you cannot launch a source-less process from within VS. The way to go is to modify a registry setting that would cause windows to launch a chosen debugger (VS for me) whenever you try and execute your destination process.

The registry key is -

HKLM\Software\Microsoft\Windows NT\currentversion\image file execution options

Add a subkey with the name of the executable to debug (e.g., notepad.exe), and give it a string value with name ‘Debugger’ and either of the values -

(1) ‘vsjitdebugger.exe’ if you want the JIT debugger-selection dialog,

(2) ‘devenv /debugexe’ if you know you’d want a new VS instance,

(3) Other debuggers of your choice – windbg, ntsd, kd, etc.   Personally, I’ve yet to encounter a situation where they’re necessary, but these do exist.

The ‘image file execution options’ key contains some more goodies, that Junfeng Zhang took the time to look up and semi-document.

Edit (1) : Gregg Miskelly elaborates on Junfengs post, then updates to VS2008.  You’ll find tons of other posts googling for IFEO, but I consider these to be the most informed.

Edit (2):  Turns out there’s a much, much easier and less destructive way – vsjitdebugger.exe can accept a process as a command line argument!

It sits at the WINDOWS/system32 folder, and running ‘vsjitdebugger.exe notepad.exe’ will do the trick.

You can create a shortcut to vsjitdebugger with your executable as an argument for extra-convenience.

Type ‘vsjitdebugger.exe -?’ for a bit more stuff (executable arguments and attaching to a running process).

Extending Error Codes – to DirectX and Beyond.

$err,hr is quite useful as is, but it only knows how to interpret built-in error codes. Happily, there is an easy way to extend it, via a section in autoexp.

All it takes is adding lines of the form -

error_code_in_decimal = the_string_you_want_displayed .

For instance, paste the following code into your autoexp.dat to decode Direct3D errors:

[hresult]
;1234=my custom error code
...
; start copying from here:
2289436696=D3DERR_WRONGTEXTUREFORMAT
2289436697=D3DERR_UNSUPPORTEDCOLOROPERATION
2289436698=D3DERR_UNSUPPORTEDCOLORARG
2289436699=D3DERR_UNSUPPORTEDALPHAOPERATION
2289436700=D3DERR_UNSUPPORTEDALPHAARG
2289436701=D3DERR_TOOMANYOPERATIONS
2289436702=D3DERR_CONFLICTINGTEXTUREFILTER
2289436703=D3DERR_UNSUPPORTEDFACTORVALUE
2289436705=D3DERR_CONFLICTINGRENDERSTATE
2289436706=D3DERR_UNSUPPORTEDTEXTUREFILTER
2289436710=D3DERR_CONFLICTINGTEXTUREPALETTE
2289436711=D3DERR_DRIVERINTERNALERROR

2289436774=D3DERR_NOTFOUND
2289436775=D3DERR_MOREDATA
2289436776=D3DERR_DEVICELOST
2289436777=D3DERR_DEVICENOTRESET
2289436778=D3DERR_NOTAVAILABLE
2289435004= D3DERR_OUTOFVIDEOMEMORY
2289436779=D3DERR_INVALIDDEVICE
2289436780=D3DERR_INVALIDCALL
2289436781=D3DERR_DRIVERINVALIDCALL
2289435164=D3DERR_WASSTILLDRAWING
141953135=D3DOK_NOAUTOGEN

2289436784=D3DERR_DEVICEREMOVED
141953141=S_NOT_RESIDENT
141953142=S_RESIDENT_IN_SHARED_MEMORY
141953143=S_PRESENT_MODE_CHANGED
141953144=S_PRESENT_OCCLUDED
2289436788=D3DERR_DEVICEHUNG
;end copy.

These were adapted from macros defined d3d9.h.  Just this week I learnt you could do the same with HRESULT codes of the CLR, and of course you could do the same for your own custom error codes.

Watching Errors in Visual Studio

System and COM routines often set an error code, that can be retrieved by GetLastError.   There are several ways to decipher this code -

(1) Use the command line net helpmsg,

(2) From the VS menu, select Tools/Error Lookup and paste the code,

and by far the most useful:

(3) keep ‘$err,hr’ on your watch window!

‘$err’ is one of several pseudoregisters – a neat feature of Visual Studio that is all but undocumented. (although some nice references can by found online).  ‘hr’ is one of several format specifiers, indicating to VS that the raw hex value is to be interpreted as an HRESULT, or a windows error code.

Now off to a first cute enhancement.

Detecting the Location Where the Error is Raised

When you want to locate the code location that raises the error, your first approach might be to step through it, repeatedly bisecting it untilyou can pin-point the culprit line. Wouldn’t it be useful to be set a breakpoint exactly where the error is set?

You could try and set a breakpoint to SetLastError – and that might actually work for some non-MS COM errors, but its certainly inlined for MS internal components.    There are better ways.

As noted in several places, disassembling GetLastError you get -

  MOV EAX,FS:[18h]
  MOV EAX,DWORD PTR [EAX+34h]
  RET

Turns out FS:[18h] points to the current thread’s environment block, and all GetLastError does is retrieve a DWORD from a fixed offset into that block. So, you could note the memory address of that DWORD, (either by disassembling or using the undocumented NtCurrentTeb) and set a data breakpoint to it, but there’s a better way still.

As Calvin Hsia notes, you can use another pseudoregister, $tib, just for that. The address of the error code is exactly -

@tib+0x34

Note that value and place a data breakpoint there, and you’re done!

This breakpoint typically induces a lot of noise, as many Windows routines set expected errors and then re-set them. Its probably best to selectively enable it only around suspect code regions.

EnumDisplayMonitors Troubles – NvCpl solutions.

Recently I had EnumDisplayMonitors presenting some strange behaviour, exposing 2 monitors when in fact only one was connected.

The nVidia Control panel API eventually came to the rescue – it exposes the entire functionality available via the nVidia control panel, and implemented in NvCpl.dll in the Windows\system32 folder. Specifically NvCplRefreshConnectedDevices(), quote from the manual, ‘refreshes the connection state cache for all display outputs on the selected GPU’.

Never heard of the ‘connection state cache’ – indeed, by the documentation another API (NvCplGetActiveDevicesString), retrieves the ‘connected device
state that was cached by the driver during such system events as bootup, logon, or opening of the display properties control panel’.

Bottom line, the following code solved the problem:

typedef BOOL (APIENTRY *NvRefreshProc)( IN DWORD );

VOID RefreshDisplayCache()
{
   HANDLE hNvCpl = LoadLibrary(_T("NvCpl"));

   // The graphics driver caches monitors setup, and sometimes gets out of sync.
   // Here we force a refresh.
   if (hNvCpl)
   {
      NvRefreshProc ProcRefresh = (NvRefreshProc)
            GetProcAddress(hNvCpl, "NvCplRefreshConnectedDevices");

      if (ProcRefresh)
            (ProcRefresh) (1);  // input flag - NVREFRESH_NONINTRUSIVE
   }
}

As an added bonus, I’m now able to programmatically set ‘fixed aspect-ratia scaling’, which I couldn’t until now.

typedef DWORD (APIENTRY *dtcfgexPROC)( LPSTR );

...

// change display to fixed aspect-ratio scaling, so as not to distort the
	// picture on wide screens.
	if (hNvCpl)
	{
		dtcfgexPROC ProcAdd = (dtcfgexPROC) GetProcAddress(m_hNvCpl, "dtcfgex");

		if (ProcAdd)
			(ProcAdd) ("setscaling 1 5"); // set scaling on display #1 to mode 5 (fixed aspect ratio)
	}

(check out dtcfgex in the manual for details.)

The API contains lots of other low level goodies. If you make extensive use of it, you might prefer to statically link against NvCpl.dll and include the header NvCpl.h (contained in this nVidia sample download).

I have no idea how to achieve both tasks on ATI cards (and whether similar cache or scaling mode even exists there) but we ship only nVidia’s – so no worries there, for now.