CRiSP Weblog

I wrote in my prior blog about adding support for SDT probes in dynamically loaded kernel modules - aimed at people writing their own drivers, rather than instrumenting the Linux kernel itself. This experiment is just about complete.

DTrace includes a demonstration driver - "/proc/hello-world" (located in the driver-2 directory). Its a standalone driver which simply adds SDT probes in the open() and read() code.

DTrace itself is modified to detect these probes as the module is loaded.

Heres an example:

/home/fox/src/dtrace@vmarch311-64: build/dtrace -l -P hworld
   ID   PROVIDER            MODULE                          FUNCTION NAME
227663     hworld       open_module                             open1 entry
227664     hworld       open_module                             open2 entry
227667     hworld       read_module                              read entry
/home/fox/src/dtrace@vmarch311-64: build/dtrace -n hworld:::
dtrace: description 'hworld:::' matched 3 probes
...

"hworld" is the provider name provided in the hello world driver, e.g.

...
 DTRACE_PROBE1(hworld, open_module, open1, entry, num_opens);
...

At the moment, the #define for this macro is in the hworld.c driver, but I will move the definition out of the driver. The macro is very ugly, but thats really because C string concatentation in macros is ugly:

#define DTRACE_PROBE1(provider, module, name, func, arg1) \
	{extern void __dtrace_##provider##___##module##___##name##___##func(unsigned long);	\
	__dtrace_##provider##___##module##___##name##___##func ((unsigned long)arg1);		\
	asm(".pushsection .dtrace_section, \"ax\"\n"); 			\
	asm(".global __dtrace_" #provider "___" #module "___" #name "___" #func "\n"); \
	asm(".type __dtrace_" #provider "___" #module "___" #name "___" #func ", @function\n"); \
	asm("__dtrace_" #provider "___" #module "___" #name "___" #func ": ret\n"); \
	asm(" int3 ; int3 ; int3 ; int3 \n"); \
	asm(".popsection\n"); \
	}

The SDT provider in dtrace is one of the simplest and many people want to create their own. This is ideal for instrumenting the kernel source or your own dynamic driver.

My approach to supporting Solaris-like SDT probes involves some (too) clever technology, but I missed the point. Nice as it is, to augment the kernel with probes, that is one problem. Instrumenting ones own drivers (such as ZFS) is something people have been asking about.

Easy! Isnt it?!

Err...well, no. Since starting the dtrace port, I skimped over this because we are not touching the kernel source tree. But lets say we want to touch our own tree. Or, more likely, our own driver, what does this involve?

Well, again, its easy! Just sprinkle your code with DTRACE_PROBE macros.

Errr....no. Hang on.

What happens next?

Nothing.

Ok, lets examine a solution. Lets say that DTRACE_PROBE is suitably defined. The requirements for this are we insert a call instruction at the probe point - just like USDT. But this call instruction has nowhere to go. With USDT, we link with drti.o and it arranges to scan all probe calls, and make them no-ops, until you ask for the probe to be tracked. At which point dtrace will replace the nop with an INT3 instruction and the kernel code will map that breakpoint to the USDT probe firing.

In the kernel, its different. We dont have drti.o. But we know a module is loading, so during the module scan, we can look for these "strange" call instructions, stub them out (actually, leave them in). We *dynamically* create an SDT probe as if it had been predeclared and the job is done.

This is what I am currently trialling. I have created a simple "Hello world" (/proc/helloworld) driver, with DTRACE_PROBE calls, and the goal is that when the driver is loaded, we can trace these probes, thus demoing a working facility.

This is strong enough for dynamically loaded modules and in theory is also the correct solution for a vmlinux which has DTRACE_PROBEs in there.

Stay tuned for some results.