CRiSP Weblog

Progress on dtrace is coming in waves now.

I had issues getting to the DTRACEIOC_ENABLE ioctl - after a bit of head scratching, this was resolved, and now we get to DTRACEIOC_GO.

Even better, we get past DTRACEIOC_GO.

What does this mean?

It means the dtrace binary talks to the kernel, can pass any D script in, and then wait for the buffered information to be made available.

I found out again, my dtrace.c kernel code was outdated, so spent some time merging the latest OpenSolaris code in - still some stuff to merge, but lots of useful things in there, such as 128-bit arithmetic for tracking big counters, and some validation checks on the way memory is accessed. (Validating memory doesnt help me, because i default to enabling probing to anywhere - need to fix this at some point so that hackers and accidents can't bring the system down).

Now we are past the DTRACEIOC_GO, i found some issues with userland dtrace - stubs i hadnt coded. Thats partially done (gethrtime() and a partial pthread cond wait function).

Heres an example 'session':

/home/fox/src/dtrace/drivers/dtrace@vmubuntu: dtrace -v -f journal_invalidatepage
dtrace: description 'journal_invalidatepage' matched 6 probes

Stability attributes for description journal_invalidatepage:

        Minimum Probe Description Attributes
                Identifier Names: Unstable
                Data Semantics:   Unstable
                Dependency Class: Common

        Minimum Statement Attributes
                Identifier Names: Stable
                Data Semantics:   Stable
                Dependency Class: Common

Not much to look at - i get no output, even after Ctrl-C. (Plus that arbitrary probe isnt interesting really).

I believe we are now firmly in phase 3 of the task: phase I was to build dtrace cmd + driver. Phase II was to get to a point where we dont crash the kernel and the userland command is functional.

Phase III is the point where the whole thing can actually start reporting something/anything.

Phase IV will be examination of SDT an PID providers so we can do really useful stuff.

Stay tuned.

PS I make releases each night if i feel there is progress or important bug fix.es

http://www.crisp.demon.co.uk/tools.html

Slowly getting there. Last week was near zero progress as I attempt to do some CRiSP catch up work, and track a near impossible bug to find. (CRiSP is valgrind pure - no detectable memory corruptions of significance, yet people are reporting strange bugs. Armed with logs, they dont help - Whoof! Out of nowhere - a GPF/Core dump; oh well).

Anyway, back at the dtrace camp - some progress. The startup code is wrong in the kernel - my driver was missing some of the subtlety of the dtrace_attach()/dtrace_open() code, so by the time cmd/dtrace tries to do an ioctl(DTRACE_ENABLE), we hit some null pointers.

I've now protected myself against this. (Such a kernel panic causes a reboot to be required); I know what i need to debug, just some more linux kernel searching to validate I am calling the correct api (dev_set_drvdata).

Had a near panic last night when my vmware/ubuntu refused to boot. I think ubuntu screwed up the /boot/grub/menu.lst - so i was booting a virgin kernel without an initrd ram disk. Fortunately, one of the many menu items was available, so i have been able to make more progress.

Progress slow last week - had to do some CRiSP work....

Am spending (too much) time investigating why Ubuntu bison/flex combination doesnt work compared to Fedora 8 bison/flex.

Unfortunately, these tools shoot themselves in the foot - they try to be compatible with old yacc/lex, but are just sufficiently different that a trivial issue becomes very difficult to debug.

Have always disliked lex since it provides so little utility, and debugging it - especially when the lex definition 'just works'.

I can see Apple, in the Darwin code, have hit the same issue, but somehow my issue is very subtle.

Oh well.

Once the portability issue is resolved, I can go back to the driver and just move things along, before I forget how this all works.

Look at the output below. Real dtrace, real symbols!

Finally able to access the modules and kallsyms to find /dev/fbt entry points to patch in the kernel. This is definitely a milestone - as now, in theory, dtrace can start patching the kernel to insert probes. I have yet to try this - next on my list, to see what actually happens.

Note that we only seem to have a subset of available kernel probes because -- I dont know! Maybe these are the only modules I am loading, and appear to be missing the kernel syms (maybe I need to modify the fbt driver to not just enumerate every module, but to enumerate every kernel/kallsyms entry).

But this gives a huge blast to move forward and start debugging D scripts.

I have truncated the output below - its showing 515 probe points in the kernel (a stripped down linux 2.6.24-4 kernel).

   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
    4        fbt         dtracedrv                         ctf_close entry
    5        fbt         dtracedrv                         ctf_close return
    6        fbt         dtracedrv                     ctf_func_args entry
    7        fbt         dtracedrv                     ctf_func_args return
    8        fbt        freq_table    cpufreq_frequency_table_target entry
    9        fbt        freq_table    cpufreq_frequency_table_target return
   10        fbt              dock      register_hotplug_dock_device entry
   11        fbt              dock      register_hotplug_dock_device return
   12        fbt              dock    unregister_hotplug_dock_device entry
   13        fbt              dock    unregister_hotplug_dock_device return
   14        fbt        parport_pc        parport_pc_unregister_port entry
   15        fbt        parport_pc             parport_pc_probe_port entry
   16        fbt        parport_pc             parport_pc_probe_port return
   17        fbt           parport    parport_ieee1284_ecp_read_data entry
   18        fbt           parport    parport_ieee1284_ecp_read_data return
   19        fbt           parport    parport_ieee1284_epp_read_data entry
   20        fbt           parport    parport_ieee1284_epp_read_data return
   21        fbt           parport      parport_ieee1284_read_nibble entry
   22        fbt           parport      parport_ieee1284_read_nibble return
   23        fbt           parport        parport_ieee1284_read_byte entry
   24        fbt           parport        parport_ieee1284_read_byte return
   25        fbt           parport                parport_wait_event entry
   26        fbt           parport                parport_wait_event return
   27        fbt           parport           parport_register_driver entry
   28        fbt           parport           parport_register_driver return
   29        fbt           parport    parport_ieee1284_epp_read_addr entry
   30        fbt           parport    parport_ieee1284_epp_read_addr return
   31        fbt           parport                   parport_release entry
   32        fbt           parport             parport_announce_port entry
   33        fbt           parport         parport_unregister_device entry
   34        fbt           parport     parport_ieee1284_write_compat entry
   35        fbt           parport     parport_ieee1284_write_compat return
   36        fbt           parport   parport_ieee1284_epp_write_data entry
   37        fbt           parport         parport_unregister_driver entry
   38        fbt           parport                  parport_put_port entry
   39        fbt           parport                  parport_put_port return
   40        fbt           parport   parport_ieee1284_epp_write_addr entry
   41        fbt           parport               parport_remove_port entry
   42        fbt           parport               parport_remove_port return
   43        fbt           parport             parport_register_port entry
   44        fbt           parport             parport_register_port return
   45        fbt           parport   parport_ieee1284_ecp_write_addr entry
   46        fbt           parport   parport_ieee1284_ecp_write_addr return
   47        fbt           parport   parport_ieee1284_ecp_write_data entry
   48        fbt           parport   parport_ieee1284_ecp_write_data return
   49        fbt          i2c_core                    i2c_new_device entry
   50        fbt          i2c_core                    i2c_new_device return
   51        fbt          i2c_core                         i2c_probe entry
   52        fbt          i2c_core                         i2c_probe return
   53        fbt          i2c_core          i2c_add_numbered_adapter entry
   54        fbt          i2c_core          i2c_add_numbered_adapter return
...

$ dtrace -l
   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR

Hooray! We went thru a ton of code in the kernel to dig that out! Now -- to find out what happened to my /dev/fbt entries...

Another bank holiday over...dtrace one step closer...

I have decided to consolidate the four drivers into a single dtracedrv.ko driver, to avoid lots of fluff with inter-driver symbol resolution. Having separate drivers causes issues at link-time and leads to a hairy order of dependency as the modules are loaded.

Now we have a single dtracedrv.ko.

Any why is is dtracedrv.ko and not dtrace.ko ?!

Because i havent finished getting the makefiles to work. There is a file called dtrace.c which has most of the kernel guts in it - but not all of it, and the linux kbuild software gets confused if i want my driver to be called by the same name as a dependent source file.

I have also added changed-line support to CRiSP whilst I am at it. Someone had asked for the ability to add a comment in column 73 of a line that is modified (presumably COBOL or Fortran code), and this nearly works. Just need to add the setup menus.

Now...off to get 'dtrace -l' to give me something to probe !

This week has seen good progress on dtrace for Linux. Much of the week has seen the /dev/fbt driver move to a point of full functionality.

/dev/fbt is used to allow monitoring around all functions in the kernel, by establishing probes on the entry and exit from a function. This, in theory, gives rise to thousands of probes. (/dev/sdt is needed for high level actions like process fork/death, etc - and will come later).

The GPL/CDDL issue in fbt is resolved by using a helper from user land to allow investigation of the running kernel.

The main thing to get to now is to see if the full dtrace userland path can be executed - eg, for 'dtrace -l' to work to see some probes, and then try some simple D scripts to see if the right things happen.

The pid provider will be needed to get access to current process properties, and hopefully wont be a big deal.

To date, the only kernel expectation is that we have reasonable compile defaults (eg modules, kallsyms, etc) and we havent had to touch or break the kernel.

dtrace wont compile without access to the running kernel sources, but its beginning to look good.