CRiSP Weblog

Progress on the dtrace implementation is continuing and my understanding of how dtrace works is improving immensely.

I have been enabling bits of ioctl() dtrace driver to try and get the cmd/dtrace binary to talk to the kernel. The next step is to find something to trace, so i have been following and understanding the 'providers' which are like 'device drivers' for dtrace. Eg fbt.c provides access to the symbol table in the kernel. This is the core mechanism for being able to monitor any part of the kernel.

Linux contains a symbol table (eg /proc/kallsyms).

But, its locked out for access by non-GPL drivers. I am studying what it does to work out a way to tunnel into it (alternatively, the symtab can be fed from user space into the driver -- more work, but may be a better alternative).

Enter the license wars. fbt.c can gain access to the public functions of kallsyms.c but only if it is declared to be a GPL driver. I am using Sun's implementation which is CDDL, so i need to find a way in. I have a way by declaring myself as GPL but that amounts to putting Sun's code under CDDL, which i do not have the right to do.

For the moment, I am being dirty and taking that route so I can experiment with the core technology and avoid too much licensing-legalese. I dont mind tackling that on its own basis but theres lots of bits to get together to make it all work.

There is a way to technically break the hurdle - which would involve yet another driver (up to three so far -- dtrace, fasttrap and fbt. More provider drivers are forthcoming, so adding another one will not be a big issue).

Its interesting to understand the legalese here - even if I am not a politician or a bigot (I think). I just want to solve the problem.

Once we have a symtab, I can work more on the dtrace driver to ensure 'dtrace -l' shows something useful, and then to try a real D script.

Still a long way to go. I am trying hard to avoid any changes to the kernel source - this will simplify deployment and installation.

Better progress now my Ubuntu compiled kernel boots properly. Been doing various tidyups to ensure everything compiles on a clean system. Changes to the include files were breaking user land dtrace and vice versa. Now stable - bit messy and will try and clean up as I understand how I want to drive this.

One way is to split the main header file -- linux_types.h -- into a separate file for kernel side and user side stuff.

dtrace -l now simply segfaults - bad calling convention, so need to fix that and have started the switch over to make cmd/dtrace use the /dev/dtrace driver entrypoint.

Hope to make more progrss today and regularly update the website with a latest snapshot of the driver.

Over the last few weeks, dtrace progress has been coming along. I now have a driver I can load in the kernel and am spending time diligently trying to step through it (initially debugging with printk() statements).

The last week was mostly wasted trying to get a VMware guest with Ubuntu 7 running so when i crash the kernel, I wont lose any work. This was incredibly painful due to the way the udev filesystem works, and the initram disk is setup. I would keep booting a 2.6.24.4 Linux kernel, and have the system not be useful because it couldnt find the hard drive.

Using google, i finally found the update-initramfs script (rather than mkinitramfs) and the kernel is bootable and good for recompiling the driver and testing.

The initial test is simply to load the driver, and do:

cat /dev/dtrace

And check /var/log/messages for the debug printks.

Now this is working, I can get back to user-to-kernel debugging and try and get cmd/dtrace to talk to drivers/dtrace.

You can watch out for daily source code snapshots on http://www.crisp.demon.co.uk/tools.html

If you grab the code, you are on your own for now - until I feel comfortable its ready for prime time.

If anyone wants to contribute - please feel free. There are going to be some fiddly bits later on to resolve (such as kernel hooking and requiring a custom dtrace kernel).

More later.

Yup - now we have something to play with. The plumbing now needs to be set up for open/read/write so we can start the hard graft and the likely kernel smashing which will ensue.

The current release builds on Ubuntu 7 releases (2.6.22) and Linux kernel 2.6.24.

More later

Today marks the day when the kernel driver dtracedrv.ko finally builds. Theres still a lot of work to get the Linux port of dtrace working, but at least (in theory) it can be loaded into a kernel.

Anyone following this progress should be careful to avoid crashing a machine they care about.

the dt_lex.l file now compiles properly and works, so the dtrace binary can now parse scripts ok.

back to the kernel driver trying to reduce compiler errors.

Some progress being made ... a long road ahead.

Trying to get a very simple command line program to parse and am hitting issues with flex vs Sun's lex code. (String parsing is going to the terminal and waiting for input).

This in turn is causing a parser error and dtrace aborts.

Still about 200 kernel driver for dtrace issues to resolve before getting close to a dtrace.o kernel module. Hopefully most of these are the same root cause compile issues.

Will try and blog each day on useful progress.

Tricky!

dtrace consists of some userland components and kernel drivers. I have a 'dtrace' binary - but its a bit of a sham - relying on Solaris headers and drivers, and quickly core dumps if you try and do anything with it. (I have about 50+ stub functions just so I can get to a binary and understand what it wants and how it works).

CTF - what is it? Well, it looks like a mostly optional component. It heavily relies on converting DWARF debug format to something the inkernel modules can deal with. The CTF code is fairly straightforward and relies on lots of solaris headers, and one has to be careful to filter out what is pure solaris and what is linux compliant.

Am going to tackle trying to build the dtrace driver, since this is the core of the whole system.

Linux's build environment is ok - its fine for me, but will present some difficulty when i give out the code.

I have been staring at the Apple dtrace port and the freebsd ones. Apple have done not a bad job - they have stripped out the stuff that is irrelevant, and FreeBSDs code is simply confusing because theres a lot of stuff left in without a clear indication of where to start and is the port complete.

More progress when theres something to shout about at a later date.

dtrace - look it up - one of the most admirable technical tools for machine and performance troubleshooting.

after waiting years for someone to port to linux, I am getting bored waiting.

Sun provide dtrace for Solaris, and its been ported to FreeBSD and MacOSX.

But the linux people ignore it.

So, time to have a go myself. At present progress isnt bad - most of the userland stuff compiles and i need to work on code changes to allow linking of the 'dtrace' binary.

After that - we go tackle the kernel part.

Suns /proc interface is good - very goo. Linux's is more obvious (since its all text files), but not as feature rich as Solaris.

So, lets see how it goes....