CRiSP Weblog

I have managed to do more work on dtrace, but much of it has been catch-up. With a new kernel - 2.6.28 - appearing over Xmas, now was the time to upgrade my laptop (now it has a shiny new HD and Ubuntu with KDE).

This showed up some issues, where symbols had gone private and inaccessible (or maybe this was due to cleanups in the compiler toolkit). My laptop is only 32bit, so, it concerns me that my 64-bit VMware Ubuntu wasnt showing the same issues.

This simply goes to show that even staying still is hard work - every time a new kernel comes out, working on the compilation issues (eg missing prototypes), or linker (undefined symbols) is a nuisance. With new kernel releases every 3 months or so (more frequently for RC releases), its a fair amount of work just playing catchup.

This goes to show that having dependencies in software, is a nuisance, and not a positive benefit. The only way around this is integration into the mainline kernel (I can't see that ever happening, because it would spoil peoples hacking fun, rather than a legal impossibility).

I've fixed a number of build issues, especially on 32-bit, and now the USDT simple examples build correctly.

I need to go back to the /proc/pid/ctl driver interface so we can see why dtracing a USDT executable doesnt work.

This week was frustrating. I had ordered a 500GB laptop drive as I was crunched for space on my main laptop. I decided to bite the bullet and install a new distro (from SimplyMEPIS, of about two+ years ago), to Ubuntu 8.10

All was brilliant: mirrored the data to the new drive via a fast USB caddy (50GB in under an hour). Swapped drives over, booted Ubuntu 8.10 32-bit, installed and it just worked.

My screen was recognized, wifi, sound worked.

But I dont like GNOME, the menus are bizarre (that maybe inheritance from my old $HOME settings).

I tried KDE4 - pretty bug too power hungry and the huge fonts drove me mad. And it was downhill from here.

WIFI wouldnt work from KDE but would from GNOME. 2 days to fix that. So called 'improvements' mean you need a Ph.D in Linux hackery (thank heaven for Google and Ubuntu forums!).

Eventually its all working to my taste (KDE3 - not too intrusive), but the fonts are either ugly or huge, and a waste of space.

I need to get back to dtrace because I was making good progress (exaggeration!) on the /proc/pid/ctl driver interface, and I was worried I might need to do regression compiling against all recent Linux kernels (hence, why the disk upgrade is a good idea).

I hope the Linux Kernel developers are laughing at me now !

I just found that the /proc/pid driver has no 'write()' interface; you can read, but you cannot write to any of the entries in /proc/pid.

So nearly close. Looking at the code in fs/proc/base.c, we can see the structure which is missing the ".write" field, so we could patch it and handle the writes, but I suspect this could be a nasty thing to patch and rely on more and more kernel implementation aspects which may change in the future.

This means I need a better way to do process control -- ptrace() is not very good, it doesn't handle threads properly (which then complicates user space).

Now I am working on a 2.6.27 kernel, I could examine the utrace() kernel interface.

fs/proc isnt dead yet, but I will need to think about what I am actually trying to do.

Some minor alignment changes to incorporate OpenSolaris 20081211 updates. Nothing much to see.

I found, the other day, that I was spending a lot of time trying to get:

task->ptrace

to compile in ctl.c. Its there in my 2.6.24.4 source tree, but on my Fedora FC8, its plainly not there. I havent got the FC8 kernel source, but its not in the headers.

So, looks like RH have pulled their usual stunt of updating kernel and causing me confusion with cross-version updates.

My kernels are looking stale, 6 months on, and I need to update to the later release, to avoid slugging it out with code which has already notably mutated.

I have a number of machines I test or compile on: 32-bit laptop, 64-bit server, VMware (Ubuntu 7 64bit), which is good from a diversity point of view, but bad because I may be tackling issues which are no longer relevant on later platforms.

C'est la vie when you are trying to fight against the GPL.

By the way, to prove I have nothing against the GPL, I did release blog.pl as GPL. Its only 500+ lines of perl, and if people can improve it, great. There are much better bloggers out there, but this fits me real cosy. (Next step, is to redo the whole crisp website with this, vs the current "C" code site generator which is so old, that its no where near as good as the Perl stuff).

Back to ctl.c....

Progress has been slow on dtrace, as it always will be, but I thought it worth saying that this is a community effort.

Someone at google.com is playing with dtrace and addressing issues I havent had time for (newer kernel support), and this is really great. (I wont mention by name, since he may want to stay in the background, but I will mention him if he is happy for this to be known). (I am attributing the changes in the Changes file, so he is there).

The good thing is that we get two sets of eyeballs looking at what I am doing, finding issues where I am blind, or looking at things which are further down my priority list.

Also, this is good because the few patches so far are making me think how to manage a group effort.

He is looking at using kgdb to help debug dtrace; I am looking at the /proc/pid/ctl driver to figure out how to map procfs(4) primitives in libdtrace/libproc into the Linux kernel; this is a rough road - the Linux ptrace() interface is my guide book, but its a delicate area of race conditions and hidden issues which can only show up on a heavily loaded active box.

So, don't think of this project as *my* project, but *our* project, and I will try and co-ordinate and be insightful, whilst accepting donations of code or suggestions etc.

Already one person has asked about having a dtrace backend to generate SystemTap or Dtrace stuff, but I cannot really comment on the feasibility of this. (Sounds like an interesting subproject, and viable).

What does worry me is the ifdef-itis creeping into the code as new kernels come out; the Linux kernel owes us nothing, and can change the code and data structures to their hearts content. This means we have to worry that dtrace may break on each release, and the interesting areas will be how to protect ourelves against the unknown.

For example, the ctl.c code needs (does it?) to patch the kernel, and looks for ASM fingerprints to find what it needs. This needs to be refactored into a userland build-time tool, which can then generate nice messages like "This kernel is unsupported because ...", rather than a bizarre entry in /var/log/messages.

Made a small change to allow random image placements in the main index page when a new blog entry is added.

I am not a photographer, but they are my pictures, and I wanted to spruce up the blog a little.

I am still thinking about adding a backend to allow comments or voting, but this is side tracking me from dtrace....

Having been annoyed with the many files created by nanoblogger, I decided to write my own, copying ideas from nanoblogger, but in Perl.

This means it is fast - instantaneously fast. Its not quite as full featured as nanoblogger, but it suits my exact use cases.

Its available here, under the GPL.

http://www.crisp.demon.co.uk/tools.html

Commenting isnt supported as yet, but I may get around to providing a tunnel for commenting (without acting as a SPAM-bait).

Sometimes, things drive me nuts. I use nanoblogger for the software, and after a few years, I decided to upgrade. (Someone wants to contribute to blog postings, so I may as well get myself upto date).

But, I am making such a big jump that my old articles are invalid, so I had to do a lot of digging and write a convertor.

In the end, it might be more functional - I am not sure.

What I do know is that building the site is slower, and very much worse, blog articles are stored as a file (good), but all files stored in there own YYYY/MM/DD dir (very very bad).

So, now I waste more disk space on this, its lots of wasted dirs and now updating the ftp site is made very slow as my ftp sync script has to trawl/validate the local + remote directories.

Added to which, its all written in bash. That would be a good thing if the lots of dirs issue above hadnt just made it unmanagable.

Looks like I am going to have to nuke/rewrite it.

I do try hard to get on with other peoples software, but sometimes, its things like this that make me wish I hadnt relied on it.

(Added to which, my limited global ftp space is limited, so its wasted that too).

Just found out my email outgoing server (homebrew) wasnt running. Been trying to figure out why outgoing mail didnt work for a while, and forgot to check the obvious.

Why homebrew? Because, like everyone else, I dont want to be a spamtrap. I gave up on sendmail years ago due to complexity and the desire to stop reading the source code to figure out what all the magic $x incantations are.

Now, I need to create a daemon to check my mail is going and mail me !

So, to Adam, Paul and the other Dtrace contributors, apologies. But now, my month old outgoing mail has gone :-)

The /proc/$$/ctl driver interface is now visible.

I have spent a lot of time over last few weeks plotting how to do this, and now we have something that works.

We have to patch a running kernel, which is a nuisance, as it creates more kernel dependencies that can break in future kernels, along with 64 vs 32 bit kernels (dont run the current release on a 32b kernel).

The code is in drivers/dtrace/ctl.c, and is quite ugly. It needs to be generated from a Perl script rather than hand crafted. (Even better: auto generation from the running kernel).

We may need to patch the kernel more to handle SDT type probes and tight interception of process creation/destruction, but at least I have something to play with now.

Of course, the whole lot may be pointless.

Someone from Sun (Adam) suggested looking at the Apple dtrace code, but that relies heavily on the MACH kernel interface for process control, and not the classic ptrace() syscall or anything resembling SVR4. So, thats out. FreeBSD has its own BSD derived and enhanced ptrace() layer, so we cannot use that as a guide either.