CRiSP Weblog

I use CRiSP to read/compose mail..mostly. I also use gmail and my original crisp mail account.

The beauty of CRiSPs mailer is that it is pure text mode, and woefully unpowerful. This means that viruses are non-existant. I can see them (as mime encoded data, or links to bogus websites), but they cannot hurt me.

Most mailers these days utilise the browser plugin to render the page, and hence, reading mail means you are exposed to the same bugs or flaws that the browser can be impacted by. (Every image, video or javascript in an email is being interpreted by a piece of code, e.g. ZLib, or PNG, or GIF, and these libraries, alas, are the cause of many problems in Windows, Linux or MacOSX). Its an arms-race to ensure core code libraries are bug free.

Yet, for a long while now, attachments sent from CRiSP didnt work. They use to; but they stopped working. I didnt know why - nor bother to look.

Now I have looked, and it wasnt CRiSPs fault. It is the complexity of actually sending mail (via SMTP) in todays world. My original ISP forbids emails from outside its own domain, in an attempt to stop spam. I had to resort to emailing from their webmail page, which is a nuisance and an overhead - mechanising the webmail page to send a message. And I had forgotten about attachments. Had I remembered, I might have added support, but gave up.

So I use GMail. Utilising some bits of perl glue, I can map the output of CRiSP email to gmail, and post. In many emails to myself over the last day, I finally managed to get the mail headers correct to send attachments. One of those "I must solve it" problems, is, now solved.

My next project in this area is to figure out how to write an android app to mimic what CRiSP does (read raw text email, rather than be clever to render). Theres just too much spam/rubbish in the world to worry about the sanctity of my portable device(s).

Now...back to dtrace.

I like this phone, despite the very poor battery life. You can see the specs on the net (4" 840x400 screen, 1GHz cpu, etc).

Battery life is poor - the phone cannot last 12h in normal use, i.e. turned on, handful of web browsing sessions lasting less than a few mins each. The microusb is very annoying, meaning I need to be armed with an adaptor or cable rather than share any other cable within easy reach.

The main Android theme is nice, but takes getting used to.

The video player has some problems (stuttering after turning on the phone, maybe background stuff is stealing too many cpu cycles).

Email has taken a while to get the hang of.

Certain apps are needed and should be builtin: a decent task monitor, a 3G bandwidth monitor, and a few other things. One needs to ensure apps are not running in the background stealing very valuable, but limited bandwidth.

The GMail client is relatively poor - it mostly works, but not having been an gmail heavy user, I found aspects confusing. (Especially as my laptop was stealing some of the emails and I kept losing things). Attachments work badly - you can read them, but you cannot attach them from the filesystem. (Workaround: use the web browser and access gmail.com to do this kind of thing).

Theres no app to turn the phone on and off at a set time. I dont understand why that is not there. Leaving the phone switched on overnight will kill the battery.

One nice thing is an app which can turn off wifi/internet so that it is not eating so much battery power (juicedefender). But it does mean that the internet is not available unless/until you turn on the phone, delaying the ability for quick on/browse mode.

The phone app needs a bit more thinking: it is far to easy to start phoning someone by accident - the edge-to-edge screen means your hands/fingers have to grasp carefully to avoid touching parts of the screen, but the result can be "random button pushing". It would be nice if the phone app could force a prompt before calling rather than doing it.

The multitude of twitter/facebook and other social media apps is confusing. I can never be sure whether they are sharing the same data or each one is re-grabbing the same data as you switch among them.

It does feel more "powerful" than an ipod/ipad/iphone, but then my focus for the device is different. Video and battery life is reasonable, but with no standard music player or video player, one can end up trying lots of Android Market apps to find a nice one which fits your requirements.

The core apps on Android need a bit more power (customisation), such that people do not implement a 100 different versions of the same basic apps.

Android needs a Virus monitoring tool. Its too easy to have apps using the bandwidth when you dont want them too. When on a frugal bandwidth budget, having apps all contacting home (for adverts, or updates, or just to say "hi!"), can be worrying.

Given that the core is just "linux", I may have to start looking to see how to rationalise the many system apps to "do what I want".

A project for later in the year, maybe.

I have some issue with dtrace on AS4 - Ctrl-c dtrace and it can crash the kernel. Doesnt happen on my Centos 4.7, so, something to look into on a rainy day.

Rather than drive myself insane on that issue, I am looking at how "symbols" work in dtrace. The files installed in /usr/lib/dtrace provide an "interface" so that D scripts can use symbols and structures in the kernel.

When I tried a simple probe like this:

$ dtrace -n 'syscall:::{printf("%d", cpu);}"

I started to realise I didnt fully understand what is going on here - for MacOSX, Solaris, or Linux. Certainly, each platform provides a CTF file which is a mapping of symbols in the kernel such that dtrace can use them. But what/where is "cpu"?

Its a strange question, because you get the CPU for free when doing a dtrace probe. But it isnt a standard variable like "pid", or "execname" (which work in Linux/dtrace). "cpu" needs to be constructed from a cpu info structure - this representing the actual cpu we are running on at the time of the probe firing.

I have the start of the CTF framework in dtrace present, but even though I have created this code, I am not happy I understand it enough to just resolve the issue.

For example, every symbol in /proc/kallsyms is available to us in D scripts, but kallsyms gives us symbols and addresses - not sizes or typedefs. So, now I need to work out what I am missing. If we can expose these symbol types into the CTF file, then we can start grabbing the data, without having to hand craft "constructors" for variables in the kernel.

Of course, a linux kernel compiled with debug info has all this data available, and we might use that. (Except I compile my kernels without debug).

I was surprised how "confusing" this area is, until one uses it enough to really get all the pieces together.

Hopefully will get something going in the next few days/weeks.

Put out an updated dtrace to fix the 32bit syscalls on a 64bit kernel, via the module part of a probe name. We set this to "x32" or "x64". By default, you would wildcard this field:

$ dtrace -n syscall::open:

but now you can elect to monitor, e.g. just 32-bit syscalls:

$ dtrace -n syscall:x32::

Took me a while to figure out how to determine if we are on a 32-bit syscall or 64-bit one. Technically, its not the syscall which is 32 or 64 bit, but the process (actually, current thread). I had a silly in the code so that no 32-bit processes were being trapped.

In examining the kernel code, theres various things that could be done by a process which might confuse the kernel. Most of the kernel makes no distinction between a 32-bit app and a 64-bit one. Things like the ELF magic part of the initial a.out file can be overwritten, without transforming a process from one more to another. The later kernels cache a bit to indicate which mode its in, but its rarely consulted.

There is a personality field in the thread structure, so it would be definitely interesting to change this to try and flip from 32 to 64 bit modes.

What is this? How does one tell we have a 32-bit process on a 64-bit processor? Its my current project - there must be a way, but Linux does a good job of hiding this from you, and the kernel.

The Linux kernel uses a layer of abstraction to allow multiple different exec file formats (or "exec_domains"). So, there are very few places in the kernel which need this distinction: a 32-bit process is a "64-bit" one, but just doesnt bother to address its address space above 4GB. (And such code typically wont utilise the extended registers or addressing modes).

So, how does one tell? How does one care?

Well, one area is a core file: the kernel knows how to write an ELF32 core file vs an ELF64 one, but this is done through the abstraction layer.

I thought, but cannot find, a reference to a bit in one of the cpu registers (CR0, CR1, etc) which tells you which mode we are in, and therefore guess I am looking in the wrong place, and/or even the cpu doesnt have a notion. (Well, it *must* do because some opcodes were remapped from 32-bit to 64-bit mode, so something exists).

At the moment, telling the two apart in a system call trap is proving elusive. (A 32-bit task will execute a different syscall entry into the kernel compared to a 64-bit app, but this may just be semantics, e.g. a 64-bit app may be able to invoke a 32-bit system call, and vice versa).

Something is going to answer me (hello Google!)

A long time ago, an email reading client was added to CRiSP. Its not a primary function, and nobody uses it. It was added as a personal need to avoid some of the problems of all mailers out there, specifically need to handle Unix mbox format files, private archiving (I have all emails from customers, dating back to around 1991 or earlier).

As a body of text, its useful for "datamining", e.g. I used it for writing a full text searching engine, and hit many problems (much of my mailbox, like yours, is full of spam), but it has allowed me to understand the complexity and issues with such research areas.

But the mail client lives on. All my mail is done through CRiSP - never through the many email tools or web portals (such as gmail, or hotmail).

The big companies understand the importance of having your data on their servers, but that is not my consideration.

The most important factor is that: CRiSP is immune to viruses! Yes, someone could bother to reverse engineer CRiSP and look for implementation flaws and allow a virus through, but CRiSPs email explicitly does "nothing clever".

[I am not trying to "sell" you CRiSPs email client - it is not pretty, but its sufficient, for me].

But its worth noting that none of the original dumb, text-mode email clients ever could suffer from viruses, because of this issue.

As mail clients have evolved, and web browsers are used, the "sum of parts" issue kicks in - that unexpected side effects of things like mime handling, attachments, javascript, and other mean that getting unchecked code through your machine is viable.

Of course, most emails these days are from exactly those applications I normally avoid, and with the myriad of encoding methodologies, I have to manually save/extract/decode the files.

[I have my own "spam" mailbox pruning tool].

Today I have been enhancing a part of CRiSP to better handle mime/base64 encoded emails. The functionality is mostly hidden/undocumented (there is an encrypt_buffer() general purpose primitive, which allows for various data conversions.

Eventually I may decide how to expose this to the end user (not as a mail client, but as a generic file-loading/parsing facilty).

Anyway, now you know what occupies a part of my time.

[I need to fix my email attachment mechanism].

Reference: http://www.crisp.com

CRiSP restores the last session when you restart the editor. Sometimes this is desirable, sometimes it is not. A browser, like Firefox, either restores the last session, or, via a plugin, lets you select which prior session to restore.

The way session management is done in a browser is very different from that in an editor. Your "desktop session" is a more a factor of your location (eg your PC, laptop, mobile, home, office), rather than a factor of the project you are working on. I like my browsers at home and office to remember the last state, and these are different from each other.

My editor state is a function of the task I am doing at this present moment, with frequent flips from one thing to another. (I will often be running 10 or more crisps in different windows to manage that state).

This is a problem, because having a startup dialog ask you which session to restore from is a distraction. CRiSP utilises the notion of state-per-working directory. Depending which directory you launch CRiSP from, means it will restore the state based on what you last did in that session.

Most of the time, this is exactly the right thing to do.

However, launching an editor direct from a desktop shortcut presents a problem, because when launched in this way, the starting directory or context is always the same. Maybe CRiSP should ask when launched from a desktop icon.

Additionally, you can launch CRiSP from an icon in a directory listing. What should happen here? Use the default desktop session state, or use a state local to the folder that the file you just clicked in is located.

CRiSP provides access to prestored session states (often called 'projects' and 'rooms').

Reference: http://www.crisp.com

My previous posting showed that 32b syscalls on 64b kernels was working. Alas, a little too premature. I have to resolve an aliasing problem where the same code is used for both 32b and 64b processes, yet my code isnt distinguishing the context we are called in.

This means that

$ dtrace -n systrace:x32::

doesnt work as I expect, but worse, we may misrepresent the syscall for a 64b app, since the syscall numbers dont align between the 32 and 64 bit callback table.

The latest release of dtrace should fix two major annoyances: module probes, and 32-bit system calls on 64-bit kernels.

Module Probes

Due to sillyness on my behalf, the code for enumerating modules on a kernel was broken, so we had no FBT probes against driver modules. In fact, the INSTR provider and FBT were clashing such that FBT always lost out (hash table lookup didnt handle the same address used by each provider). This now works.

Stumbled upon this when trying to figure out why there were no floppy or ext3 FBT probes.

32-bit syscalls on 64-bit kernel

Another thing I had not realised (or I did, but totally forgot), was that we didnt see 32-bit apps executing syscalls, since a 64-bit kernel has two system call tables. Alas, the code to handle this is slightly ugly (nearly doubling up some key functions), but this is now resolved. I have taken the liberty of using the "module" part of a probe to distinguish the probes, so that the user experience is maintained, but the user can now elect to trace one or the other or both. Heres an example:

/home/fox/src/dtrace@vmub10-64: dtrace -n syscall::: -l | head
   ID   PROVIDER            MODULE                          FUNCTION NAME
268731    syscall               x64                              read entry
268732    syscall               x64                              read return
268733    syscall               x64                             write entry
268734    syscall               x64                             write return
268735    syscall               x64                              open entry
268736    syscall               x64                              open return
268737    syscall               x64                             close entry
268738    syscall               x64                             close return
268739    syscall               x64                              stat entry
/home/fox/src/dtrace@vmub10-64: dtrace -n syscall::: -l | grep -v x64 | head
   ID   PROVIDER            MODULE                          FUNCTION NAME
269329    syscall               x32                   restart_syscall entry
269330    syscall               x32                   restart_syscall return
269331    syscall               x32                              exit entry
269332    syscall               x32                              exit return
269333    syscall               x32                              fork entry
269334    syscall               x32                              fork return
269335    syscall               x32                              read entry
269336    syscall               x32                              read return
269337    syscall               x32                             write entry

This is actually a nice solution to a problem which dtrace didnt let you do before, e.g. watch for 32-bit apps running on the system only.

I still have more annoyances to fix (stack() address lookups give some records without symbolic names, ability to run from the install tree whilst being able to access the things defined in the /usr/lib/dtrace include tree, and more stuff on basic data access for cpuinfo and other common subsystems).

Have fun.

I keep toying with the idea behind adding some form of social networking support to CRiSP. http://www.crisp.com.

CRiSP has network connectivity support, it supports an RSS feeder, showing news items in the contents sidebar. It can be useful, although with the advent of more mobile connectivity and heavyweight web browsers, it may not be as useful as it once was. Certainly, if you are coding, or have a private feed of RSS updates, then it can be useful.

Social networking is similar - you may be having a heavy day of coding but want to be informed of updates from twitter, or facebook, or whatever. Being able to show the updates periodically may be useful.

One has to be careful in supporting these types of feeds due to the ever changing APIs and layers of complexity they add (SSL authentication is a good thing, but the requirements to be able to use all the differing encryption types means it is a problem ensuring all the dependencies are available).

Been a while since I spoke about CRiSP ( http://www.crisp.com ). CRiSP has had a long evolutionary life - a very powerful and productive editor, which provides many features. Rather than talk about what it does, lets start a series talking about what it doesn't do, or maybe what it should do.

Todays topic will be line wrapping and word wrapping.

Line wrapping is a useful feature. When CRiSP was implemented, the goal was horizontal scrolling. When CRiSP was born, a 20MB hard drive was considered large; floppies were the media of choice for backup (or QIC tape). Most files were small, and saving space, using tabs instead of spaces was important.

When you control the editor and you control the files - everything is warm and cozy.

But when you start viewing other peoples files or auto generated files, you dont have so much control. If lines are no more than a few screen lines in width, line wrap is important.

[We all take it for granted, in word processors, and browsers, that text will flow automatically as you resize these days].

The computational effort to dynamically resize is moderate. [Browsers have a huge and horrible task to determine layout - its a recursive and iterative process to layout the items on a web page, and web page authors do what it takes to make an effect, with no concern for the CPU power needed to achieve that effect - hence the arms race for cpu power across computers, laptops and even mobile phones].

Word wrap - again, a technique we all take for granted - is more difficult. Conceptually, its easy: just lay out words whilst they fit onto a line and then start a new line when we run out of room.

The problem becomes that we now need to know how we layed out the words for those times as the user clicks on words or parts of words.

An additional difficulty is handling very long lines - lines longer than the screen can display, whilst handling infinitely long files. Once we start doing this, problems arise, like in the browser, where we have to be careful that we dont jump around the screen and we can map mouse clicks to the thing that was clicked on.

If we go one level further, then we hit proportional width fonts. But we will save that one for another day.

[Have you ever loaded a "large" file or web page into the browser and wondered why it can be slow, or why the memory consumption can go through the roof? Thats because browsers dont optimise for "infinite", but for reality. An editor is different.]

If you want to find out more about CRiSP, visit http://www.crisp.com.