I've started reading the first volume of Christopher Alexander's four-volume work The Nature Of Order. It has apparently taken him twenty-seven years to research and write, and it looks to contain some fairly heavy ideas. As such, it'll be a challenge to make it right through to the other end of book four, but I believe that there could be some profoundly significant ideas in it that could have a great impact on what I could do with Ngake.

If he can deliver on his promise to provide "a theory in which statements about relative degree of harmony, or life, or wholeness - basic aspects of order - are understood as potentially true or false," and generally rescue architecture from its apparently dire state, then this should be very useful in designing information architectures that people will find effective and satisfying to work with. Or so I imagine.

Anyway, it occurred to me that if I kept notes in my weblog about my reactions to the books, things that I found particularly useful or startling or thought-provoking, and ideas about how I might be able to apply what I'm learning, then this might help me persevere right through the series where otherwise I'd run out of steam.

So we'll see how we go.

[t-n-o-o] # .

I'm tweaking the CCL parse-c-decls.lisp (yes, I'm still working on the CCL-Maverik FFI) and I found this paper by Kurt Nørmark about Lisp hooks.

Also, I see that Luke Crook has written a HOWTO for the CCL FFI.

[Update: I've implemented my own version of the Lisp hooks, based on Kurt's paper.]

[lisp] # .

OK. On the 17th, we switch to a new Paradise.net plan, and our monthly bandwidth quota goes from 500MB to 1536MB (or possibly 1536MiB). But apparently there's no way to avoid getting stung for the extra 600MB or so of ICMP traffic that the Welchia worm generated for us.

I'm moving synchromesh.co.nz and micro.co.nz to Free Parking, 'cause they're cheaper and they have online DIY DNS management.

Here are a few links I'll come back and check out next week (I've started reading BoingBoing again):

• A Taiwanese shirt folding video
• Some groovy Shockwave art
• A demo of a new 3D desktop in Java
• A fan-assembled trailer for the movie The Hobbit

I did "splurge" on some NASA images to use as wallpaper on my laptop:

• The space shuttle over Cook Strait
• Mir over New Zealand
• And there's more at NASA's Visible Earth: New Zealand, including this 11.6MB monster, described here.

[random] # .

As the latest chunk of Tranz Rail work nears completion, I am of course distracted by other things. Such as the next phase of my AP5HID library.

The story so far:

I did a little dynamic programming to figure out an approximate solution to the positions of the LEDs. I have made a little VRML model of my results, and it definitely looks a lot like my P5 glove, which is good.

I checked the Essential Reality discussion fora, and it seems that ER have gone into survival mode, which is not entirely surprising I suppose, but it does imply that we won't be getting any driver updates for a while.

However, I did find a few quoted IRC comments attributed to Jason McMullan, author of the open-source Linux driver the C source of which I used to decode the USB datastream. Some choice quotes:

I'm working on a "best fit plane" technique to derive tilt from the IR data points.

Interestingly, only 4 leds are returned by the device at a time. The other leds are interpolated.

Everytime you poll the P5 device for info, the mouse device sends a mouse position packet. ... Actually, that the *only* time it sends mouse data.

Interestingly, if you poll the P5 "too fast", it resets the device.

"[I] draw the current IR positions, and the ***puted [sic] surface normal of the hand (which is, currently, often wrong).

So, I plugged best fit plane into Google and found this, which smells like the right direction to me. Verily, the mighty Google has been a tremendous boon to the solo R&D-er.

[p5] # .

Last night at about 2015, my sister Kate gave birth to a healthy 8 pound baby girl. Mother and daughter are doing well, much to everyone's relief. Chris is looking pretty cheerful about it all, too. ;) No name has been decided on as yet.

Perhaps now we can get back to work...

[Update: It has been confirmed, the name is to be "Holly."]

[cool] # .

While composing an email to Mark Billinghurst about his GloveGRASP work, I followed a few more links and found some other potentially useful stuff:

• 3DUI mailing list (not sure whether it's still alive, though)
• OWorld
• Evaluation of Virtual Grabbing/Manipulation Techniques (via this page)
• 3D Widgets Library (apparently this is "available for public use")
• 3DUI Tutorial

Lots of interesting stuff to read in amongst that lot. Although some of it does look a little stale... Oh, and I think something like this done in glorious VR would be pretty cool, also.

[ngake] # .

Via a Joel Spolsky article, we have ESR's book The Art of Unix Programming. Looks like it could actually be worth a read. If only for other interesting links to be found within.

And now I've just ordered Christopher Alexander's magnum opus. I'm pretty sure it'll be worth the money. Not that I've actually yet tackled the last such book I bought.

[interesting] # .

Via NTK, vnc2swf - a VNC recording tool for Flash. This could be potentially useful for Ngake, both as a demonstration of the user interface and as a bug reporting mechanism.

[cool] # .

There's an interesting article on freedesktop.org where one Jim Gettys discusses the current state of X, multimedia and other useful topics related to the Linux desktop. Lots of potentially useful links (e.g. FreeType 2 and Cygwin/X), which is why I mention it here.

[interesting] # .

Well, I think I've finally managed to get my packages, symbols and functions successfully compiled into the Corman Lisp kernel. I can't be completely certain that it's working 100%, but it seems pretty stable. So at last I can move on to the next bits:

• Providing a Lisp interface to the Maverik VR kernel.
• An interface to the OpenAL library would be good, too.
• Implementing my hierarchical state machine library in C and Lisp.
• Start the UI framework...
• Finish the P5 glove driver and integrate that.

I guess it's time to stick the project(s) into source control...

[ngake] # .

Well, you might've thought that five weeks holiday would've been enough, but despite (or perhaps because) of that, I've needed most of the last three weeks as well before I've felt sufficiently "reinvigorated" to want to do much work.

However, I'm now feeling much better. Not that I was feeling really down, but I was having some difficulty feeling positive about things for a while there. And then plink, suddenly I'm rearing to go...

I did resolve not to feel guilty about it, which helped. Plus I'm getting into the meditation a bit more, and I'm hoping that will help me cultivate my equanimity in future.

'You think I'm experiencing a mood swing, Slick?'

'Yeah.'

'I'm enjoying a mood swing.'

Shit, have I really been working for myself for almost four years? Fortunately I now find it fairly easy not to freak myself out about this fact, but I admit I dread having to describe to anyone exactly what I've been doing all this time.

But still, I am (fairly) happy with where I am now and how I'm doing. Obviously I would like to be further forward than I am, but then I was even more willfully naïve about the problems I had to tackle when I started than I thought at the time.

So I'm still short on the "actual working software" front, but I'm pleased with my design, I've learnt a lot, I'm working better now than I ever have, and in the meantime the IT world is slowly moving towards the point where something like the Ngake VR IDE application in all its glory could actually have something like a receptive market waiting for it. So there.

Which isn't to say that a bit of a break working with some other talented, motivated people on some suitably interesting project for a decent lump of cash wouldn't be worth a go for a while. Stay tuned...

[bits] # .

I saw the Matrix Revolutions movie last week. I liked it, I thought they did a great job tying it all up. I'm now really looking forward to the Directors' Cut Special Extended Edition Deluxe Boxed Set...

[cool] # .

There's an interesting article in the MIT Technology Review about Charles Simonyi and intentional software (and his company).

In a few years, this stuff will become mainstream. This will validate, I believe, the ideas I had when I started Synchromesh (my ideas about manipulating higher-level models of code, if not my ideas about the effectiveness of VR interfaces). If I could just make a bit more progress with the actual software, that would be good.

[interesting] # .

I came across Edge magazine (again, I think), while checking to see what Jaron Lanier has been up to.

The magazine looks to have some very interesting articles in it. I read one on personal fabrication by Neil Gershenfeld that I thought was very insightful.

[interesting] # .

At some point Ngake is supposed to be able to fully parse idiomatic C and C++. This is going to be tricky. I've found some good information and a lot of useful links here. This page has a link to the Ph.D. thesis of one Edward Willink, entitled Meta-Compilation for C++, which looks like it could be well worth reading. All 412 pages of it...

I've also found some documentation for the Edison Design Group's C++ Front End that I'll be checking out.

Of course, I'm hoping that integration with Visual Studio will provide an effective alternative/complementary approach.

[ngake] # .

I think the #lisp channel on IRC is pretty cool. I'm trying not to act too much like a newbie. Today I shared a moment of Australasian humour with a guy called Luke, and when this happened

<dan-b> nice Road To Lisp, synchromesh
* dan-b catches up on essential reading
<synchromesh> Cheers, I'd forgotten about that.

I must admit I experienced a small thrill. Great, I'm a Lisp fanboy... Context: Dan Barlow is one of the principal maintainers of Steel Bank Common Lisp (SBCL), and the author of CLiki (the Common Lisp Wiki), amongst other groovy free Lisp stuff.

The Road to Lisp is a "newby-by-newby informal survey" on CLiki where people can post a description of how they came to be using Lisp. I wrote one myself. It has made a (the?) Top Ten. Cool! ;) [Update: the RtLS has been moved to the ALU Wiki, and my entry is here.]

And the filk? Well, someone posted a link to this song, being the recording of these lyrics. Like it says at the top, no warranty expressed or implied...

[lisp] # .

I've tidied up this blog, mainly as a procrastination exercise. Note the "archives" at the top, and the use of CSS to make it look exactly like it did before. A pleasant distraction, until I had to break out my mouldy Camel book, which was a sign that it was time to get back to some real work. After lunch...

I also found some groovy PDF quick reference cards here: CSS1, bash, vim etc.

[bits] # .

An interesting story in the Guardian about Cybersyn. I found this story quite impressive, since it took place around the time I was born. It just goes to show what you can achieve with the right mix of brilliance, revolutionary fervour and centralised authority. And the Americans are the bad guys! Also, it arguably makes today's IT "innovations" aimed at allowing people to work together and be more productive by sharing information blah blah seem rather tired and unambitious, I think, given that we're supposed to have made all this "progress" in the intervening thirty years. Or not. Whatever. Anyway, a sense of perspective is important.

Another good story in the New York Times magazine concerning affective forecasting, or how we determine how happy (or not) we think outcomes wil make us. Apparently we're not very good at it.

I've also (re-)discovered IRC, specifically the #lisp channel. I'm using mIRC, which looks like it might be worth USD20. #lisp is looking quite interesting so far; already I've found a few pointers (some fairly oblique) to useful stuff, like Building Applications Using DUIM, which describes the Dylan successor to CLIM (although this wasn't the DUIM vs. CLIM comparison that was mentioned). It's interesting to see what Lispers chat about, too...

[links] # .

I had lunch with Fiona today, which was nice. She's engaged, which is great! There's a lot of it about, it seems. I can't believe it's September already. I don't know where August went... but I'm sure that's connected to me finally getting to work exclusively on Ngake. Oh well, I just have to see how much I can get done before we go away (in three weeks!). And we did buy a house (well, a small flat) last month. ObLink: I found this, Bruce Tognazzini talking about how they experimentally verified that using the mouse is faster than using keyboard shortcuts. But that was 1989...

[random] # .

I received the International Lisp Conference 2002 videos last week; about two gigabytes of Windows Media movies, showing speakers from the conference talking about Lisp, AI, the Web, and Bioinformatics. Some big names (or at least, people I've heard of): Queinnec, Serrano, Stallman, Corman, Lenat, Norvig, Gabriel. I've only watched the Roger Corman one so far, though. I'd like to include them in my Lisp study program (hah!): watch one or two each week and post a short summary here.

[lisp] # .

I've finally got my Lisp source file parser in awk working (on my third attempt). Now to quickly code up the rest of it and get the CCL Lisp source formatted as HTML; I've spent too long on this already.

[lisp/corman] # .

Bernadette came across the word "monad" while doing an assignment for her yoga course. I'd heard it used to describe the stateful stuff in Haskell, but I didn't know what the word meant. According to Chambers's Twentieth Century Dictionary:

monad, mon'ad, n. the number one: a unit: an ultimate unit of being, material and psychical: a spirit: God: a hypothetical primitive living organism or unit of organic life: a flagellate of the genus Monas or akin to it: a univalent element, atom or radical.-Also adj.-adjs. monad'ic, -al; monad'iform, like a monad.-ns. mon'adism, monadol'ogy, a theory or doctrine of monads, esp. Leibniz's; mon'as, a monad: Monas, a genus of flagellates. [Gr. monas, -ados, a unit-monos, alone.]

[interesting] # .

What I need is a program that grovels through the Lisp files in the CCL sys directory and produces massively crosslinked, properly indented, colourised HTML files. Features would ideally include:

• A frame-based layout (à la JavaDoc) organised by package, file, and function/macro.
• Links from each function/macro symbol to either its definition or a "called by" table.
• The ability to handle CCL's hybrid assembly language/Lisp.
• Re-indentation/pretty-printing. Perhaps with links to the "original" source, rather than trying to keep end-of-line comments etc.
• Links to macroexpanded versions of definitions.

Well, I've found the XP pretty-printer that comes with Corman Lisp, and I'm printing the original MIT AI Lab technical report that seems to be the documentation.

[lisp/corman] # .

If you introduce reality into a virtual world, it's no longer a virtual world: it's just an adjunct to the real world. It ceases to be a place, and reverts to being a medium. Immersion is enhanced by closeness to reality, but thwarted by isomorphism with it: the act of will required to suspend disbelief is what sustains a player's drive to be, but it disappears when there is no disbelief required.

Richard A. Bartle, Not Yet, you Fools!

[interesting] # .

The current Ngake to-do/"issues" list follows:

• Extend the maverik_wrapper.dll library to hide all references to structures passed by value in libmaverik.dll (i.e. function parameters as well as return values).
• Then, I can finish relax.lisp and finally figure out where the P5 IR LEDs are. This will also be my first "non-trivial" CCL & Maverik program.
• Once I've done that, I can re-visit my ap5hid.lib P5 glove interface library.
• The first point raises the question: should I spend the time to figure out how to improve the CCL FFI so that it can handle structures passed on the stack? It would be educational and useful and might earn me some kudos on the mailing list, but I can work around it by writing these glue libraries (ugly a solution as that is). I should spend a few days investigating it and see how far I get, I suppose.
• Another thing I need to think about is an alternative to executing DEFUN-DLL forms at run-time to link to foreign functions. With Ngake everything will either be statically linked or at least linked to a known set of functions in a DLL. Once again, delving below the hood of the CCL FFI would allow me to customise a useful solution.
• Then there's the Ngake core implementation, for which I'll need dynamic hierarchical state machine code in C and Lisp, event queuing/dispatch, event registry, type factory etc.

And we leave for the UK in six weeks...

[ngake] # .

I've finally released all the programs (and even some documentation!) to South Park, so that they can start their acceptance testing in earnest. Huzzah!

Now I'm free to do, ooh, all sorts of stuff:

• Try to figure out why the monolithic FreeBSD kernel I cunningly installed on the hard disk in the Libretto won't boot.
• Get our new old network hub off my desk and onto the wall with the others.
• Decide which combination of FreeBSD, Debian, Gentoo etc. I want to run on which machines, and actually install/upgrade stuff to suit. Then we'll at least be able to tunnel GRE-type protocols, which would be useful.
• And of course get as much Ngake code written as I can before we go overseas next month...

At least I've got rid of the millstone that was the old ThinkPad 760 I foolishly bought; I hope David actually gets some useful benefit from it while he's overseas...

[bits] # .

This guy is the man.

Complexity is a perceived quality that comes from the difficulty in understanding or describing many layers of inter-related parts. An Organic Information Visualization provides a means for viewers to engage in an active deconstruction of a data set. The complexity is pulled apart through a combination of real-time user interaction as well as control of the data set through modification of the rules used for representation.

From the introduction to his Masters thesis, Organic Information Design.

[ideas] # .

I'm listening to "Trade", from Bob Mould's latest album Modulate. Apparently it was originally a Hüsker Dü song, but to me it sounds suspiciously like Alan Parsons. It's still a good album, though.

Oops, I've just bought LoudBomb and LiveDog98. Kids, don't drink & surf.

[random] # .

I checked out the ER discussion boards yesterday and noticed that they've released some Linux drivers. Similar deal to the Windows SDK: a closed-source library and some sample C++ files. Still, all grist for the mill, as it were.

I also downloaded (somewhat accidentally) the new P5 Windows SDK, all forty-odd megabytes of it. This unpacked to over eighty megabytes, about seven of which was useful, the rest being build byproducts and gratuitous auxiliary "media" files for the sample applications. So I posted a message complaining about the bloated SDK.

I'm looking forward to finishing up the South Park work, so that I can get back to this for a few days and make a bit more progress.

[p5] # .

There's a review of a WAP version of Elite. Wow, major flashbacks to all the hours I spent playing this game on my humble Acorn Electron. I love the quote at the end of the review: "converting one out of date technology to another." ;)

[cool] # .

Someone linked to Eidola (pronounced "EI-doh-lah," preferably), "a representation-independent, object-oriented, visual programming language." Produced by one Paul Cantrell. It doesn't seem to have been worked on recently, and seems somewhat Java-oriented, but it certainly bears closer inspection.

Not everyone seems impressed by it, though.

[interesting] # .

He mentions the Communications of the ACM; I'd decided to join the ACM after they sent me a membership form, but I haven't got around to it yet (or been paid...). I will, though.

But that's not the interesting part. He talks a bit about Game elements in non-game applications and using ambient music in an interface, which I'd been thinking about myself (no, really). [Update: I found another good post by one Larry O'Brien, talking about how increases in hardware performance haven't impacted programmer productivity.]

[interesting] # .

Lemonodor reports that one David Lichteblau ("light blue") has hacked up a version of Franz' Emacs-Lisp-Interface (ELI) to run on SBCL. To be specific, he's implemented the Lisp side of its Lisp-Emacs protocol (LEP). At first glance, the code doesn't look too scary; I wonder how hard it would be to port to Corman Lisp? Probably easier than trying to get ILISP to work, and with several potential advantages:

1. The end result might well be better (e.g. the LEP protocol is bidirectional).
2. I could write and maintain it myself; no "ownership" hassles.
3. Being "implementation specific," I would find it easier to write.

One thing's for sure, editing in XEmacs, saving, and re-evaluating (load "...") in the CCL IDE is definitely the wrong thing.

David mentions in his README that SBCL needs a port of Eric Marsden's XREF cross-referencing facility. A quick Google finds Eric mentioning that he's apparently working on improvements to XREF in CMUCL. So if/when a port to CCL is attempted, it would be good to start with his latest code. I wonder what compiler hooks are needed?

[Update: I think I've found the most recent version of xref.lisp, dated 12 June.]

[Update: Neil van Dyke mentions in his July 3rd weblog entry that he's thinking of using the Franz ELI/LEP protocol for his Quack project. He's packaged the Franz files up here.]

[lisp/corman] # .

Ted Leung points to Successful Lisp, by David Lamkins. Apparently I'm allowed to make an offline copy for my own use, so I have. Thanks, David!

[lisp] # .

Another interesting Lispnik who likes writing up his ideas for the edification of others. Some good articles about basic game architecture in Lisp, including a simple event system.

[lisp] # .

The Ngake VR IDE will, of course, need to provide the equivalent functions of a text editor. The "text" that you edit is the visual manifestation of some bit of the underlying software model, in the same way that an icon or other widget is generated as part of a view model. Indeed, ideally the editing of text will be agglomerated with the manipulation of user interface elements in general.

Obviously you can't edit program source code without editing text. But I don't want to make a text editor embedded in a VR scene.

So that means that one of the "user manipulation modes" has to be something that feels like using a text editor, except it isn't just text you're editing; you can manipulate any small, inline, non-text widgets that happen to be in there as well. This means layout managers, focus & selection handlers etc. that together provide an editor-like idiom.

Reading about CLIM, Genera, and Lisp Machines, it becomes apparent that this sort of thing has been done before, in the "desktop" world:

One of the most common ways to get a foothold in Genera for debugging something is to (1) find a place where the thing you want to use appears visually, (2) click Super-Left to get the object into your hands for read-eval-print, inspection, etc, (3) use (ed (type-of *)) to find its source reliably and with no foreknowledge of what the type is, who wrote the program, what their file conventions were, who loaded it, or any of myriad other things that other systems make me do.

Kent M. Pitman, comp.lang.lisp 1999

Links:

• Portable Hemlock, by Gilbert Baumann
• CL-Emacs page on CLiki
• CLIM II Specification, hosted by Mike McDonald (author of McCLIM)
• The Craft of Text Editing, by Craig A. Finseth
• A comp.lang.lisp post describing the Deuce editor
• If It Works, It's Not AI: A Commercial Look at Artificial Intelligence Startups by Eve M. Phillips (thesis) - unrelated to editors, but interesting nonetheless.

What's the central philosophy behind the evolution of computing in general and Genera in particular?

Free programmers from thinking about unnecessary details and they can accomplish more. The more abstractly they are allowed to think, the bigger and harder the problems they can solve.

Genera Concepts manual

[ngake] # .

As seen on lemonodor, there's a "newby-by-newby" survey on CLiki, concerning "what led folks to give Lisp a serious try and what they think of it." So I took it.

Also, via CLiki, this discussion of user interface management systems in general and CLIM in particular.

And, the Symbolics Lisp Machine Museum. Argh, I must get back to work...

[lisp] # .

Via Joel Spolsky, some interesting-looking articles by Steve Pavlina.

[interesting] # .

At some point I'm going to have to code up the Lisp indentation rules.

[lisp] # .

His Coherence Engine blog has some good stuff. For example:

The fundamental question we should attempt to answer is how do we make representations of knowledge -- such as blogs, wikis, The Brain, knowledge management systems, libraries, encyclopedias, etc. -- provide three competing services: a graceful way for new readers to learn the content; a way for experienced users to find new information; and a way to encourage growth not just at the edges (temporal or otherwise), but organically throughout the system.

OK, perhaps that isn't that earth-shattering an observation, but generally he's pretty good.

[interesting] # .

There's an interview with James Gosling where he talks about his work on software tools that work with parse trees. In the comment forum I found a couple of good links, including one to ace, a "syntax driven C preprocessor (a[nd] baby theorum [sic] prover)" that he wrote in 1989.

Just goes to show that these things take time... It was interesting reading the comments on the interview; the same "Lisp has been doing this for decades," vs. "'average', 'working' programmers don't want/need these sophisticated tools/languages" ideas that I've seen quite often elsewhere.

I'm persevering on the basis of this theory:

1. I can produce a VR IDE that is actually really effective.
2. Using my VR IDE (and the Ngake framework it is based on) I can then produce novel VR applications that are themselves really effective, because (a) Ngake is very good, (b) I will have carefully identified suitable market niches, and (c)the applications will have been developed in collaboration with domain experts.
3. People will buy these novel applications because they will find them to be really effective (thanks to some good marketing).
4. At some point, other software developers may also decide that my VR IDE is quite effective, and it may become a successful product of its own. And thus will I earn the respect and adulation of my peers. ;)

[interesting] # .

I've started writing AP5HID.LIB, an interface DLL for the P5 glove on Windows 2000/XP (it uses the high-resolution timers, which aren't available on Windows 9x AFAIK). The General Reality SDK DLL has a few shortcomings (e.g.C++ interface, no filtering), so I thought I'd roll my own in C. This has other benefits, like allowing me to port some of my SynLib routines to C, for use with Corman Lisp and Maverik.

Starting with the USB headers from the Microsoft DDK and the USB input record format described in Jason McMullan's P5 Linux library, I've developed it to the point where I can decode the sensor positions and bend data.

I'm now working on figuring out the 3D coordinates of the sensors, given the distances between them all (measured to 0.1mm resolution with a pair of digital calipers). This is proving a useful exercise in both fundamental coordinate geometry and expositive Lisp programming.

[p5] # .

A quick Google and there you have it: how to swap your Caps Lock and left Control keys in one swift registry entry. However, the example given on the page is buggy: the correct procedure is to create the binary value Scancode Map in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout with the following data:

0000 0000 0000 0000 0300 0000 1D00 3A00 3A00 1D00 0000 0000

It bloody works, though. Just like Linux...

[Updated 3/12/03: Damn, looks like I copied that byte sequence incorrectly. It should be correct now.]

[cool] # .

Someone has put the old Infocom Hitch-hikers Guide to the Galaxy text-only adventure game on the web as a Java applet. Cool. Not today, though; too much work on. Must... resist... [Update: more games here, and supporting materials here.]

[cool] # .

Another good pointer from Chis Double, this time to a post describing why continuations were removed from REBOL. The interesting part was a comparison of finite state machines versus stack machines for language interpreters. Not sure how/if this relates to Ngake, since although it is based on state machines, it isn't a language interpreter.

Also: Real Programmers Do Not Use Parser Generators, Why Bison is Becoming Extinct, and Real Programmers Don't Write Specs ("Strong typing is for people with weak memories"). Finally, a good article on abstractions.

Aside: I think I want Enter the Matrix (the computer game) for my birthday.

[ideas] # .

This is quite cool. Bevan and I got a chuckle out of this when we saw Trinity and the other "good guys" using the "right tools for the job." Nmap is "an open source utility for network exploration or security auditing... Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (ports) they are offering, what operating system (and OS version) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics."

[Update: It has been pointed out that using a 10.0.0.0 IP address was a missed opportunity for another tie-in, though.]

There's a good review of The Matrix Reloaded in Salon. I've stashed a local copy here, although you can read theirs for free (if you sit through an ad). [Update: the NY Times review is here.]

[cool] # .

Some people really like this Test Driven Development lark. There are a few TDD tools for Lisp available.

[lisp] # .

I feel lucky to be a geek in such a time as when films like this are being made.

[cool] # .

Only a day and a half until the Matrix Reloaded premiere... We saw The Animatrix on the weekend. That Second Renaissance Part 2 was pretty grim, but I liked them all. Roll on the DVD release in June.

• An interesting article in the Christian Science Monitor.
• Someone has typed up all of Keanu's lines from the first film. It's a pity the original page has been yanked.
• This Slate article is quite good, too.

Also, there's an interview with the guys who did the video for the Queens Of The Stone Age song Go With The Flow. (Nothing to do with The Matrix, obviously.)

[cool] # .

I think there's almost enough groundswell to start a petition that "the essays are all very good, but we want Paul Graham to ship Arc."

Or perhaps I'm exaggerating. ;) Oh well. I found this MP3 webcast by the man himself, which I shall listen to later. And now, back to writing my weekly report.

Now that I've settled on Lisp (again), I can continue doing the exercises from his book.

[lisp] # .

I've taken the clconsole.cpp code that comes with Corman Common Lisp and edited it into ngake-0.cpp and some support files. It seems to work well. Now I'm ready to bolt on the Maverik half, hook up the electrodes, raise the copper weathervane through the skylight into the dark and stormy sky and wait for the lightning to strike.

The question is, what interface should I provide to the CCL server? (I think I'll just call it the "Lisp engine", or possibly just "CCL".) I don't want to just "cheat" and provide, say, a socket connection to another console process.

Well, what's it for? Ngake contains code written in C, C++ and Lisp. So the Lisp engine is there to execute the Lisp code. Another purpose is to allow the system to be developed at least partially while it is running. That is, the user (as a Ngake developer) can load, examine, modify and save the Lisp source code, and the Lisp engine can recompile it into the running system. Additionally, the Lisp engine accepts "commands" from the user, i.e. it allows the user to invoke the Ngake Lisp functions directly. This is a complementary interface to the "virtual reality user interface" (VRUI). So, in summary, the purpose of the Lisp engine is:

1. To execute Ngake Lisp source code (e.g. state machine event handlers).
2. To compile at runtime, via the REPL, freshly updated Lisp source (via the "development environment"/"tweaking" interface).
3. To execute commands/expressions etc. from the user.

The point is, if I want Ngake to be "self-hosting" as soon as possible, then I have to provide the "load, examine, modify, save" capabilities mentioned above. Eventually, this will be provided by the first Ngake application, that is, the first application of the Ngake framework will be to build a "VR IDE" sufficient to fully describe and maintain both Ngake and the application itself. In the meantime, I have a small problem of bootstrapping.

The key idea is self-reference.

1. What are the elements of Ngake that need to be represented/modified within itself? (The set of elements will include the elements used to represent all the elements.)
2. What are the operations on those elements?
3. What is a useful initial representation of each element?

[ngake] # .

In other news, earlier tonight I went to the Splash Gordon annual sale and picked up a new weight-integrated BCD and some ex-rental (but still OK) regs, gauges & computer for a grand. Not a bad deal, really.

[random] # .

I'm still conflicted about the whole Lisp-or-Scheme, Windows-and/or-Linux thing (much to my sister's amusement).

A solid platform for growth

You have to have a Windows version, obviously. But I'm convinced that although the quality of their software is in general very high, Microsoft, not to put too fine a point on it, are the Borg (no offence, Jon ;)). I'm not saying they're actually evil, but let's just say that it's very important to me that my software runs on at least one non-Windows platform. At this point in time, Linux is the best choice there.

I'm also convinced that Lisp or Scheme (plus some C/C++ glue) is the way to go as far as development languages are concerned. I don't have enough experience in writing this kind of software to design a solution in enough detail to be able to implement it entirely in C/C++/Java/etc. without getting bogged down. I'm learning what I need as I go (which is fine), and Lisp's "extend the language in the direction of the solution" approach, coupled with the killer combo of garbage collection, the read-evaluate-print loop (REPL), macros, and a fully-integrated compiler, allows me to explore my way forwards without sacrificing any of the performance benefits of lower-level languages. It really looks like being the secret weapon I need to successfully realise my vision.

There are other requirements that point to Lisp as the Right Thing. First, the success of Ngake depends on the success of its 3D VR user interface. I anticipate an incredible amount of tweaking and fiddling with the UI code, often while another "guinea pig" user is sitting (im)patiently next to me, repeating some usability test for the n^th time. So I need the REPL of some embedded language in there somewhere so that I can tune it on the fly.

Second, the hierarchical state machines that drive the system have to operate at interactive frame rates, so performance is important. Having to keep rewriting bits of event handler code in C++ because Python (or whichever interpreted embedded language I'm using) isn't fast enough would defeat the purpose I've just described. Lisp is the only language I've seen (except perhaps GOO) that allows you to compile (and recompile) code you've entered/modified interactively.

So ideally I'd like a Lisp implementation that lets me develop on Windows and Linux. (Due to a limited apparent choice, I started looking at Schemes as well.) My other requirements are:

• A good foreign function interface, since Maverik makes heavy use of callback functions from C into your code.
• A read-evaluate-print loop (REPL), i.e. an interpreter. This rules out Bigloo, as far as I can tell.
• A compiler, ideally one that can be invoked repeatedly from the REPL. CHICKEN looks like it comes close: you could invoke chicken and gcc on your Scheme file and dynamically load the resulting object code into your application process, but only once. You can't then tweak the Scheme, recompile and reload.
• Must be affordable. Corman CL is; LispWorks might be; Franz Allegro CL, unfortunately, definitely isn't.

In short, Corman Common Lisp meets all my requirements except that it doesn't have a Linux version. This isn't surprising; its level of support for e.g. native threads is, shall we say, incompatible with Linux, where native threads aren't even particularly native. Also, its excellent FFI speaks of a focused, pragmatic approach to working with foreign code in order to produce better applications; the Linux Lisp systems that I looked at (CMUCL and SBCL) didn't feel like that.

So I guess I should relax about the Linux version, let go of my Bill Gates phobia for a while and focus on getting the bloody code written using the best tool(s) available.

[lisp] # .

Here's my current scorecard for building CHICKEN Scheme on Windows and Linux:

• Windows 2000, Visual C++ 6, CHICKEN Build 7: Success
• Gentoo Linux, gcc 3.2.2, CHICKEN Build 7: Success
• Windows 2000, gcc 3.2.2+Cygwin, CHICKEN Build 7: Failure
• Gentoo Linux, gcc 3.2.2, CHICKEN CVS: Failure

The Cygwin build fails with undefined reference to `_WinMain@16' in cygchicken-0.dll, for which the FAQ says "try adding an empty main(), or perhaps you have `-lm' too early in the command line." There's no -lm at all: the full transcript is:

gcc -Wl,--base-file,.libs/cygchicken-0.dll-base -Wl,-e,__cygwin_dll_entry@12 -o \
    .libs/cygchicken-0.dll  libchicken_la-runtime.lo libchicken_la-library.lo libchicken_la-eval.lo \
    libchicken_la-syntax-case.lo libchicken_la-profiler.lo libchicken_la-scheduler.lo
/usr/lib/gcc-lib/i686-pc-cygwin/3.2/../../../libcygwin.a(libcmain.o)(.text+0x7c): undefined reference to `_WinMain@16'
collect2: ld returned 1 exit status
make[1]: *** [libchicken.la] Error 1

The CVS sources (updated this morning, after Felix's email) build on Linux, but when I try to run csi I get Error: (string-append) bad argument type - not a string: quote. Yesterday's error was... nope, it's scrolled off the top of my xterm, but it was something about `##sys#error-hook' is not defined - execution terminated.

Still, I have working versions on Linux and Windows, so I'm OK for the moment. It would just be nice to be able to update & build from CVS, rather than having to keep downloading snapshots. Although admittedly that does make it easier to build on both platforms.

[Update: The CVS build problem has been fixed.]

[lisp/chicken] # .

Bernadette bought a new Canon IXUS 400 digital camera a couple of days ago. It's very cool. It can stitch photos together. The source of the image below is 18000+ pixels wide.

I've also managed to get Jeff's (1998 vintage) Libretto working, after buying a replacement system board on eBay. It needs a new CMOS battery, and the floppy drive is poked, but hey, it works. It even has the full 32MB of RAM now. It's only a Pentium 120 MMX; you can overclock them to a mighty 166MHz, but it's like, what's the point? ;) Note the wireless Ethernet card in the photo below.

After reading this interview with some of the FreeBSD core developers, I thought I'd stick it on the Libretto (christened "bit"). I've ordered a FreeBSD 4.8 DVD from Aussie, since there are currently no decent local mirrors. Paradise has 4.7, and you can download the 4.8 mini-ISO, but that's about it, although the NZ FreeBSD Users Group are working on a full mirror, apparently. This way I might get some stickers (like I did with the OpenBSD 2.6 CD). Installing it with no floppy will be a bit of a challenge, but if I do it via the old ThinkPad it should work.

It now has two useful purposes:

• to run Emacs, CLISP, MzScheme, Haskell, Erlang, OCaml, GOO, Smalltalk and any other language I'm working my way through the textbook for
• to accept Compact Flash cards from Bernadette's camera (via a PCMCIA adapter; apparently the 16-bit bus is too slow to support a USB card) and provide secondary storage for photos & movies, particularly while we're overseas later this year

If you look closely in the corner of the photo above, you'll see my tickets to the Matrix Reloaded midnight premiere in two weeks time. These are the upholstered seats, with the 'La-Z-Boy' footrest and the little table for your beer. The bastards at Warner Brothers/Roadshow may have denied us our rightful place as the first in the world to see this film (note it says "Fri 16" and not "Thu 15"), but at least the cinema is showing it at the earliest possible opportunity. I can't wait...

[cool] # .

I found this via Joey DeVilla's blog. It's a graph showing the relationships between Gibson's "Sprawl" books. Doesn't show the antique dealer Julius Deane or any the AIs, though.

And on Karlin Lillington's blog I saw a link to a 1967 CBC broadcast of a young, hippie Bill Gibson. Very cool. Apparently he's going to stop blogging, 'cause it interferes with his novel-gestation processes. Bummer, but fair enough, I suppose.

[Update: the man himself comments on the footage here.]

[cool] # .

Well, I've installed and/or built SBCL 0.pre8.80 and CMUCL 18e on golem. When I run the equivalent of the following code:

(load-foreign "/home/john/src/maverik-6.2/lib/libmaverik.so")
(define-alien-routine ("mav_initialiseNoArgs" initialise-no-args) void)
(initialise-no-args)

in each, I get the Maverik "splash screen" in CMUCL, but in SBCL I get:

GNU Maverik v6.2 - Copyright (C) 1999-2002 Advanced Interfaces Group
XIO:  fatal IO error 0 (Success) on X server ":0.0"
      after 7 requests (6 known processed) with 0 events remaining.

which is bad. Well, the last two lines are. Actually, I see the above when running via the PuTTY ssh client; if I run it locally, it says:

GNU Maverik v6.2 - Copyright (C) 1999-2002 Advanced Interfaces Group
XIO:  fatal IO error 0 (Success) on X server ":0.0"
      after 0 requests (0 known processed) with 0 events remaining.

[lisp/sbcl] # .

Similar to FLET and LABELS, MACROLET allows you to define local macros, that is, macros that are defined in the lexical environment in which the macrolet form appears. The format is the same as DEFMACRO. The definition can include declarations, but these do not include the bodies of the expander functions in their scope.

Also, the "consequences are undefined if the local macro definitions reference any variable or function bindings that are visible in [the] lexical environment."

However, the following test code produced the same output on CCL, SBCL, CMUCL, Allegro CL and CLISP:

(defun macrolet-foo (x y)
  (macrolet ((m (x) `(format t "1: ~A 2: ~A~%" ,x x)))
    (m y)))

* (macrolet-foo "one" "two")
1: two 2: one
NIL

[lisp] # .

I have eg5.lisp working, modulo the keyboard handling. The callbacks work, though. I don't need that "defun-callback" stuff in parse-c-decls.lisp, so I'll rip it out.

I've thought of adding a CamelCase-to-dashed name mangler to the header parser, but (a) it would probably increase the number of name collisions, and (b) I think the mixed style helps the 'foreign' code stand out. The whole point is to build abstractions on top of this in Lisp, and thus hide the nasty C-isms.

I guess the keyboard handling is broken because the clconsole.exe and the Maverik window are competing for the keystrokes. Some slightly more cunning way of embedding the Lisp in the Maverik application is required.

But my next trick will be to get the equivalent of eg5.lisp running in Allegro CL and/or CHICKEN Scheme (and/or MzScheme, since both Schemes have SWIG back-ends).

I am now officially ahead of where I was in December.

[lisp/corman] # .

I've managed to get my transliterated Maverik sample program eg4.lisp to work under CCL 2.01, and I had to set far too many breakpoints in disassembled code on the way.

In the end, breaking up my monolithic chunk of premasticated C headers into about five (polylithic?) bits before feeding them to parse-c-decls.lisp seemed to make the difference, echoing a recent thread on the mailing list. In other words, I don't know exactly what was causing the problem. ;) Of course, all this code worked fine under version 1.5.

I've modified my parse-c-decls.lisp to stick "+" characters around constants parsed from #defines, to avoid any potential name collisions. (Is "+" the standard idiom? Or should I be using "*"?)

Now I can try to get eg5.lisp working, that being the one with the callbacks from C to Lisp.

Damn, my schedule says I'm supposed to be writing SQL stored procedures today... Oh well, I'll continue this tomorrow.

[lisp/corman] # .

Crap. The FFI code I managed to get working (and grab a screenshot of) in December no longer works in CCL 2.01; apparently it recursively enters the GC, which is bad. Hmmm, I recall mention of this on the mailing list...

To be precise, if I run up the console and say (load "d:/src/.../eg1.lisp) (eg1), it immediately drops me back to the command line. If I cons anything first, then I get the error dialog.

OK, there was a bug in eg1.lisp [inconceivable!] ;).

Hmmm, eg4.lisp seems to be failing when the Maverik code accesses memory I've allocated and initialised via Lisp. Let's see if we can't build & debug all this lot in Visual Studio .NET. First thing, set the zlib project run-time library to "multithreaded" to match CormanLispServer, so the thing will link.

[lisp/corman] # .

The first in a series. Most of the special forms on the list looked at least vaguely familiar, but THE didn't ring any bells.

the lets you declare some or all of the type(s) of a (possibly multiply-valued) expression. e.g.:

(the fixnum (+ 5 7)) =>  12
(the integer (truncate 3.2 2)) =>  1, 1.2
(the (values integer) (truncate 3.2 2)) =>  1, 1.2
(the (values integer float) (truncate 3.2 2))   =>  1, 1.2
(the (values integer float symbol) (truncate 3.2 2)) =>  1, 1.2

CCL 2.01 doesn't check the bracketed type forms. Apparently "[t]he consequences are undefined if any result is not of the declared type"; CCL will raise an error when a single type specifier symbol is given, e.g. for (the symbol (+ 2 2)) but not for (the (values symbol) (+ 2 2)).

[lisp] # .

I've just installed Corman Common Lisp (CCL) version 2.01 and patches. My modifications to parse-c-decls.lisp haven't made it into the latest source. Oh well. At least the file is unchanged from version 1.5, so my file is still a drop-in replacement.

I occasionally wonder how Roger is getting on, since he switched to being Corman Technologies full-time.

[lisp/corman] # .

I wrote my first SQL stored procedure today. I'm not sure whether this is a step forward or not... [ooh, programmer snobbery alert]

[bits] # .

I see from his weblog that Chris Double is using the CHICKEN Scheme-to-C compiler. It looks quite good, and is supported by SWIG. So I'll add it to my list of candidate Lisp platforms, bringing the total to (approximately) five:

• Corman Common Lisp on Windows
• Allegro CL on Windows and Linux
• CMUCL and/or SBCL on Linux
• MzScheme on Windows and Linux
• CHICKEN on Windows and Linux

FFI issues play a large part; if I'm using MAVERIK, then I need to be able to do callbacks from C into Lisp. Also, threading: yes? No? Maybe? Co-operative? Native?

[lisp] # .

I just checked the Essential Reality discussion groups, and saw that there's now a Linux driver for the P5 data glove available here. Woohoo! Now I can add that to my list of foreign functions to interface to...

[p5] # .

OK, I've installed Blosxom 2.0rc2, since I can't get Blogger to successfully publish my blog on my web server any longer. And now that things are starting to happen at an observable rate, I want to be able to keep a decent record again.

[random] # .