last.fm has this nifty feature where you can write [artist]Meshuggah[/artist] and it’ll turn into a link to that artist’s last.fm page. A set of custom BBCode tags, basically. The project I’m working on, got-djent.com, is also a website about music, and will contain a band/album database maintained by the users in a wiki-like fashion. On the front page, it will be possible to post news articles. I thought it would be nice if band and album pages could be linked from these articles easily, by using custom tags.

Installing the customfilter module

What I set out to implement was having <band>Meshuggah</band> turn into <a href="http://got-djent.com/node/2">Meshuggah</a>, e.g. a link to that band’s page. Something like this is best implemented as a “custom filter” in Drupal 6. To be able to do this, you have to install and enable the customfilter module first. You can get it here. Once you have uploaded it to the modules/ directory of your Drupal install, you can enable it under Administer > Site Building > Modules (admin/build/modules).

Drupal is pretty nifty.

Creating the filter

Now we can create a custom filter. To do this, go to Administer > Site configuration > Custom filters (admin/settings/customfilter) and click Add filter. Fill out the fields (only the Name field is compulsory). I called my filter “Node links”. It is also a good idea to disable filter caching here (it is enabled by default), for the time being. This makes debugging a lot less frustrating, should any issues arise. Once you have verified that the filter works correctly, you can enable caching again.

To use the filter, you have to add it to a input format. I chose to be lazy and just add it to the builtin “Filtered HTML” input format, since that was already being used everywhere. To do this, Go to Administer > Site configuration > Input formats (admin/settings/filters) and click the configure link for “Filtered HTML”. Under the Filters heading, Tick the box next to your custom filter and click the Save configuration button.

To make sure the filter is applied at the right time, click configure again, and this time click the Rearrange link at the top. This will allow you to rearrange the order in which the filters are applied. For some reason, the default order rendered my filter useless so I had to change this. I dragged my custom filter all the way to the top. This is probably a bit overkill, but at least it works properly now.

Adding replacement rules

For the filter to actually do anything, you have to add replacement rules to it. In the Custom filters overview, click the filter’s name, then click Add rule at the top. Give the rule a name (my filter has two rules, called “bands” and “releases”). Now comes the interesting part: you have to specify a pattern and a replacement text.

The pattern is specified as a regular expression. More information about regular expressions can be found at regular-expressions.info. If you don’t know what they are, you might want to familiarise yourself with the concept before you continue reading. The pattern I used for the “bands” rule was relatively simple:

#<band>(.*?)</band>#i

This matches text between <band></band> tags. The #’s are the delimiters. The i modifier flag at the end makes the expression case insensitive. The parentheses form a capture group, which means that whatever is matched between them will be available when we construct the replacement text. Within the capture group, we use lazy matching (*? is the lazy version of *) to make sure that the filter works correctly when the tag is used multiple times. The * wildcard is greedy by default, which means that it attempts to consume as many characters as possible; <band>foo</band> <band>bar</band> would result in a single match, where (.*) matches foo</band> <band>bar! Adding the question mark prevents this.

Next, we have to specify the replacement text. We have two options here: the simplest is just entering some text, in which $1 or ${1} is included as a placeholder, which will be replaced by whatever was matched inside the first capture group of the regular expression. Note that you can also use $2, $3, … when you have multiple capture groups. $0 always contains the entire match. Unfortunately, this method isn’t suited for what we’re trying to do. Luckily, there is a more advanced option available as well: we can tick the PHP Code box, and then enter some custom PHP code to “compute” the replacement text instead.

Computing the replacement text

The customfilter module now expects us to write some PHP code which computes the replacement text and stores it in the $result variable. It can access the regular expression matches through the $matches array. $matches[n] corresponds to $n, so the value we are interested in here is $matches[1]. We have to find a node with the band node type, whose title corresponds to this value.

The simplest way to do this is to use an alternative (and badly documented) syntax for Drupal’s node_load function:

$nd = node_load(array('title' => $matches[1], 'type' => 'band'));

The title matching is case insensitive, which is convenient. However, this only works properly if we can find an exact (case insensitive) match. Seeing as bands and albums can have pretty long names sometimes (like, say, Fredrik Thordendal’s Special Defects), it would be even more convenient if we could specify the name only partially. The solution is to use Drupal’s search functionality.

I’m not sure, but I think you have to enable the search optional core module for this to work. Instead of immediately calling node_load, we run a search first. For this, we can use the do_search function. Note that we need some trickery here to limit the search results to certain nodetypes (the 3rd and 4th arguments):

$res = do_search($matches[1], 'node', "INNER JOIN {node} n ON n.nid = i.sid", "(n.type = 'band')");

do_search only returns the 10 best results, which is fine because we only need a single one in this case. If you intend to use it for other purposes, this is something to take into account, though. The results are returned as an array. The sid field of a result corresponds to the node id of the matched node. This means that we can acquire the node object we need as follows:

$nd = node_load($res[0]->sid);

Now all we need to do is use the node information to create a link. Drupal provides a convenient function for this called l:

$result = l($nd->title, 'node/' . $nd->nid);

Now, we can enter <band>thordendal</band>, and it will be replaced by a correct link: <a href="http://got-djent.com/node/36">Fredrik Thordendal's Special Defects</a>. Nice. There are a few edge cases that we haven’t taken into account yet, though: what if there are no matches, for example? The nice thing to do is then to just return the text between the tags, without linking anywhere:

$res = do_search($matches[1], 'node', "INNER JOIN {node} n ON n.nid = i.sid", "(n.type = 'band')");
 
if (empty($res)) {
  $result = $matches[1]; // return the unchanged match if no node is found (the tags are removed)
}
else {
  $nd = node_load($res[0]->sid); // get the node corresponding to the first (best) result
  $result = l($nd->title, 'node/' . $nd->nid); // create a link
}

There is one more problem with this: do_search complains if you give it a single keyword that is shorter than 3 characters. It would be nice if this error was supressed and nothing happened, instead:

if (strlen(trim($matches[1])) < 3) {
  $result = $matches[1]; // search keywords need to be at least 3 characters, so do nothing
}
else {
  $res = do_search($matches[1], 'node', "INNER JOIN {node} n ON n.nid = i.sid", "(n.type = 'band')");
 
  if (empty($res)) {
    $result = $matches[1]; // return the unchanged match if no node is found (the tags are removed)
  }
  else {
    $nd = node_load($res[0]->sid); // get the node corresponding to the first (best) result
    $result = l($nd->title, 'node/' . $nd->nid); // create a link
  }
}

All done! Setting up <release> tags for albums works in exactly the same way. Note that using do_search has another advantage in this case: when there are two albums with the same name, the artist can be specified to differentiate between them. This also enables linking to albums with names shorter than 3 characters. To link to the album “I” by the band “Xerath” (an awesome album by the way ), you can enter <release>Xerath I</release>. To link to the album “I” by the band “Meshuggah” (also a brilliant album), enter <release>Meshuggah I</release>.

Stuff that I forgot to mention earlier

To experiment with PHP code using the Drupal API, you can use the method described here. In short: enable the PHP filter optional core module, stuff your code into the body field of a new node, set the input format to “PHP code” and preview it.

It is also interesting to note that this works great with the pathauto module; the links inserted are the aliased versions, if they exist. The l function takes care of this, apparently. This means that <band>Meshuggah</band> will link to http://got-djent.com/band/meshuggah, rather than to http://got-djent.com/node/2. Awesome .

A downside of this method is that it isn’t very DRY. The same code has to be added for each content type. I guess it could be interesting to turn this into a full fledged Drupal module, but I don’t have the time to write and maintain it at the moment. If you think this is useful and would like to implement it, by all means, go ahead.

syncdb is not agile

Django’s syncdb management command, used to synchronise the database with the model definitions, is often criticised because it refuses to alter already existing tables. Django’s development team stands by this. They reason that altering the tables to match changed models is a very complex and ambiguous process, which is too risky to be completely automated. They are probably right. There are so many parameters to take into account. In addition, it is basically impossible to tell whether a field has been changed, or whether a field has been removed and another one added.

This means that, when you change your models during development (and you most likely will, frequently), syncdb is of no use to update your database schema accordingly. Django’s documentation suggests that you use the sql command, which prints out the SQL that you could use to create the entire database schema matching the current model definitions from scratch. It is then up to you to distill some ALTER TABLE statements out of this.

What we like so much about web development frameworks is, amongst other things, the abstraction they offer in terms of persistence: Object-relational mapping, or ORM for short. In layman’s terms, we don’t have to write SQL queries, the framework writes them for us. We can just write our code in an object oriented fashion, where each entity in the application’s domain model is represented by a model object. Usually it suffices to call the save method on these objects to persist them. Most of the time this works great. Some ORM toolkits go pretty far in optimising the generated queries, too. But I digress.

Django - A web development framework

Django has no manners

All of a sudden, though, Django expects us to write DDL queries. From scratch, pretty much. How rude! It is inevitable that a database schema evolves during the development process, and it probably changes quite often too. We would very much like to avoid having to write queries each time. Many people have tried to find more elegant solutions for this, which gave rise to database migrations.

Migrations are cool

In essence, migrations are versioned scripts that define elementary schema changes, preferably one at a time (e.g. “add the column ‘email_address’ to table ‘user’”). They can be SQL scripts, but very often they are written using an additional abstraction layer (which sits in between SQL and ORM toolkits) that makes them database-agnostic. Preferably, a migration also specifies how to revert the changes it makes. The versioned nature of these scripts makes it possible to quickly bring the database schema up to date, or revert it to an earlier version; in most cases without data loss. They can also be used to recreate the schema elsewhere: in another database, one that is used solely for testing for example, or even on another database engine, if they are database-agnostic.

Ruby on Rails comes with migrations baked in. You don’t have to use them, but you’d be crazy not to. It provides a simple API (or DSL, if you’re one of those guys ) to make schema changes independently of the database engine used. Django doesn’t come with anything of the sort. There are quite a few plugins that take care of this however, most notably django-evolution and South. As I have already mentioned in an earlier post, the past few weeks I’ve been seeing quite a bit of South1.

South - intelligent schema migrations for Django

South is awesome

South has one killer feature that sets it apart from most other similar tools: --auto. Whenever you create a migration with South, it “freezes” all your model definitions into the migration file. It isn’t a pretty sight, but it’s very, very useful; it makes it possible to detect changes to models automatically! This means that in most cases, South can write your migrations for you. All you have to do is review them and correct them if necessary. This holds the middle ground between Django’s radical “I ain’t touchin’ that” approach and automatically performing all changes. By introducing the possibility to review the changes, you get the best of both worlds.

checkdb --fix is evil

Automatically performing all changes is basically what checkdb --fix, the script I implemented, does. Usually, this is a very bad idea. In fact, we are already going to use South, so you would think there is no need for such a thing. However, checkdb is intended to be used as a diagnostic tool, and as a “last resort” option if migrations don’t cut it, for whatever reason. One important use case for checkdb is to verify that the database schema is correct before running any tests. This is, as far as I know, primarily what it will be used for. I used South’s excellent database API to implement it, in order for it to be as database-agnostic as possible.

Unfortunately, I am not at liberty to release the code into the wild (at least I think I’m not). I can, however, discuss a an issue I ran into while writing it. One of the biggest problems I encountered was the lousy introspection support in most Python DB API implementations. Database introspection is pretty essential to get a good view of what the schema looks like.

DB API implementations are lousy

The Python DB API is a specification that many Python database APIs adhere to, in order to provide uniform access. Alas, once you start writing queries this becomes a problem again, because each database engine has its own dialect of SQL. Anyway, what a DB API implementation should look like is specified in PEP 249. For introspection purposes, it defines a cursor.description attribute, which describes the structure of the result columns that were returned by the last query performed with the specified cursor. For each column, seven values are supplied: (name, type_code, display_size, internal_size, precision, scale, null_ok). The values in question are set to None if they are irrelevant for the given column, or if they are not implemented.

The latter seems to happen rather a lot. Sometimes the implementations are even plain wrong, like MySQL’s, which returns an invalid internal_size for VARCHAR columns. SQLite’s only implements the first value (name)! postgresql_psycopg2 is actually the “best” implementation, but it doesn’t implement null_ok, which specifies whether the given column is nullable, and which is really quite important for checkdb. Actually, I haven’t run into any implementations that return a sensible value for null_ok. How hard can it be to make this work? Anyway, this meant I had to find a workaround.

Standards are useful…

While googling, I discovered something called INFORMATION_SCHEMA, which is part of the SQL-92 standard. Basically, it’s a “virtual” database that contains metadata about the databases themselves. This kind of feature is called a database catalog, and most database engines support it. But until recently, each had its own way of structuring this information. MySQL supports INFORMATION_SCHEMA since version 5, PostgreSQL has supported it since version 7.4. There are a couple of others, but these two were the most relevant for my work.

I got kind of lucky, because a fix for MySQL’s erroneous internal_size had already been written. What I had to implement was null_ok. The solution was to run the following query:

SELECT is_nullable FROM INFORMATION_SCHEMA.COLUMNS
WHERE table_name = 'tbl_name' AND column_name = 'col_name' AND table_catalog = 'db_name';

In PostgreSQL, this returns YES or NO, which I then converted to a boolean and inserted in to the tuple returned by cursor.description, for the specified column.

…if they are unambiguous

In MySQL however, this does not work. So much for standards. You have to use this instead:

SELECT is_nullable FROM INFORMATION_SCHEMA.COLUMNS
WHERE table_name = 'tbl_name' AND column_name = 'col_name' AND table_schema = 'db_name';

Note the subtle difference; in MySQL, the database name is stored in table_schema, whereas it is stored in table_catalog in PostgreSQL. Note that both columns exist for both database engines, but the other always contains a worthless value; table_schema is always set to public in PostgreSQL, for example. There is probably a very good reason for this. Right?

I thought I’d do a write-up of this, because database introspection seems to be neglected very often in Python, both in terms of implementation and documentation. Let me know if it helped you!

On a different note, I fixed a slight cosmetic issue with article titles on this blog. They should no longer overlap with the dates, even if your resolution is lower than 1680×1050

Nope, this is a rant. I just spent almost two hours trying to figure out how to get Apache to remove the “www.” from URLs. mod_rewrite, goddamn. It isn’t so much the horrible syntax, or the contradictory information to be found about it, as the fact that it was basically ignoring what I wrote the whole time. Really, really frustrating. I don’t know why, either. Feel free to continue reading and let me know what I was doing wrong, if you want.

The reason I’ve been wasting my time on this is because I’m going to set up a second website on my VPS: got-djent.com. It’s going to be a community portal for people who are interested in djent. Djent is a music thing. Some would argue it is a genre on its own, but I’m not going to speak out on that. If you’re into progressive music and/or metal, you might want to check it out. Anyway, I run a fairly large last.fm group called “djent”. Recently, we decided that we’d like to have some more options available than just a shoutbox and a forum with limited functionality, so we’re going to set up a Drupal-based community portal. Perhaps I’ll write more about my Drupal adventures later.

To make this work, I had to set up Apache so there would be two virtual hosts, one for each website. My main site (this one) is the default. I set the ServerName directive to got-djent.com for the other one, and that seems to be working fine now. Everything was fine up to here.

Unfortunately, www.got-djent.com was still pointing to my main site. My guess is that this is because the hostname doesn’t match the ServerName of the virtual host, so it falls back to the default. Luckily, there is also a ServerAlias directive, so I added ServerAlias www.got-djent.com below that and it worked fine.

Some keyboards have to endure even worse fates.

However, “www” is useless. What is it there for anyway? To remind us that we’re surfing the World Wide Web? Frankly, I think most internet denizens can tell a URL from an email address these days, without needing the “www” prefix to make it blatantly obvious. We would be better off without it. It would save us a lot of keystrokes. And put less strain on our W keys, somehow it seems unfair that they have to endure such ongoing abuse

On a more serious note, the prefix affects caching; if you visit http://example.com/foo/bar.html, and then http://www.example.com/foo/bar.html later, the entire page will be reloaded (including all images hosted on the same domain) because your browser can’t tell that the URLs actually point to the same page. This means that it’s definitely a good idea to make only one of these available. Just serving the user a 404 page on the other is a bit rude though, so the nice thing to do is to install a redirect. A website has been set up to promote this cause: no-www.org1.

And so, I set out to make my domains class B compliant. I got there eventually, but it was quite a pain. What I needed to do, basically, was throw this into my Apache configuration somewhere:

RewriteEngine On

RewriteCond %{HTTP_HOST} ^www.got-djent.com$ [NC]
RewriteRule ^(.*)$ http://got-djent.com/$1 [R=301,L]

RewriteCond %{HTTP_HOST} ^www.benanne.net$ [NC]
RewriteRule ^(.*)$ http://benanne.net/$1 [R=301,L]

Someone once said that mod_rewrite is voodoo. If you think these five lines aren’t particularly easy on the eyes, you can’t even begin to fathom what ghastly horrors lurk in the depths of URL rewriting. Seriously, this is nothing. Most of it can obviously be attributed to its extensive use of regular expressions, but even aside from that, it’s hardly elegant.

Let’s go through these five lines step by step. What RewriteEngine On does should be fairly obvious. RewriteCond %{HTTP_HOST} ^www.got-djent.com$ [NC] is a bit less transparent. RewriteCond describes a condition for a rewrite rule. Basically, it lets you match a set of variables associated with the incoming request to a regular expression. %{HTTP_HOST} is a variable that contains the host name that was used for the current request2. This is matched against ^www.got-djent.com$, a regular expression which will only match “www.got-djent.com” itself. The [NC] is an option flag that indicates the matching should be case insensitive (it stands for no case). A RewriteCond always affects the first RewriteRule that follows it.

Next up is RewriteRule ^(.*)$ http://got-djent.com/$1 [R=301,L]. As you can see, RewriteRule also takes two parameters: a regular expression and a… well, it’s a URL of sorts, but it has funny characters like dollar signs in it – a substitution pattern. The regular expression extracts information from the request URL3 and this is used to interpolate the backreferences (like $1) in the substitution pattern. In this case, we have the regular expression match everything4 and paste this after the domain name without the “www”.

This directive also has two option flags. [L] is for last; processing will stop after applying this rule, and any following rules will be ignored. [R] is for redirect, which is what we want to do here. Note that just [R] would send out a 302 Moved Temporarily status code. We explicitly specify 301 so it sends out 301 Moved Permanently. The last two lines are the same as the previous two, but for my other domain. It is probably possible to consolidate these two into a single rule, but I’ve had enough of this for a while.

One thing I’m wondering about is why the dot wildcard (.) usually isn’t escaped in the regular expression provided to RewriteCond. Out of the stuff I found on Google, roughly half would escape them (www\.got-djent\.com) and the other half wouldn’t. I left the backslashes out here, because it looks a little bit nicer this way and they don’t seem to be doing anything. My guess is that they should actually be escaped, but because they are matched against %{HTTP_HOST} they can only ever match a dot anyway. The regular expression itself might also match, say, wwwwgot-djent_com, but that hostname doesn’t point to my server in the first place. If you have another explanation for this, or if you think mine is incorrect, please let me know in the comments.

Now, my main issue wasn’t the rules themselves, but where to put them. Nothing seemed to work. I put them in httpd.conf, in sites-available/ and in conf.d/5, and threw in some other directives without any luck. The rules just would not get picked up. I know the files in question were all being read though, because I also put RewriteLog "/var/log/rewrite.log" and RewriteLogLevel 9 in there for debugging purposes, and these definitely were being picked up. I tested this by browsing the Drupal install I’m working on, which has some rewrite rules to make its URLs prettier. Those rewrites would show up in the log just fine, but my own rules were doing nothing at all.

The solution, of course, is to put them inside an .htaccess file. However, not just any .htaccess file; it has to be the one in the document root of the default virtual host. There is a simple explanation for this: www.got-djent.com does not match the ServerName (I had obviously removed the ServerAlias directive earlier) of the got-djent.com virtual host, so the request never reaches it. Instead, it goes to the default virtual host. It took me the better part of two hours to finally realise this. Once I knew this, the solution was obvious.

I still don’t know why placing the directives in any of the main configuration files has no effect. According to Google this should work just fine, and to me it seems like a more natural place to put them in. I hope this post is helpful to anyone who is running into the same kind of problems. For others it will probably be a painfully boring read. I’m not sorry for this.

To justify the title, I should probably mention that my hate for XSLT is far worse. I respect the power of Apache’s URL rewrite rules, and the inevitable complexity that comes with this, but XSLT is just a giant sack of unnecessarily complicated shit. Aaaaaaaanyway. I am going to watch an episode of A Bit of Fry and Laurie and then go to bed. I think I’ve deserved that. Keep an eye out on got-djent.com if you’re interested, it’ll be going live in a few weeks or so. Good night

Anyway, the application I am working on is similar to Google Calendar in terms of user interface. Basically, it allows VTK’s2 praesidium and volunteers to indicate when and how they wish to help for every activity they organise. For each activity, a list of shifts is presented in a timetable. Usability is the main concern here, so everything is mouse-driven and easy to use. At least, that’s what it is intended to be. There’s a screenshot here, which should help to get the general idea. The most interesting part is the administration interface (for the creation of the shift lists), which allows for shifts to be drawn, resized and moved around on the grid with the mouse. jQuery and its many plugins (jQuery UI in particular) have made it relatively easy to get this done.

However, there was one thing that bothered me: DOM elements only receive mouse events when the mouse pointer is within their bounds. This is expected behaviour, but sometimes it’s nice to be able to have a certain element capture the mouse (i.e. keep receiving mouse events as long as a button is pressed, regardless of the position of the pointer), for example when dragging to scroll. Normally, when you accidentally move the mouse pointer outside of the scrollable area, the scrolling will stop because the DOM element is no longer receiving the mousemove event. In addition, this means that the element will never receive the mouseup event. I often perform cleanup actions in a mouseup handler; in this example, the mouse cursor might be changed during dragging, so it has to be reset afterwards. If the event is never triggered, this leaves the application in a “dirty” state.

For my student job last summer, I used Silverlight 2, which allows you to call CaptureMouse3 on any interface element. After this, all mouse events will be sent to this element, regardless of where they occur. This behaviour can be removed again with ReleaseMouseCapture. This makes it easy to create user friendly mouse-driven applications; just call the former in the mousedown handler, and the latter on mouseup.

A captured mouse.

Such a thing, at least to my knowledge, does not exist for web pages. However, this can be circumvented in a number of ways. For starters, it is interesting to note that Firefox4 always executes mouse event handlers bound to the document element when a mouse button is pressed and its window is active, even when the mouse pointer is actually outside of the window boundaries. This is interesting because it allows for Silverlight’s CaptureMouse behaviour to be mimiced.

Obviously, it makes no sense to bind mousedown handlers to document; the event would always be triggered, and there would be no way to find out which element was actually clicked. However, we could identify the clicked element in the mousedown handler and tuck it away somewhere; the mousemove and mouseup handlers are then able to access this information and realise the desired behaviour, even though they are not bound to the target element.

There are two ways to go about this (at least afaik, there might be more). One is to have each of the mousedown handlers store the element in a variable, which the other handlers check whenever they are invoked (e.g. whenever the corresponding event is triggered on the document element). This variable need not be global; it just has to be part of a scope that is available in all handler functions. You can easily prevent it from being global by wrapping your code like this:

(function() {
    // your code goes here
})();

An anonymous function is created and it is executed immediately. It serves no purpose except to create a local lexical scope; any variables declared inside the function (using the var keyword) are not available outside it. Note that this practice is recommended to jQuery plugin developers as well. But I digress.

Here’s an example of how this technique could work:

$(document).ready(function() {
    /* The code is already wrapped inside a function here, so we need not use
       the previously mentioned technique to protect the "targetElement"
       variable from being exposed globally. */
 
    var targetElement = null;
 
    $("#div1, #div2").mousedown(function() { targetElement = this; });
 
    $(document).mousemove(function() {
        if (targetElement.id == "div1") {
          // handler for #div1
        }
    });
 
    $(document).mousemove(function() {
        if (targetElement.id == "div2") {
          // handler for #div2
        }
    });
 
    // same for mouseup
});

The handlers could also be combined into a single one, containing a (potentially) large if-else structure, or maybe even a switch on targetElement.id. This all looks a bit awkward though, to me at least.

Another way to go about this is by binding the mousemove and mouseup events inside the mousedown handler (and unbinding them again on mouseup). This is the technique I’m currently using in the application I mentioned earlier. You can define the dynamically attached handlers inside the mousedown handler5, which gives them access to its scope; this eliminates the need for shared variables.

Here’s how it looks:

$(document).ready(function() {
    $("#div1").mousedown(function() {
        var target = this;
 
        var moveHandler = function() {
            // handler for #div1
        };
 
        var upHandler = function() {
            // handler for #div1
 
            $(document).unbind("mousemove", moveHandler);
            $(document).unbind("mouseup", upHandler);
        };
 
        $(document).mousemove(moveHandler);
        $(document).mouseup(upHandler);
    });
 
    // same for #div2
});

To be honest, I’m not sure which of these performs best. The first method has the advantage of only requiring binding at the start, whereas the second requires the mousemove and mouseup handlers to be rebound on every mousedown event. For the second method, on the other hand, only a single handler is ever bound to the mousemove and mouseup events (however, this is also true for the first, if all handlers are combined). It does not require any (potentially expensive) “identity checks” on the target element, either. Testing the performance of these different techniques would probably yield some interesting results, so I might do that later.

Both of these approaches contain a lot of boilerplate code. It would be nice if we could abstract this into a plugin. Thanks to jQuery’s extensibility, we can accomplish this fairly easily. I implemented the second of the described techniques, because it was easiest. I’m not sure how to go about implementing the first, because of the required identity checks.

So here it is:

• jquery.mousecapture.js – the plugin; tested with jQuery 1.3.1, but it should work with older versions.
• A simple demo – try it yourself! Start dragging on the black box, and move the mouse away from it; the move and up handlers will still be executed properly.

You use it like this:

$("#test_div").mousecapture({
    "down": function(e, s) {
        s.data = Date.now();
        $(this).html("mousedown triggered on " + s.data);
    },
    "move": function(e, s) {
        $(this).html("mousemove triggered on " + Date.now() + "capturing since " + s.data);
    },
    "up": function(e, s) {
        $(this).html("mouseup triggered on " + Date.now() + "capturing since " + s.data);
    },
});

Please take in account that I wrote the plugin while writing this article, so I haven’t tested it in practice yet. I might refactor the shift management application to use it, though. Feel free to suggest improvements, particularly pertaining to the sharing of data between the different handlers. I’m not too satisfied with how it handles that, but it does the job.

Update: it’s only a few lines of code, but I threw it on GitHub anyway.

I vastly prefer this:

class Array
  def rand
    self[Kernel::rand(self.size)]
  end
end
 
a = [1, 1, 2, 42, 24024]
a.rand # => 24024
a.rand # => 42

to this:

def array_rand(arr)
  arr[Kernel::rand(arr.size)]
end
 
a = [1, 1, 2, 42, 24024]
array_rand(a) # => 1
array_rand(a) # => 24024

Yuck. Looks like PHP1!

Open classes are also very dangerous. But hey, that’s Ruby for ya. Ruby trusts programmers to do the right thing, rather than patronise them. If they don’t, then that’s their problem, not Ruby’s. I think that makes a whole lot of sense. In a way it’s also kind of brave, because Ruby gets a lot of flak for this from people who think it’s irresponsible to give programmers this kind of power. Good for them, I guess – I think it’s rad.

I won’t go into much detail here about why they could be considered harmful; enough has already been written about that. What it usually boils down to is this: you might be modifying a method in an existing class somewhere. Someone else might be doing this too, while using your code. If he isn’t aware of what you did, he will expect the method he is modifying to exhibit its usual behaviour. Oops. Debugging this kind of thing is pretty annoying, as you can well imagine.

One could argue that that someone else should have taken the time to get to know your code better before using it. But I think reopening classes (monkey patching, if you will) is so entrenched in Ruby’s culture that this is not realistic. Rails does it, so it’s probably okay, right?

Well, in some cases it isn’t; particularly when other people are going to use your code, e.g. when you’re writing a library. I use this feature a lot; it keeps my code clean, readable and object oriented. I need Symbol#to_proc and Array#rand! But I don’t do this when I’m writing code that other people are just going to use, without taking the time to study its internal workings. Expecting them to do that is unreasonable in most cases. Thank god I don’t write a lot of library code, because it’s something I really miss.

Wouldn’t it be cool if… we could monkey patch on a per scope basis? We could edit the interfaces of objects we use to our liking, and nobody would ever need to know! I hope I didn’t get your hopes up with the title of this post, because it can’t be done. Yet. Well, probably never. I don’t know

Of course, there is mixology, or the more frivolously named mixico, one of _why’s creations. These two basically do the same thing; they are C extensions that allow you to mix out a module from a class (e.g. remove it from the inheritance chain). By default, Ruby only allows you to mix them in.

Alas, they are of little use for what I want to achieve, as far as I can tell. What it comes down to is that I want to change the interface of an object within a static scope; say, within a module declaration, or maybe within a single source file. You could do something like:

require "rubygems"
require "mixology"
 
module Translator
  module StringExtensions
    def pig_latin
      self.split(" ").collect do |word|
        word[1..-1] + word[0,1] + "ay"
      end.join(" ")
    end
  end
 
  str = "Hello world"
 
  str.mixin StringExtensions
  puts str.pig_latin # => "elloHay orldway"
 
  str.unmix StringExtensions
  puts str.pig_latin # NoMethodError: undefined method `pig_latin' for "Hello world":String
end

No comments about my pig_latin implementation please, I know it stinks. Just an example.

Anyway, this is worthless. For starters, it looks like crap. It would help if we could do this on a per class basis, instead of per object, but even then it would still be pretty verbose. In fairness, I don’t really have any ideas how to solve this problem syntactically in an elegant way. But that doesn’t matter because this doesn’t do what I want: if my object were to leave the static scope of the Translator module (like when I’d pass it as a parameter to an external method), it would still include the mixin. Exactly what I’m trying to avoid.

I guess I could try and write something like this myself but it really sounds like an arduous task, if it is possible at all. For starters, I don’t know anything about Ruby’s internal workings, or writing native extensions. My knowledge of C is fairly decent, but I’m not very fond of the language. In addition, I’d have to figure out a good solution for syntax. Nope… i’m affraid scoped mixins (or maybe mixin closures?) will remain but a dream.

In Ruby at least. Only a few days after I started thinking about this idea, I came across _why’s newest project on GitHub. It’s a mixin-oriented language called ‘Potion’. Conceptually, it separates behaviour from data, and allows local modifications to the behaviour part of objects: scoped mixins! I haven’t played with it, but it looks like an interesting experiment, to say the least.

Update: well what do you know. This seems to do what I described, pretty much. But it looks a bit scary.

root@evidence:~# /opt/ruby-enterprise-1.8.6-20081215/bin/passenger-install-apache2-module
[...]
Compiling and installing Apache 2 module...
cd /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/passenger-2.0.6
/opt/ruby-enterprise-1.8.6-20081215/bin/ruby -S rake clean apache2
/opt/ruby-enterprise-1.8.6-20081215/bin/ruby: No such file or directory -- rake (LoadError)

--------------------------------------------
It looks like something went wrong
[...]

Sucks. I was already bothering the guys in #passenger on FreeNode when Google sent me here; apparently this bug is already known and a patch seems to have been committed. The problem is that it’s using the rake that belongs to the primary Ruby installation to get things done, so it only occurs when you have more than one version of Ruby installed. I have MRI Ruby 1.8.6 installed in /usr/bin. I haven’t used that yet, so I haven’t bothered to install rake for it either, which explains why the module installer can’t find /usr/bin/rake.

Most suggestions in the Google Groups thread are pretty useless, but someone suggested

ln -s /opt/ruby-enterprise-1.8.6-20081215/bin/rake /usr/bin/rake

I don’t like symlinks that much, even if they’re only temporary, so I decided to “fix” it this way:

export PATH=/opt/ruby-enterprise-1.8.6-20081215/bin:$PATH

… and then it worked great.

At least, I think so. I still have to edit my Apache configuration and reboot the server. Actually, I don’t think I have a Rails application around to test it with… but I plan to fix that, soonish

Just thought I’d post this, seeing as other people have had the same problem.

Migrating WordPress blogs sure could be easier. I used this guide, but I had to change some URLs in the database manually before everything would work properly. Ah well, it could have been worse! I will also try to finish the stylesheet and add some cosmetic improvements in the near future. I don’t know if anybody has noticed, but the archive pages are mess right now. I also need to tweak things a bit further to balance reasonable performance with memory usage. Apache is such a hog. As a consequence, the server will probably be down occasionally in the next few days. My apologies on beforehand if this bothers anybody.

So yeah. Update your bookmarks (ha ha). http://code.lemuria-nation.net/ is now http://benanne.net/code/! I have installed a redirect at the old location, but I don’t plan on renewing that domain, so it’ll be gone in a month or so.

Seeing as this is just a short note to let everybody know about the move, I can throw in some completely unrelated stuff: read Abstruse Goose. It’s awesome! Also check out Brendan Gregg’s collection of special sysadmin tools!

There, that’s it. Don’t worry, my next post will be chock-full of code again

Methods vs. blocks

Methods and blocks in Ruby are similar things; they are both containers for code, and both of them can accept a number of arguments. On the other hand, the scoping rules and argument semantics for blocks and methods are very, very different.

Let’s start with scope. Blocks have access to the local variables of the scope they are created in; this is called static scoping. This implies that they can be used to create closures. Closures are a very useful construct for callbacks, for example2. Methods do not have this property.

Another difference is the way they handle arguments. Methods are pretty anal about them; if they expect you to give them three, you’d better give them three or they will complain by raising an ArgumentError.

# relevant output is in the comments
def foo(arg)
  arg
end
 
foo("bar") # => "bar"
foo # ArgumentError: wrong number of arguments (0 for 1)
foo("bar", "baz") # ArgumentError: wrong number of arguments (2 for 1)

Blocks are a different story. In general, they are a bit looser about accepting arguments they don’t expect. It depends, however. This is an aspect of Ruby that I’ve always found very confusing: there are two kinds of argument semantics for blocks. To understand how this is possible, a couple of things have to be explained first.

Procs vs. lambdas

There are two ways to create an object that encapsulates a block in Ruby: lambda3 or Proc.new. Let’s call the former type of object a lambda, and the latter a proc4. Procs and lambdas can be executed by using their call method. Procs and lambdas have different argument semantics, amongst a few other differences. Let’s compare them:

p2 = Proc.new { |foo, bar| [foo, bar] }
l2 = lambda { |foo, bar| [foo, bar] }
 
p2.call("foo", "bar") # => ["foo", "bar"]
p2.call("foo") # => ["foo", nil]
p2.call # => [nil, nil]
p2.call("foo", "bar", "baz") # => ["foo", "bar"]
 
l2.call("foo", "bar") # => ["foo", "bar"]
l2.call("foo") # ArgumentError: wrong number of arguments (1 for 2)
l2.call # ArgumentError: wrong number of arguments (0 for 2)
l2.call("foo", "bar", "baz") # ArgumentError: wrong number of arguments (3 for 2)

As you can see, procs really don’t care about what they’re given. Missing arguments are replaced with nil, and obsolete arguments are just ignored. Lambdas, on the other hand, seem to be as strict as methods. However, this is only true for lambdas with two or more arguments. Wha? Yeah, I know.

It gets worse

When a proc or lambda is created from a block with 0 or 1 arguments, different rules apply. Let’s look at a block with no arguments first:

p0 = Proc.new { puts "nothing here" }
l0 = lambda { puts "nothing here" }
 
p0.call # => nil
p0.call("foo") # => nil
 
l0.call # => nil
l0.call("foo") # => nil

Looks like both of them don’t really care: all arguments are ignored. This was to be expected for procs, but for lambdas this behaviour is a bit surprising. When our block has a single argument, other mechanisms come into play:

p1 = Proc.new { |foo| foo }
l1 = lambda { |foo| foo }
 
p1.call("foo") # => "foo"
p1.call("foo", "bar") # => ["foo", "bar"] # warning: multiple values for a block parameter (2 for 1)
p1.call # => nil # warning: multiple values for a block parameter (0 for 1)
 
l1.call("foo") # => "foo"
l1.call("foo", "bar") # => ["foo", "bar"] # warning: multiple values for a block parameter (2 for 1)
l1.call # => nil # warning: multiple values for a block parameter (0 for 1)

Again, lambdas and procs behave identically. When they are called with too many arguments, these are interpreted as an array, and a warning is issued. It’s almost like the splat operator is implicitly present. What’s a bit curious is that something similar also happens when no arguments are supplied; except nil is assigned to the argument, rather than [], which I would have found a more logical choice.

Strict vs. loose

Let’s bring some structure into all of this; we could say that there are two kinds of argument semantics in Ruby: strict and loose. Methods always have strict semantics. Procs always have loose semantics. Lambdas have loose semantics when they have 0 or 1 arguments, and strict semantics otherwise.

Of course, turning blocks into objects isn’t something we do very often. At least I don’t — yield is so much handier! It looks like using yield and implicit block parameters gives you loose semantics:

def run(*args)
  yield(*args)
end
 
run("bar", "baz") { |bar, baz| [bar, baz] } # => ["bar", "baz"]
run("bar") { |bar, baz| [bar, baz] } # => ["bar", nil]
run("bar", "baz", "bananas") { |bar, baz| [bar, baz] } # => ["bar", "baz"]

I can’t think of any good reasons why these things are the way they are. It’s quite complicated and not very well documented. If anyone knows any, please don’t hesitate to let me know.

Enter define_method

define_method lets you define methods dynamically. It turns a block in to a method, pretty much. As if things weren’t confusing enough! It exhibits some pretty strange behaviour, as mentioned in Paul Gross’s post. But, knowing what we know now about argument semantics, it looks a lot like define_method turns blocks into lambdas and lets you call them as methods.

First, I just want to illustrate how define_method can indeed be useful because it allows you to make use of static scoping:

class Foo
  bar = "bar"
  def get_bar
    bar
  end
end
 
Foo.new.get_bar # NameError: undefined local variable or method `bar'
 
class Foo
  bar = "bar"
  define_method :get_bar do
    bar
  end
end
 
Foo.new.get_bar # => "bar"

I am aware that as examples go, this one is pretty contrived. I’m sorry, I can’t think of anything better

I figured out all of this by trying to reimplement define_method in pure Ruby. The whole proc vs. lambda thing was the missing link. Once I realised this, it was pretty straightforward. I’ll show you the end result, but I’m warning you, it isn’t pretty:

class Foo
  @@blocks = {}
 
  def self.my_def(name, &blk)
    @@blocks[name] = lambda(&blk) # IMPORTANT!
    eval("
    def #{name}(*args)
      @@blocks[:#{name}].call(*args)
    end
    ")
  end
 
  my_def :no_args do
    p "no args" 
  end
 
  my_def :one_arg do |one|
    p one
  end
 
  my_def :two_args do |one, two|
    p one
    p two
  end
end

This is basically Paul Gross’s example, but I replaced define_method by my own implementation. I used a class variable hash to store the blocks, because I had no idea how to make them available inside the created methods otherwise. Remember that method definitions are not statically scoped

There is hope yet

It is said that a lot of this has changed in Ruby 1.9. I haven’t gotten round to testing it yet, but I think I will. I hope they have considerably simplified these things. They should, if they haven’t.

He wanted to know what happens when you add both values indexed by number, and values indexed by strings (e.g. expando properties – no, I’m not making this up); particularly how the Array.length property would deal with that. I told him that in Javascript, length always returns the largest integer key value plus one. I’m not sure if that’s 100% correct, but that’s how I remember it, and his tests seemed to confirm this.

// Relevant output is in the comments.
var foo = ["bar"];
foo.length; // => 1
foo[1000] = "baz";
foo.length; // => 1001

He also mentioned that when adding more elements to the array with Array.push, assigned keys are also incremented from the largest available key value:

foo.push("bananas");
foo.indexOf("bananas"); // => 1001
foo.length; // => 1002

It’s almost as if:

Array.prototype.push = function(value) {
  this[this.length] = value;
}

I remember doing it like that, before I found out about the existence of push. Yuck. I guess it sorta makes sense for regular arrays, but it’s still fishy.

Then I started wondering if this behaviour could cause an overflow in an array with only two elements… because I figured that would be quite hilarious. Say we create an empty array, and insert an element with Javascript’s largest possible integer value as its key. Then, we push another element onto it. Chaos ensues! Right?

Turns out it’s not as simple as I first thought: there is no such thing as an “integer” in Javascript. All numbers are floats. Sweet Jesus! For those who don’t know how floats are stored internally: it’s comparable to scientific notation. In 1.0e20, 1.0 is the significand or mantissa, and 20 is the exponent. A floating point number consists of those two parts, represented as integers. Well, in reality it’s not as simple as that, but it isn’t far off.

What this implies is that for small numbers, precision is very high. The smaller the exponent2, the smaller the difference between floats with successive mantissas. For very large numbers, on the other hand, this difference becomes, well, pretty huge. In fact, at some point it becomes larger than one, which is a bit of a bummer when you’re forced to use them as integers. A demonstration:

1e300; // => 1e+300
1e300 + 1; // => 1e+300
1e300 == 1e300 + 1; // => true

Eh?
Javascript truncates floats. It ignores the least significant digits that cannot be represented.

The largest number that can be represented accurately in Javascript is…
wait for it…

0xfffffffffffff80000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000

… or about 1.798e+308. That’s 257 digits, not including the hexadecimal prefix 0x, in case you were wondering.

However, to make our array go completely bonkers, we don’t even have to go that far. 1e300, which is a bit easier to type, will do just fine. Get a load of this:

var foo = [];
foo[1e300] = "bar";
foo.length; // => 0
foo.indexOf("bar"); // => -1

What’s going on here? As of yet I haven’t been able to explain this behaviour. Please shed your light on this in a comment if you can figure it out! I would be eternally grateful. It almost seems like the element hasn’t been added at all! But this shows that it’s still in there:

for (var k in foo) {
  alert(k + ": " + foo[k]); // displays a dialog box with "1e+300: baz"
}

So I guess it mimics the behaviour of an expando property; those are also ignored for the calculation of the length property because they are not numbers. Except this one is. Weird.

Let’s see what happens if we push another element onto the array. Seeing as 1e300 + 1 == 1e300, maybe the original element will be overwritten?

foo.push("baz");
foo.length; // => 1
foo.indexOf("baz"); // => 0

Seriously, wtf?
It looks like the push operation uses the incorrect length property, which is set to 0, and concludes that this is the first element to be added to the array. Both elements are still there however, as can be shown by applying the previous for loop on it again.

I guess 1e300 is a pretty intangible number. “Nothing to worry about”, you’d think, “why would I want to use numbers that large in any sort of application?”. It turns out that indices larger than about 4e9 have this kind of effect. 4e9 Is only a couple of billion, really. For Javascript it’s probably a lot, but generally speaking it’s not that much.

In addition, I found that Javascript can discern between numbers that are only 1 apart until about 9e15, which makes me wonder why this weird behaviour suddenly pops up at about 4 billion. It seems so arbitrary. Anybody have any clues?

I guess the moral of the story is: don’t do this. Don’t use large numbers as array keys in Javascript. Also, don’t place more than 4 billion elements in a single array. Although I don’t think you were planning on doing that

Note that all of these experiments were conducted in Firefox 3.0.5 on Linux using Firebug. It would be interesting to see how other browsers respond to this nonsense, but I don’t have the time or the motivation to test those myself. If you do, please let me know what happens3.

Ruby on Rails

Rails is an MVC web development framework for Ruby. Rails is awesome and magical! Rails helps speed up development, and it’s just a lot of fun to use because it takes care of all the stuff that is annoying about developing web applications. Most of it, anyways.

Merb is an MVC web development framework for Ruby. Merb is probably awesome too, though I wouldn’t know because I’ve never really used it. “Merb” is a contraction of Mongrel and ERb, because originally all it did was serve ERb template pages through Mongrel. However, it’s turned into a lot more than that, and has become somewhat of an alternative to Rails.

Merb is Mongrel + ERb

The main difference with Rails is its underlying philosophy: Merb is framework-agnostic, focusses on performance and has a fixed public API. Until recently, Merb had the advantage over Rails of being thread safe. Check out this interview with Yehuda Katz if you want to know more about Merb and how it differs from Rails.

Apparently, Merb and Rails are joining forces. It almost sounds like a bad April fools joke, except it’s Christmas. Merb and Rails are different. In addition, it seemed like their development teams and communities sorta hated each others guts until very recently.

Rails is a full stack framework: it provides the M, the V and the C in MVC, and more. However, so is Merb. The difference is that Rails is opinionated: you get ActiveRecord to take care of ORM, Prototype to help you write cleaner, meaner javascript. Rails’s different components fit together like legos.  They were written for eachother, and this synergy allows you to do some pretty cool stuff. Writing your javascript code using Ruby, for example (RJS).

Merb on the other hand is framework-agnostic. You use whatever tools you want to use; as long as someone’s written an adapter for it, Merb will know how to deal with it. This is why Merb boasts support for ActiveRecord, DataMapper and Sequel (among others), jQuery and Prototype, … you get the drift.

Using other tools with Rails is definitely possible, but Rails was not built with this in mind, and it shows. You can use your favourite libraries if you want to, but it’s probably going to leave you with a bitter taste: a lot of handy shortcuts just aren’t going to work anymore. The synergy that makes Rails so… well, Rails, fades away. Note that a lot has been done to try and remedy this; take a look at the jRails project, for example. But remedies won’t do, as far as I’m concerned

This is just one of the things that’s made me want to give Merb a try; I happen to be quite an avid fan of jQuery. I’ve attempted to use it for small projects a couple of times (mind you, by most people’s standards all of my web development projects are small), but I always ended up going back to Rails because I just didn’t have the time to learn all the new stuff. Slices? Parts? Say what? I found that Merb and Rails handle things quite differently, even though they’re both MVC-based. Rails still did a great job of course, despite my pet peeves with it.

I wonder what will happen to Rails when things like framework agnosticism get thrown in the mix. I would love to give DataMapper and jQuery a try, but how will this affect the synergy that makes Rails so productive? Ideally, things like AJAX form helpers would work with any javascript library you can think of, but how realistic is this? Won’t the necessity to implement the same functionality with so many different backends hold back innovation? I’m very curious to see how they will handle this kind of problem.

In other completely unrelated news, my apologies for not posting  anything in the past few months. In fairness, I didn’t really have anything to write about. This doesn’t seem like the place to discuss school projects (although I did write a pretty cool cache hierarchy simulator in Ruby, maybe I’ll go into that later, we’ll see).