Recently in Perl Category

At some point, I really have to write about the new whatthefuck.com mail architecture. It is made up of some pretty epic hacks, but the new system is one hundred times more reliable and versatile than the old system.

Okay, I suppose that isn't saying much.

I do have to say this. I started out building on top of a courier/postfix based system, and ended up rolling my own on top of qpsmtpd, and will be writing a custom IMAP daemon for the newly-created system. The first version is very basic, but there are so many neat things I can do with it.

The new server with a terabyte and a half of space is heading to the data center this week, but if this works how I hope, there may be a few more servers heading over there soon enough. FreeBSD and ZFS is pretty much admin-heaven. PostgreSQL is a wonderful thing as well.

Some of you know that I'm a hackintosh user. I've been an Apple fan for a long time, but nothing in their lineup currently works for what I want out of a machine. My last Apple machine was a black MacBook with the original Intel Core Duo. It went in for service three times, and finally, Apple just denied anything was wrong with it. I ended up selling that machine, and the hard drive that was originally in it died a week later.

Huh.

Right now, I use a Lenovo/IBM ThinkPad T61 14" Widescreen edition, with a Core 2 Duo at 2.2GHz, 2gb RAM, a 100gb drive, and DVD-RW DL drive. The screen resolution is a comfortable 1440x900, I have some wireless, and life is good. I'm also running OS X 10.5, licensed but violating EULA, because it's one of the greatest operating systems we currently have available. There are a few little issues with running Leopard on this machine. Most things work, but I don't currently have a driver for the built in ethernet, the machine won't sleep if I have bluetooth enabled, the machine won't wake up if I enable the PC card slots, I can't control the brightness of the LCD, and when the laptop runs out of battery, it shuts off instead of going to sleep.

There are a few people working on two of the issues, namely the brightness issue with the Intel X3100 graphics controller, and a driver for the Intel 82566M Gigabit Ethernet controller. No sense in duplicating their efforts. The only kext I've ever written was a small driver to enable the tablet serial port on the X61 tablet that I had before this, so people could use TabletMagic and get full tablet functionality in OS X. Ethernet driver creator, I am not.

But, hey, I give you one little turd to make your life easier, and you may even find it useful if you're on a real Mac. This little package, 'sleep_monitor', installs a LaunchDaemon and a binary on your machine to keep track of how much battery you have left and put your machine to sleep at a certain threshold. It's a simple idea, but most Mac authors would charge $20 for the privilege. I give you it for free, but you don't get a GUI. Sorry.

When you install the package, it immediately starts. It will wait until you have 4% of your battery remaining, and puts your machine to sleep. On my T61 with the 4 cell battery, that usually kicks me into sleep when I have about 5 minutes of use left in the menu bar, and that will give me many hours of sleep to find an outlet to charge things up. Want to change the threshold? Edit /Library/LaunchDaemons/com.abstractwankery.sleep_monitor.plist and change the second ProgramArgument to the percentage you'd like to kill out at. Have an 8 cell battery? 2-3% is fine. Have a nearly dead battery? Maybe 15% is more your thing. Leave the % sign out of it, it will only cause problems.

SleepMonitor 0.1
Mac OS X 10.4, 10.5

Why does Twitter suck?
This morning, my manager asked if I used Twitter at all. While I have an account, and I have a bunch of 'Friends' on there, my attention span for the service has never really lasted much longer than 2-3 days. Every time someone asks me about it, I grab a client or two, try things out, and realize yet again how many things really annoy me about the service. The top two were how much of a pain it was to post, and what a bigger pain it was to keep a feed going. The best I could find for OS X was Twitterrific, and it was just counterintuitive. The free version would put ads in, the 'auto reappear' never did, and it would just silently fail half the time and not post anything new for hours. Awesomesauce.

Granted, a few things have been fixed. The SMS posting service makes it really easy to actually post remotely, as none of the S60 clients are very good, and the Java clients really suck. They don't seem to be down as much, and the speed isn't terrible. Rumor has it, they're getting rid of Ruby on Rails, which just makes me jump with glee.

Since then, for posting, someone put together an AppleScript called Tweet, which allows you to easily post to Twitter through QuickSilver. It works well for me, I just hit Command-Space, hit period, type my tweet, tab, then tweet it. It sounds complicated, but it's really easy to deal with. I like it, and it was easy as hell to do.

Then, there's the feed problem. RSS is too slow, I don't want it cluttering up Google Reader. Twitterrific still isn't working for me, as pretty as it is. There are two Dashboard modules, and neither of them work well, and silently fail at that. What I really wanted was something that would just post tweets to Growl and be done with it. I found something, but it was a Ruby script with manual configuration and little extensibility. I didn't want to screw with it, so off to /dev/null it went.

Hey, there's a point.

In the end, I wrote a perl script called TwitterGrowl to do exactly what I want it to do. To make life easier, it relies on the Twitter login information in your keychain to log in, and prompts you to create one if it doesn't find anything. It reports when there's a login failure, or Twitter goes into Suckfest, or when a system maintenance issue is posted. Better yet, I packaged it up into an easy to use, double clickable application. Pop it into your Applications folder, drag it to your Login Items, and it'll go into the background and sign in when you log in. Easy as pie. All of the required modules are in the application package, and you can feel free to browse the source by viewing the package contents and heading into the Resources folder.

So, now that I have a steady "works for me", would anyone else like to give it a shot and see if anything breaks? Comment here with any issues or comments you find, and if no one posts, it either works great or I am a total failure. :)

TwitterGrowl 0.1 (Mac OS X 10.4 or 10.5, Universal)

Looks like we're starting to get an idea together for the new Cocoa Touch platform. Stay tuned. :)

In other news, the www.whatthefuck.com relaunch has been going really well. Some old timers have come back, and the site is more stable than it ever has been. The speed is still up to snuff too, even though we're using a platform far more abstracted than the original. The original was self rolled Perl, embedded HTML, no CGI.pm, no templating, self-created cookies, the whole shebang. The new system is using the Catalyst Framework as the core, with DBIx::Class for database abstraction, Template Toolkit for the viewable HTML and XML, and utility processes using the Moose OO Framework for Perl. I'm quite happy with the result, with the manageability and maintainability, and the speed I get through FastCGI on Apache.

I've started doing a call out to users on new features and suggestions. From the latest list I put out, I think that after the first half is complete, I'll be ready to do a full launch of the site and start getting the word out.

Oh, and I fixed commenting on the blog. Sorry about that.

For those who managed to miss it, Perl 5.10.0 was released to CPAN today. This is going to be so cool.

If anything, Moose is now faster, and defined-or is awesome.

Brad Fitzpatrick posted a blurb on his blog about qpsmtpd, a SMTP server written in Perl. The whole server was designed to be controlled by plugins, so by itself, it answers connections and speaks the whole SMTP process. Each step has various hooks that can call out to different plugins, from the moment you connect, through the data stream, and through disconnect.

The cool thing about this is that you can inject certain checks to each part of the process without dealing with esoteric config files like the ones found in sendmail and postfix. With simple, short plugins with cached perl instances, you can intercept common spam tricks and viruses before it even touches your filesystem. That is cool. I played with it for a bit last night, and it looks really good. It's shockingly fast, and supports either prefork or Danga::Socket to spawn.

It appears that the primary purpose of the MTA is to act as a front end to a bigger MTA such as qmail or postfix. This is fine and dandy, but I was hoping to integrate it as a full delivery agent. There isn't much there to "seal the deal", as you can follow a mail all the way through, and then it either wants to deliver it to another SMTP server or inject it directly into another MTA's spool. There is a 'Maildir' plugin, but it will output to one Maildir. Yes. One Maildir. It seems to be best suited for a spamtrap or honeypot.

So, for anyone who is interested, I put together a quick and dirty hack based on the Maildir plugin. It requires a SQL database to check domains and users against, but you could comment that piece out if you didn't care. It still needs some work, it was created only so I can start playing with it. You have been warned. :)

The PostgreSQL Schema:
There's nothing special here, so you could use MySQL or SQLite pretty easily.

create table public.domain(
"domain_id" int4 not null default nextval('domain_domain_id_seq'::regclass) ,
"is_active" int2 not null default 1 ,
"domain_name" varchar(255) not null 
)
 WITHOUT OIDS;
ALTER table "public"."domain" OWNER TO "pgsql";
alter table "public"."domain" add primary key(domain_id);

create table public."user"(
"user_id" int4 not null default nextval('user_user_id_seq'::regclass) ,
"domain_id" int4 not null ,
"can_receive" int2 not null , 
"can_login" int2 not null ,
"username" varchar(255) not null 
)
 WITHOUT OIDS;
ALTER table "public"."user" OWNER TO "pgsql";
alter table "public"."user" add primary key(user_id);

The Perl plugin:
Stick this in plugins/queue/maildir-domain. This requires the Clone module from CPAN.

use File::Path qw(mkpath);
use Sys::Hostname qw(hostname);
use Time::HiRes qw(gettimeofday);
use Clone qw(clone);
use DBI;

sub register {
	my ( $self, $qp, @args ) = @_;
	
	$self->{_mdConfig} = {
		maildirPath	=> '/home/smtpd/mail',
		dbConnect	=> 'DBI:Pg:database=mail',
		dbUser		=> 'pgsql',
		dbPass		=> ''
	};

	my $hostname = ( hostname =~ m/([\w\._\-]+)/ )[0];
	$self->{_hostname} = $hostname;
	
}

sub getDatabaseConnection {
	my ( $self ) = @_;
	
	# Cache a database handle
	unless (defined $self->{_dbh}) {
		$self->{_dbh} = DBI->connect(
			$self->{_mdConfig}->{dbConnect},
			$self->{_mdConfig}->{dbUser},
			$self->{_mdConfig}->{dbPass}
		);
	}
	
	# Cache used queries
	unless (defined $self->{_queries}) {
		$self->{_queries} = {};
		
		# Get a domain 
		$self->{_queries}->{domain} = $self->{_dbh}->prepare_cached(q{
			SELECT domain_id, domain_name
			  FROM domain
			 WHERE domain_name = ?
		});
		$self->{_queries}->{user} = $self->{_dbh}->prepare_cached(q{
			SELECT user_id
			  FROM "user"
			 WHERE domain_id = ? AND username = ? AND can_receive = 1
		});
	}
}

# Use the stoored/prepared query to grab a domain record from a domain name
sub getDomainByName {
	my ( $self, $domainName ) = @_;
	
	my $result = {};
	$self->{_queries}->{domain}->execute($domainName);
	while (my $rec = $self->{_queries}->{domain}->fetchrow_hashref) {
		$result->{domainId} = $rec->{domain_id};
		$result->{domainName} = $rec->{domain_name};
	}
	
	if (defined $result->{domainId}) {
		return $result;
	}
	
	return undef;
}

# Use the stored/prepared query to grab a user based on domain ID and username
sub getUserByName {
	my ( $self, $domainId, $username ) = @_;
	
	my $userId;
	$self->{_queries}->{user}->execute($domainId, $username);
	while (my $rec = $self->{_queries}->{user}->fetchrow_hashref) {
		$userId = $rec->{user_id};
	}
	
	return $userId;
}

sub hook_queue {
	my ( $self, $masterTransaction ) = @_;
	
	my $maildirCounter = 0;
	my $successCount = 0;
	$self->getDatabaseConnection();
	
	foreach my $recipient ($masterTransaction->recipients()) {
		
		# Get info from database
		my $domainRef = $self->getDomainByName($recipient->host);
		next unless (defined $domainRef);
		my $userId = $self->getUserByName($domainRef->{domainId}, $recipient->user);
		next unless (defined $userId and $userId > 0);
		
		# Get maildir, verify it exists
		my $maildir = join("/",
			$self->{_mdConfig}->{maildirPath},
			$recipient->host,
			$recipient->user
		);
		foreach (qw/cur tmp new/) {
			my $dir = $maildir . "/" . $_;
			mkpath($dir, 0, 0700) unless (-e $dir);
		}
		
		# Get a copy of the transaction for this recipient
		my $transaction = clone($masterTransaction);
		
		# Parse time
		my ( $time, $microseconds ) = gettimeofday;
		$time = ( $time =~ m/(\d+)/ )[0];
		$microseconds =~ s/\D//g;
		
		# Generate identifier
		my $unique = "P$$" . "M$microseconds" . "Q" . $maildirCounter++;
		my $file = join(".", $time, $unique, $self->{_hostname});

		$transaction->header->add( 'Delivered-To', $recipient->address, 0 );

		# Create new message in Maildir
		open( MF, ">$maildir/tmp/$file" )
		  or $self->log( LOGWARN, "could not open $maildir/tmp/$file: $!" ),
		  return ( DECLINED, "queue error (open)" );

		$transaction->header->print( \*MF );
		$transaction->body_resetpos;
		while ( my $line = $transaction->body_getline ) {
			print MF $line;
		}

		close(MF)
		  or $self->log( LOGWARN, "could not close $maildir/tmp/$file: $!" )
		  	 and return ( DECLINED, "queue error (close)" );

		# Associate as new message
		link("$maildir/tmp/$file", "$maildir/new/$file")
		  or $self->log( LOGWARN, "could not link $maildir/tmp/$file to $maildir/new/$file: $!" )
		  	 and return ( DECLINED, "queue error (link)" );
		
		# Remove from /tmp
		unlink "$maildir/tmp/$file";
		$successCount++;
	}

	my $messageId = ($masterTransaction->header->get('Message-Id') or '');
	$messageId =~ s/[\r\n].*//s;
	
	# If we delivered to anyone, call it good!
	if ($successCount > 0) {
		return ( OK, "Queued! $messageId" );
	} else {
		return ( DENY );
	}
}

I upgraded the blog to the newest Movable Type 4 beta, and you're seeing it live now. Let me know if you run into any difficulties.

My first impression is "oh my god, this is really nice!" I've been shopping around for a new blogging tool as MT has been irritating me in a lot of ways, especially since WordPress has been gaining momentum. I couldn't bring myself to install a PHP blog tool, not to mention the fact that this server has a legacy PHP4 installation due to once of the web sites hosted on the machine.

The spam controls are now included out of the box, the editing system is now completely sane, and the whole thing is much easier to navigate. More later, as I have a chance to play with it.

Not sure what was going on with me this morning, but I think I was trying to pick a fight in #catalyst this morning. I was at work all of about ten minutes before I asked a simple question, amounting to: "Are there any plans to bring Catalyst to a wider audience?" Confusion followed, followed by a decent discussion. The general point I was trying to make is simple. If you ask someone who works near web development if they've ever heard of Ruby on Rails, chances are, they have. If you ask them about Catalyst, you'll usually end up with a shrug. Those who are willing to listen further generally stop listening as soon as you say "framework for perl".

There is a stigma attached to perl for various reasons -- people view perl as unreadable, or slow, or hard to develop in on a large scale. Most people have a perception of perl that dates back to the late 90's, coding against cgi-lib.pl or CGI.pm. Hell, back then, I was rolling my own CGI scripts so they were 'lightweight'. God forbid anyone look at my code from back then. Perl was pushed as a rapid-development, but hacky language, and most people produced a lot of code that looked like line noise or otherwise, and that is what is burned into people's minds. There is an unfortunate percentage of the current perl population that still writes kludgy one script wonders and calls it a web application, and that's also bad for the community, in my opinion. Frankly, a language that holds contests on who can make the most unreadable code, or who can fit the most into one line, generally deserves that stigma.

The other side of the coin is Perl 6, the upcoming complete rewrite of Perl. If you Google for perl 6, you'll find articles talking about its impending release dating back to 2001, yet we still don't have a final revision. The bytecode interpreter is far from complete, and the closest thing there is to a working, usable interpreter is written in Haskell, another higher level language. The whole thing feels kludgy and incomplete to an outsider, and that's probably because compared to other modern object oriented languages like Ruby and Python, Perl 6 is kludgy and incomplete.

So, disillusionment and wankery abounds when it comes to perl, and a lot of it is deserved. It's a perception problem, and one that is almost impossible to solve. But, hell, I'm stubborn, and other people have made far inferior products rise from the ashes.

Those who remember a few years back realize that not many people were aware of or used Ruby before Rails came out, and a lot of old perl hands fell right into Ruby because of its similarities. I find Ruby to go against 'what I mean', so it's a reach when I start pounding out any code. I was hoping to find an alternative when I stumbled upon Catalyst, and I've been hooked since. I think quite a few people would see the same thing, if they only knew what was there.

My plea to #catalyst was simple. Catalyst is a diamond in the rough, a flexible, fast, and powerful web application framework that is very easy to use once you get over the first learning curve. It is an excellent demonstration of modern OO perl development, despite any flaws or issues that still remain in the framework. The problem, however, is there is very little marketing or outreach happening, and without any kind of constant public opinion, the userbase stagnates and eventually shrinks, leaving frameworks like Catalyst to die on the vine. Someone within the Catalyst community, or even tangentally related to the Catalyst community needs to find a way to bring people back into the perl fold by making them aware of the strengths of the framework.

A few things would need to happen, in no particular order. Note that this is only my opinion, and I'd be happy to be told I was wrong.

• They want their wiki moved from Trac to MojoMojo before they do any major wiki work. Fine, I'll give that one.


• Get some nice looking skeletons into the default Catalyst project template. Make it look pretty. For some reason, this actually works, and makes people want to do the same.


• Catalyst's primary development happens on mailing lists and on IRC. This should be outlined somehow within the wiki so creep doesn't occur.


• The primary focus of catalystframework.org should be marketing and outreach -- a bulletproof example of the stability and scalability of an application written in the framework, with easy to deal with tutorials and a complete set of hyperlinked documentation. Links should be given to external sites who use the framework, as well as third party Catalyst tutorial sites.


• There needs to be a third party tutorial and development site! You can toot your own horn, but it's hard to convince people that what you have works really well unless they can see other people getting together and doing things really well. The real championing of OS X doesn't happen on apple.com, so the real championing of Catalyst shouldn't be on catalystframework.org. Luckily, it's not, as the site is a bore and makes the project look dead, as it is hardly updated.


• Along with the third party site comes some community support. Bring people together. Show people to IRC, bring people to a forum. Get a forum of a couple of hundred and hold CatalystConf or something similar. The key to getting the product into the eyes of many is to show people that there is an active following behind it. Lesser languages and frameworks do it and give people confidence to continue developing. Check out Lasso and REALbasic if you have no idea what I mean.


• Tell people you're using Catalyst. If you run a site that is truly great, and starting to get public attention, mention the framework. The Rails apps are doing it. The PHP people just try to hide that it's PHP behind it.

I'm not talking about zealotry, here. I'm not talking about Catalyst being the best or brightest, or that perl is the best or the brightest. I'm not talking about how Ruby, Python, or PHP suck (this time). I'm only talking about bringing a really great language and a fantastic framework into the foreground so people are at least aware of the options before they talk some smack.

Or maybe, just maybe, I'm completely full of it.