Recently in Internet Category

At some point, I really have to write about the new whatthefuck.com mail architecture. It is made up of some pretty epic hacks, but the new system is one hundred times more reliable and versatile than the old system.

Okay, I suppose that isn't saying much.

I do have to say this. I started out building on top of a courier/postfix based system, and ended up rolling my own on top of qpsmtpd, and will be writing a custom IMAP daemon for the newly-created system. The first version is very basic, but there are so many neat things I can do with it.

The new server with a terabyte and a half of space is heading to the data center this week, but if this works how I hope, there may be a few more servers heading over there soon enough. FreeBSD and ZFS is pretty much admin-heaven. PostgreSQL is a wonderful thing as well.

Why does Twitter suck?
This morning, my manager asked if I used Twitter at all. While I have an account, and I have a bunch of 'Friends' on there, my attention span for the service has never really lasted much longer than 2-3 days. Every time someone asks me about it, I grab a client or two, try things out, and realize yet again how many things really annoy me about the service. The top two were how much of a pain it was to post, and what a bigger pain it was to keep a feed going. The best I could find for OS X was Twitterrific, and it was just counterintuitive. The free version would put ads in, the 'auto reappear' never did, and it would just silently fail half the time and not post anything new for hours. Awesomesauce.

Granted, a few things have been fixed. The SMS posting service makes it really easy to actually post remotely, as none of the S60 clients are very good, and the Java clients really suck. They don't seem to be down as much, and the speed isn't terrible. Rumor has it, they're getting rid of Ruby on Rails, which just makes me jump with glee.

Since then, for posting, someone put together an AppleScript called Tweet, which allows you to easily post to Twitter through QuickSilver. It works well for me, I just hit Command-Space, hit period, type my tweet, tab, then tweet it. It sounds complicated, but it's really easy to deal with. I like it, and it was easy as hell to do.

Then, there's the feed problem. RSS is too slow, I don't want it cluttering up Google Reader. Twitterrific still isn't working for me, as pretty as it is. There are two Dashboard modules, and neither of them work well, and silently fail at that. What I really wanted was something that would just post tweets to Growl and be done with it. I found something, but it was a Ruby script with manual configuration and little extensibility. I didn't want to screw with it, so off to /dev/null it went.

Hey, there's a point.

In the end, I wrote a perl script called TwitterGrowl to do exactly what I want it to do. To make life easier, it relies on the Twitter login information in your keychain to log in, and prompts you to create one if it doesn't find anything. It reports when there's a login failure, or Twitter goes into Suckfest, or when a system maintenance issue is posted. Better yet, I packaged it up into an easy to use, double clickable application. Pop it into your Applications folder, drag it to your Login Items, and it'll go into the background and sign in when you log in. Easy as pie. All of the required modules are in the application package, and you can feel free to browse the source by viewing the package contents and heading into the Resources folder.

So, now that I have a steady "works for me", would anyone else like to give it a shot and see if anything breaks? Comment here with any issues or comments you find, and if no one posts, it either works great or I am a total failure. :)

TwitterGrowl 0.1 (Mac OS X 10.4 or 10.5, Universal)

I have a great idea for a web application, something nice and bubbly for people to hook into.

A random number generator.

You get a certain number of uses per month. We hire three or four people to roll dice, and input the result into the system. Each API call shifts the next value off the top.

I'm going to be rich.

Brad Fitzpatrick posted a blurb on his blog about qpsmtpd, a SMTP server written in Perl. The whole server was designed to be controlled by plugins, so by itself, it answers connections and speaks the whole SMTP process. Each step has various hooks that can call out to different plugins, from the moment you connect, through the data stream, and through disconnect.

The cool thing about this is that you can inject certain checks to each part of the process without dealing with esoteric config files like the ones found in sendmail and postfix. With simple, short plugins with cached perl instances, you can intercept common spam tricks and viruses before it even touches your filesystem. That is cool. I played with it for a bit last night, and it looks really good. It's shockingly fast, and supports either prefork or Danga::Socket to spawn.

It appears that the primary purpose of the MTA is to act as a front end to a bigger MTA such as qmail or postfix. This is fine and dandy, but I was hoping to integrate it as a full delivery agent. There isn't much there to "seal the deal", as you can follow a mail all the way through, and then it either wants to deliver it to another SMTP server or inject it directly into another MTA's spool. There is a 'Maildir' plugin, but it will output to one Maildir. Yes. One Maildir. It seems to be best suited for a spamtrap or honeypot.

So, for anyone who is interested, I put together a quick and dirty hack based on the Maildir plugin. It requires a SQL database to check domains and users against, but you could comment that piece out if you didn't care. It still needs some work, it was created only so I can start playing with it. You have been warned. :)

The PostgreSQL Schema:
There's nothing special here, so you could use MySQL or SQLite pretty easily.

create table public.domain(
"domain_id" int4 not null default nextval('domain_domain_id_seq'::regclass) ,
"is_active" int2 not null default 1 ,
"domain_name" varchar(255) not null 
)
 WITHOUT OIDS;
ALTER table "public"."domain" OWNER TO "pgsql";
alter table "public"."domain" add primary key(domain_id);

create table public."user"(
"user_id" int4 not null default nextval('user_user_id_seq'::regclass) ,
"domain_id" int4 not null ,
"can_receive" int2 not null , 
"can_login" int2 not null ,
"username" varchar(255) not null 
)
 WITHOUT OIDS;
ALTER table "public"."user" OWNER TO "pgsql";
alter table "public"."user" add primary key(user_id);

The Perl plugin:
Stick this in plugins/queue/maildir-domain. This requires the Clone module from CPAN.

use File::Path qw(mkpath);
use Sys::Hostname qw(hostname);
use Time::HiRes qw(gettimeofday);
use Clone qw(clone);
use DBI;

sub register {
	my ( $self, $qp, @args ) = @_;
	
	$self->{_mdConfig} = {
		maildirPath	=> '/home/smtpd/mail',
		dbConnect	=> 'DBI:Pg:database=mail',
		dbUser		=> 'pgsql',
		dbPass		=> ''
	};

	my $hostname = ( hostname =~ m/([\w\._\-]+)/ )[0];
	$self->{_hostname} = $hostname;
	
}

sub getDatabaseConnection {
	my ( $self ) = @_;
	
	# Cache a database handle
	unless (defined $self->{_dbh}) {
		$self->{_dbh} = DBI->connect(
			$self->{_mdConfig}->{dbConnect},
			$self->{_mdConfig}->{dbUser},
			$self->{_mdConfig}->{dbPass}
		);
	}
	
	# Cache used queries
	unless (defined $self->{_queries}) {
		$self->{_queries} = {};
		
		# Get a domain 
		$self->{_queries}->{domain} = $self->{_dbh}->prepare_cached(q{
			SELECT domain_id, domain_name
			  FROM domain
			 WHERE domain_name = ?
		});
		$self->{_queries}->{user} = $self->{_dbh}->prepare_cached(q{
			SELECT user_id
			  FROM "user"
			 WHERE domain_id = ? AND username = ? AND can_receive = 1
		});
	}
}

# Use the stoored/prepared query to grab a domain record from a domain name
sub getDomainByName {
	my ( $self, $domainName ) = @_;
	
	my $result = {};
	$self->{_queries}->{domain}->execute($domainName);
	while (my $rec = $self->{_queries}->{domain}->fetchrow_hashref) {
		$result->{domainId} = $rec->{domain_id};
		$result->{domainName} = $rec->{domain_name};
	}
	
	if (defined $result->{domainId}) {
		return $result;
	}
	
	return undef;
}

# Use the stored/prepared query to grab a user based on domain ID and username
sub getUserByName {
	my ( $self, $domainId, $username ) = @_;
	
	my $userId;
	$self->{_queries}->{user}->execute($domainId, $username);
	while (my $rec = $self->{_queries}->{user}->fetchrow_hashref) {
		$userId = $rec->{user_id};
	}
	
	return $userId;
}

sub hook_queue {
	my ( $self, $masterTransaction ) = @_;
	
	my $maildirCounter = 0;
	my $successCount = 0;
	$self->getDatabaseConnection();
	
	foreach my $recipient ($masterTransaction->recipients()) {
		
		# Get info from database
		my $domainRef = $self->getDomainByName($recipient->host);
		next unless (defined $domainRef);
		my $userId = $self->getUserByName($domainRef->{domainId}, $recipient->user);
		next unless (defined $userId and $userId > 0);
		
		# Get maildir, verify it exists
		my $maildir = join("/",
			$self->{_mdConfig}->{maildirPath},
			$recipient->host,
			$recipient->user
		);
		foreach (qw/cur tmp new/) {
			my $dir = $maildir . "/" . $_;
			mkpath($dir, 0, 0700) unless (-e $dir);
		}
		
		# Get a copy of the transaction for this recipient
		my $transaction = clone($masterTransaction);
		
		# Parse time
		my ( $time, $microseconds ) = gettimeofday;
		$time = ( $time =~ m/(\d+)/ )[0];
		$microseconds =~ s/\D//g;
		
		# Generate identifier
		my $unique = "P$$" . "M$microseconds" . "Q" . $maildirCounter++;
		my $file = join(".", $time, $unique, $self->{_hostname});

		$transaction->header->add( 'Delivered-To', $recipient->address, 0 );

		# Create new message in Maildir
		open( MF, ">$maildir/tmp/$file" )
		  or $self->log( LOGWARN, "could not open $maildir/tmp/$file: $!" ),
		  return ( DECLINED, "queue error (open)" );

		$transaction->header->print( \*MF );
		$transaction->body_resetpos;
		while ( my $line = $transaction->body_getline ) {
			print MF $line;
		}

		close(MF)
		  or $self->log( LOGWARN, "could not close $maildir/tmp/$file: $!" )
		  	 and return ( DECLINED, "queue error (close)" );

		# Associate as new message
		link("$maildir/tmp/$file", "$maildir/new/$file")
		  or $self->log( LOGWARN, "could not link $maildir/tmp/$file to $maildir/new/$file: $!" )
		  	 and return ( DECLINED, "queue error (link)" );
		
		# Remove from /tmp
		unlink "$maildir/tmp/$file";
		$successCount++;
	}

	my $messageId = ($masterTransaction->header->get('Message-Id') or '');
	$messageId =~ s/[\r\n].*//s;
	
	# If we delivered to anyone, call it good!
	if ($successCount > 0) {
		return ( OK, "Queued! $messageId" );
	} else {
		return ( DENY );
	}
}

Incidentally, I just got an interesting error when posting this entry using Movable Type 4.0RC1.

I found the solution on a Japanese blog translated to English. If you run into this, remove the ExtensibleArchives directory in your plugins directory, and try again. Everything will work at that point. I didn't quite understand the rest of it. ;)

I upgraded the blog to the newest Movable Type 4 beta, and you're seeing it live now. Let me know if you run into any difficulties.

My first impression is "oh my god, this is really nice!" I've been shopping around for a new blogging tool as MT has been irritating me in a lot of ways, especially since WordPress has been gaining momentum. I couldn't bring myself to install a PHP blog tool, not to mention the fact that this server has a legacy PHP4 installation due to once of the web sites hosted on the machine.

The spam controls are now included out of the box, the editing system is now completely sane, and the whole thing is much easier to navigate. More later, as I have a chance to play with it.

Irritation: When your colocation provider shuts off port 22 for the whole network, temporarily (of undisclosed time), while you're in the middle of working.
Serious Irritation: Knowing you could have installed your firewall/VPN box last week.
Denouement: Realizing you aren't paying for it, and you have no room to really say a thing.

Everyone needs to watch this one: Swift

If you like Safari's renderer on OS X, whether you use Safari, Shiira, OmniWeb, or something else, this is something to watch. This is the open source WebKit, the foundation for Safari, ported to Windows and given a simple interface. My assumption is this is going to build out quite a bit, and should be cool to watch.

Now to see if someone will send it over to GTK2. :)