Recently in Development Category

If you're reading from the main site, don't be too alarmed by the Google AdSense ad. It's mostly there for testing, but it's a little bit of an experiment as well.

I've been involved with the development, support, and administration of running various web sites for the last ten years or so, and yet, I've never been a part of a site that has run any kind of advertising campaign. Doing things for yourself changes some of your views on internet advertising, as it's one of the few ways you can reliably generate any income. People, for the most part, do not want to pay for anything they do online.

Most of you know that I run www.whatthefuck.com, a free content and networking site that was content and networking before it was cool. I had two major issues during the life of this site: first, a lack of time to properly maintain it, and second, a lack of ability to monetize the domain properly. I created the site with a friend when we were both just out of high school, and it was created in the same line of thinking as the old school BBSes that we used to run and sign into daily. Because of that, it never really occurred to use to try to make money off of it, and when it finally did, the economy was running down the drain.

Over the last year, I've been rebuilding the site piece by piece on top of the fantastic platform of Perl, PostgreSQL, Catalyst, and DBIx::Class. It "relaunched" last November, very quietly, to see who would notice. Those who were still sticking around noticed immediately, as the site worked, worked well, and was still speedy. The first mission was accomplished, and the last year has been me working on it in my spare time to restore the feature set that was original envisioned back in 1999.

The difference between 1999 and 2008 in terms of the web is like comparing video games from around 1994 to now. In 1990, a sole shareware developer could crank out fun, easy to play games as shareware for $20 and make a killing. Now, you need a few million in the bank, and a team of artists, designers, modelers, and marketers to publish a video game.. and there's no guarantee that it'll be a hit. You can't throw together some basic HTML, add a few tables and server side form processing, and call it a success anymore. You need transitions, glass effects, AJAX, Web 2.0, blah, blah, blah.

(Sometimes, this goes overboard, like in the case of Ruby on Rails developers. This is why a simple service like Twitter is still having issues -- overdoing something simple, creating a platform you just can't extend.)

To bring this long story back to the ground, the next big push out from wtf is the free e-mail support that users enjoyed back when whatthefuck.com launched. Back then, it was essentially three things: a mailbox viewer, a simple mail viewer (no MIME! wtf?!), and a message composer. Yeah, there were mail folders. Sure, later on, we added an address book, but that's as fancy as it got. Eventually, the whole system collapsed under failed hardware and too little space. Now, no one is going to care about a mail service unless it -- at the very least -- does real time folder management, tagging, dynamic resizing, MIME decoding, spam filtering, and live updates. That, and 20mb per user doesn't cut it anymore, but there's no way we're going to give out 5gb at this point. Even with a minor increase in allowed space, I have to buy a lot of hardware to support this endeavor, so I have to come up with a revenue source.

Enter advertising networks, and this morning's research.

I've hit a bit of a breaking point, though. I tried to put Google SiteSearch on wtf a few months ago, but Google blocks wtf from making any requests. Apparently, we're on their blocked/forbidden list for a violation of their policy. The only thing I see in there that could potentially be a policy violation is 'excessive swearing' due to the domain name, but that kinda sucks. I've seen sites with Google ads that were doing far more subversive things than our forum.

I just wanted to see if I could get an ad up anywhere else. Turns out, I can. Now, I go look for another advertising network that doesn't have adult ads, while still allowing a domain like whatthefuck.com.

Some of you know that I'm a hackintosh user. I've been an Apple fan for a long time, but nothing in their lineup currently works for what I want out of a machine. My last Apple machine was a black MacBook with the original Intel Core Duo. It went in for service three times, and finally, Apple just denied anything was wrong with it. I ended up selling that machine, and the hard drive that was originally in it died a week later.

Huh.

Right now, I use a Lenovo/IBM ThinkPad T61 14" Widescreen edition, with a Core 2 Duo at 2.2GHz, 2gb RAM, a 100gb drive, and DVD-RW DL drive. The screen resolution is a comfortable 1440x900, I have some wireless, and life is good. I'm also running OS X 10.5, licensed but violating EULA, because it's one of the greatest operating systems we currently have available. There are a few little issues with running Leopard on this machine. Most things work, but I don't currently have a driver for the built in ethernet, the machine won't sleep if I have bluetooth enabled, the machine won't wake up if I enable the PC card slots, I can't control the brightness of the LCD, and when the laptop runs out of battery, it shuts off instead of going to sleep.

There are a few people working on two of the issues, namely the brightness issue with the Intel X3100 graphics controller, and a driver for the Intel 82566M Gigabit Ethernet controller. No sense in duplicating their efforts. The only kext I've ever written was a small driver to enable the tablet serial port on the X61 tablet that I had before this, so people could use TabletMagic and get full tablet functionality in OS X. Ethernet driver creator, I am not.

But, hey, I give you one little turd to make your life easier, and you may even find it useful if you're on a real Mac. This little package, 'sleep_monitor', installs a LaunchDaemon and a binary on your machine to keep track of how much battery you have left and put your machine to sleep at a certain threshold. It's a simple idea, but most Mac authors would charge $20 for the privilege. I give you it for free, but you don't get a GUI. Sorry.

When you install the package, it immediately starts. It will wait until you have 4% of your battery remaining, and puts your machine to sleep. On my T61 with the 4 cell battery, that usually kicks me into sleep when I have about 5 minutes of use left in the menu bar, and that will give me many hours of sleep to find an outlet to charge things up. Want to change the threshold? Edit /Library/LaunchDaemons/com.abstractwankery.sleep_monitor.plist and change the second ProgramArgument to the percentage you'd like to kill out at. Have an 8 cell battery? 2-3% is fine. Have a nearly dead battery? Maybe 15% is more your thing. Leave the % sign out of it, it will only cause problems.

SleepMonitor 0.1
Mac OS X 10.4, 10.5

Why does Twitter suck?
This morning, my manager asked if I used Twitter at all. While I have an account, and I have a bunch of 'Friends' on there, my attention span for the service has never really lasted much longer than 2-3 days. Every time someone asks me about it, I grab a client or two, try things out, and realize yet again how many things really annoy me about the service. The top two were how much of a pain it was to post, and what a bigger pain it was to keep a feed going. The best I could find for OS X was Twitterrific, and it was just counterintuitive. The free version would put ads in, the 'auto reappear' never did, and it would just silently fail half the time and not post anything new for hours. Awesomesauce.

Granted, a few things have been fixed. The SMS posting service makes it really easy to actually post remotely, as none of the S60 clients are very good, and the Java clients really suck. They don't seem to be down as much, and the speed isn't terrible. Rumor has it, they're getting rid of Ruby on Rails, which just makes me jump with glee.

Since then, for posting, someone put together an AppleScript called Tweet, which allows you to easily post to Twitter through QuickSilver. It works well for me, I just hit Command-Space, hit period, type my tweet, tab, then tweet it. It sounds complicated, but it's really easy to deal with. I like it, and it was easy as hell to do.

Then, there's the feed problem. RSS is too slow, I don't want it cluttering up Google Reader. Twitterrific still isn't working for me, as pretty as it is. There are two Dashboard modules, and neither of them work well, and silently fail at that. What I really wanted was something that would just post tweets to Growl and be done with it. I found something, but it was a Ruby script with manual configuration and little extensibility. I didn't want to screw with it, so off to /dev/null it went.

Hey, there's a point.

In the end, I wrote a perl script called TwitterGrowl to do exactly what I want it to do. To make life easier, it relies on the Twitter login information in your keychain to log in, and prompts you to create one if it doesn't find anything. It reports when there's a login failure, or Twitter goes into Suckfest, or when a system maintenance issue is posted. Better yet, I packaged it up into an easy to use, double clickable application. Pop it into your Applications folder, drag it to your Login Items, and it'll go into the background and sign in when you log in. Easy as pie. All of the required modules are in the application package, and you can feel free to browse the source by viewing the package contents and heading into the Resources folder.

So, now that I have a steady "works for me", would anyone else like to give it a shot and see if anything breaks? Comment here with any issues or comments you find, and if no one posts, it either works great or I am a total failure. :)

TwitterGrowl 0.1 (Mac OS X 10.4 or 10.5, Universal)

Looks like we're starting to get an idea together for the new Cocoa Touch platform. Stay tuned. :)

In other news, the www.whatthefuck.com relaunch has been going really well. Some old timers have come back, and the site is more stable than it ever has been. The speed is still up to snuff too, even though we're using a platform far more abstracted than the original. The original was self rolled Perl, embedded HTML, no CGI.pm, no templating, self-created cookies, the whole shebang. The new system is using the Catalyst Framework as the core, with DBIx::Class for database abstraction, Template Toolkit for the viewable HTML and XML, and utility processes using the Moose OO Framework for Perl. I'm quite happy with the result, with the manageability and maintainability, and the speed I get through FastCGI on Apache.

I've started doing a call out to users on new features and suggestions. From the latest list I put out, I think that after the first half is complete, I'll be ready to do a full launch of the site and start getting the word out.

Oh, and I fixed commenting on the blog. Sorry about that.

Yesterday morning, Apple introduced what they're calling the Cocoa Touch SDK, otherwise known as the iPhone SDK. This development toolkit allows developers familiar with Objective-C the ability to create applications for the iPhone or iPod Touch platform, and provides a distribution method through Apple's App Store.

The development kit includes beta of the updated XCode, supporting "iPhone Application" as a development target, supporting libraries and frameworks, documentation, and "Aspen Simulator", a virtual iPhone for testing purposes. I played around with the new development environment for about 30 minutes, successfully creating a little "Hello, world!" application, and then toyed around with the simulator for a while. I'm surprised at how complete the simulator is, especially compared to the virtual device given with the Maemo development toolkit (the underlying platform for Nokia's Internet Tablet devices, the 770, N800, and N810). You have full access to Mobile Safari for testing web applications, as well as the Photos application and the Contacts application, presumably for developers to be able to test integration with core Touch services. Through option-clicking the interface, you're able to emulate the pinching feature of the multi-touch interface, and really able to exploit many of the features without using the actual device. I think it provides a really great interface to piece together a good application, requiring an actual device only later on in the development process, so you don't mar a phone from the getgo. Quite a bit easier than Symbian or Maemo development, and roughly equivalent to Windows Mobile development. The only thing WinMo has on them at this point is that their interface development features within Visual Studio are far more mature -- Interface Builder is not yet available in the development kit beta. You're drawing canvasses on your own. ;)

Last night, Apple released something like 20 videos over iTunesU at the iPhone Dev Center on getting started with iPhone development. I'm hoping to start going through them this weekend, and see what I can come up with. I'm not necessarily chasing the millions in VC funding that is being offered, I just like to create cool things. I don't actually have a device yet, but this has moved me to think about picking up an iPod Touch in the next few weeks.

I still can't bring myself to leave T-Mobile to get an iPhone, and having to reunlock my device every time there's a software update really doesn't do it for me. :)

Today is a good day, as I'm slowly rolling out the (very) beta of the new www.whatthefuck.com. I'll be doing some testing on the production server today, and going to see what's broken tomorrow.

I'm actually rather lucky to have a small active userbase at this point -- it's just enough to consider it a beta pool, and let me know what's horribly broken before things go very wrong. As nothing is being overwritten, I can always roll back to original if things go awry.

Mr. Kawasaki, I hope you're right. You told me to just ship it, and I'm going to, dammit.

Brad Fitzpatrick posted a blurb on his blog about qpsmtpd, a SMTP server written in Perl. The whole server was designed to be controlled by plugins, so by itself, it answers connections and speaks the whole SMTP process. Each step has various hooks that can call out to different plugins, from the moment you connect, through the data stream, and through disconnect.

The cool thing about this is that you can inject certain checks to each part of the process without dealing with esoteric config files like the ones found in sendmail and postfix. With simple, short plugins with cached perl instances, you can intercept common spam tricks and viruses before it even touches your filesystem. That is cool. I played with it for a bit last night, and it looks really good. It's shockingly fast, and supports either prefork or Danga::Socket to spawn.

It appears that the primary purpose of the MTA is to act as a front end to a bigger MTA such as qmail or postfix. This is fine and dandy, but I was hoping to integrate it as a full delivery agent. There isn't much there to "seal the deal", as you can follow a mail all the way through, and then it either wants to deliver it to another SMTP server or inject it directly into another MTA's spool. There is a 'Maildir' plugin, but it will output to one Maildir. Yes. One Maildir. It seems to be best suited for a spamtrap or honeypot.

So, for anyone who is interested, I put together a quick and dirty hack based on the Maildir plugin. It requires a SQL database to check domains and users against, but you could comment that piece out if you didn't care. It still needs some work, it was created only so I can start playing with it. You have been warned. :)

The PostgreSQL Schema:
There's nothing special here, so you could use MySQL or SQLite pretty easily.

create table public.domain(
"domain_id" int4 not null default nextval('domain_domain_id_seq'::regclass) ,
"is_active" int2 not null default 1 ,
"domain_name" varchar(255) not null 
)
 WITHOUT OIDS;
ALTER table "public"."domain" OWNER TO "pgsql";
alter table "public"."domain" add primary key(domain_id);

create table public."user"(
"user_id" int4 not null default nextval('user_user_id_seq'::regclass) ,
"domain_id" int4 not null ,
"can_receive" int2 not null , 
"can_login" int2 not null ,
"username" varchar(255) not null 
)
 WITHOUT OIDS;
ALTER table "public"."user" OWNER TO "pgsql";
alter table "public"."user" add primary key(user_id);

The Perl plugin:
Stick this in plugins/queue/maildir-domain. This requires the Clone module from CPAN.

use File::Path qw(mkpath);
use Sys::Hostname qw(hostname);
use Time::HiRes qw(gettimeofday);
use Clone qw(clone);
use DBI;

sub register {
	my ( $self, $qp, @args ) = @_;
	
	$self->{_mdConfig} = {
		maildirPath	=> '/home/smtpd/mail',
		dbConnect	=> 'DBI:Pg:database=mail',
		dbUser		=> 'pgsql',
		dbPass		=> ''
	};

	my $hostname = ( hostname =~ m/([\w\._\-]+)/ )[0];
	$self->{_hostname} = $hostname;
	
}

sub getDatabaseConnection {
	my ( $self ) = @_;
	
	# Cache a database handle
	unless (defined $self->{_dbh}) {
		$self->{_dbh} = DBI->connect(
			$self->{_mdConfig}->{dbConnect},
			$self->{_mdConfig}->{dbUser},
			$self->{_mdConfig}->{dbPass}
		);
	}
	
	# Cache used queries
	unless (defined $self->{_queries}) {
		$self->{_queries} = {};
		
		# Get a domain 
		$self->{_queries}->{domain} = $self->{_dbh}->prepare_cached(q{
			SELECT domain_id, domain_name
			  FROM domain
			 WHERE domain_name = ?
		});
		$self->{_queries}->{user} = $self->{_dbh}->prepare_cached(q{
			SELECT user_id
			  FROM "user"
			 WHERE domain_id = ? AND username = ? AND can_receive = 1
		});
	}
}

# Use the stoored/prepared query to grab a domain record from a domain name
sub getDomainByName {
	my ( $self, $domainName ) = @_;
	
	my $result = {};
	$self->{_queries}->{domain}->execute($domainName);
	while (my $rec = $self->{_queries}->{domain}->fetchrow_hashref) {
		$result->{domainId} = $rec->{domain_id};
		$result->{domainName} = $rec->{domain_name};
	}
	
	if (defined $result->{domainId}) {
		return $result;
	}
	
	return undef;
}

# Use the stored/prepared query to grab a user based on domain ID and username
sub getUserByName {
	my ( $self, $domainId, $username ) = @_;
	
	my $userId;
	$self->{_queries}->{user}->execute($domainId, $username);
	while (my $rec = $self->{_queries}->{user}->fetchrow_hashref) {
		$userId = $rec->{user_id};
	}
	
	return $userId;
}

sub hook_queue {
	my ( $self, $masterTransaction ) = @_;
	
	my $maildirCounter = 0;
	my $successCount = 0;
	$self->getDatabaseConnection();
	
	foreach my $recipient ($masterTransaction->recipients()) {
		
		# Get info from database
		my $domainRef = $self->getDomainByName($recipient->host);
		next unless (defined $domainRef);
		my $userId = $self->getUserByName($domainRef->{domainId}, $recipient->user);
		next unless (defined $userId and $userId > 0);
		
		# Get maildir, verify it exists
		my $maildir = join("/",
			$self->{_mdConfig}->{maildirPath},
			$recipient->host,
			$recipient->user
		);
		foreach (qw/cur tmp new/) {
			my $dir = $maildir . "/" . $_;
			mkpath($dir, 0, 0700) unless (-e $dir);
		}
		
		# Get a copy of the transaction for this recipient
		my $transaction = clone($masterTransaction);
		
		# Parse time
		my ( $time, $microseconds ) = gettimeofday;
		$time = ( $time =~ m/(\d+)/ )[0];
		$microseconds =~ s/\D//g;
		
		# Generate identifier
		my $unique = "P$$" . "M$microseconds" . "Q" . $maildirCounter++;
		my $file = join(".", $time, $unique, $self->{_hostname});

		$transaction->header->add( 'Delivered-To', $recipient->address, 0 );

		# Create new message in Maildir
		open( MF, ">$maildir/tmp/$file" )
		  or $self->log( LOGWARN, "could not open $maildir/tmp/$file: $!" ),
		  return ( DECLINED, "queue error (open)" );

		$transaction->header->print( \*MF );
		$transaction->body_resetpos;
		while ( my $line = $transaction->body_getline ) {
			print MF $line;
		}

		close(MF)
		  or $self->log( LOGWARN, "could not close $maildir/tmp/$file: $!" )
		  	 and return ( DECLINED, "queue error (close)" );

		# Associate as new message
		link("$maildir/tmp/$file", "$maildir/new/$file")
		  or $self->log( LOGWARN, "could not link $maildir/tmp/$file to $maildir/new/$file: $!" )
		  	 and return ( DECLINED, "queue error (link)" );
		
		# Remove from /tmp
		unlink "$maildir/tmp/$file";
		$successCount++;
	}

	my $messageId = ($masterTransaction->header->get('Message-Id') or '');
	$messageId =~ s/[\r\n].*//s;
	
	# If we delivered to anyone, call it good!
	if ($successCount > 0) {
		return ( OK, "Queued! $messageId" );
	} else {
		return ( DENY );
	}
}

This amused the hell out of me. :)

Because you can't tell a great hacker except by working with him, hackers themselves can't tell how good they are. This is true to a degree in most fields. I've found that people who are great at something are not so much convinced of their own greatness as mystified at why everyone else seems so incompetent.