Recently in whatthefuck.com Category

At some point, I really have to write about the new whatthefuck.com mail architecture. It is made up of some pretty epic hacks, but the new system is one hundred times more reliable and versatile than the old system.

Okay, I suppose that isn't saying much.

I do have to say this. I started out building on top of a courier/postfix based system, and ended up rolling my own on top of qpsmtpd, and will be writing a custom IMAP daemon for the newly-created system. The first version is very basic, but there are so many neat things I can do with it.

The new server with a terabyte and a half of space is heading to the data center this week, but if this works how I hope, there may be a few more servers heading over there soon enough. FreeBSD and ZFS is pretty much admin-heaven. PostgreSQL is a wonderful thing as well.

If you're reading from the main site, don't be too alarmed by the Google AdSense ad. It's mostly there for testing, but it's a little bit of an experiment as well.

I've been involved with the development, support, and administration of running various web sites for the last ten years or so, and yet, I've never been a part of a site that has run any kind of advertising campaign. Doing things for yourself changes some of your views on internet advertising, as it's one of the few ways you can reliably generate any income. People, for the most part, do not want to pay for anything they do online.

Most of you know that I run www.whatthefuck.com, a free content and networking site that was content and networking before it was cool. I had two major issues during the life of this site: first, a lack of time to properly maintain it, and second, a lack of ability to monetize the domain properly. I created the site with a friend when we were both just out of high school, and it was created in the same line of thinking as the old school BBSes that we used to run and sign into daily. Because of that, it never really occurred to use to try to make money off of it, and when it finally did, the economy was running down the drain.

Over the last year, I've been rebuilding the site piece by piece on top of the fantastic platform of Perl, PostgreSQL, Catalyst, and DBIx::Class. It "relaunched" last November, very quietly, to see who would notice. Those who were still sticking around noticed immediately, as the site worked, worked well, and was still speedy. The first mission was accomplished, and the last year has been me working on it in my spare time to restore the feature set that was original envisioned back in 1999.

The difference between 1999 and 2008 in terms of the web is like comparing video games from around 1994 to now. In 1990, a sole shareware developer could crank out fun, easy to play games as shareware for $20 and make a killing. Now, you need a few million in the bank, and a team of artists, designers, modelers, and marketers to publish a video game.. and there's no guarantee that it'll be a hit. You can't throw together some basic HTML, add a few tables and server side form processing, and call it a success anymore. You need transitions, glass effects, AJAX, Web 2.0, blah, blah, blah.

(Sometimes, this goes overboard, like in the case of Ruby on Rails developers. This is why a simple service like Twitter is still having issues -- overdoing something simple, creating a platform you just can't extend.)

To bring this long story back to the ground, the next big push out from wtf is the free e-mail support that users enjoyed back when whatthefuck.com launched. Back then, it was essentially three things: a mailbox viewer, a simple mail viewer (no MIME! wtf?!), and a message composer. Yeah, there were mail folders. Sure, later on, we added an address book, but that's as fancy as it got. Eventually, the whole system collapsed under failed hardware and too little space. Now, no one is going to care about a mail service unless it -- at the very least -- does real time folder management, tagging, dynamic resizing, MIME decoding, spam filtering, and live updates. That, and 20mb per user doesn't cut it anymore, but there's no way we're going to give out 5gb at this point. Even with a minor increase in allowed space, I have to buy a lot of hardware to support this endeavor, so I have to come up with a revenue source.

Enter advertising networks, and this morning's research.

I've hit a bit of a breaking point, though. I tried to put Google SiteSearch on wtf a few months ago, but Google blocks wtf from making any requests. Apparently, we're on their blocked/forbidden list for a violation of their policy. The only thing I see in there that could potentially be a policy violation is 'excessive swearing' due to the domain name, but that kinda sucks. I've seen sites with Google ads that were doing far more subversive things than our forum.

I just wanted to see if I could get an ad up anywhere else. Turns out, I can. Now, I go look for another advertising network that doesn't have adult ads, while still allowing a domain like whatthefuck.com.

Looks like we're starting to get an idea together for the new Cocoa Touch platform. Stay tuned. :)

In other news, the www.whatthefuck.com relaunch has been going really well. Some old timers have come back, and the site is more stable than it ever has been. The speed is still up to snuff too, even though we're using a platform far more abstracted than the original. The original was self rolled Perl, embedded HTML, no CGI.pm, no templating, self-created cookies, the whole shebang. The new system is using the Catalyst Framework as the core, with DBIx::Class for database abstraction, Template Toolkit for the viewable HTML and XML, and utility processes using the Moose OO Framework for Perl. I'm quite happy with the result, with the manageability and maintainability, and the speed I get through FastCGI on Apache.

I've started doing a call out to users on new features and suggestions. From the latest list I put out, I think that after the first half is complete, I'll be ready to do a full launch of the site and start getting the word out.

Oh, and I fixed commenting on the blog. Sorry about that.

Tuesday morning, I took whatthefuck.com down for an hour and moved everything to the new 4.0 release of the site. You cannot undersatnd how happy this makes me, it's been something I've been wanting and trying to do for about four years now.

The original www.whatthefuck.com is a series of CGI scripts strung together in a semi-cohesive mess of perl with inline HTML. The main update that the old system got was moving from cascading flat files to MySQL, and it was barely relational. The new web site is the culmination of many things I've learned since October of 1999. Some of it is a bit rushed, and I get to go in and clean up now, but it works, and the few failures I've seen have been quickly patched. I need to go through some security checks still, but things are "looking good".

The new site is still written in Perl, but utilizes the Catalyst MVC Web Application Framework. I've switched to PostgreSQL for the database, and DBIx::Class as the perl interface to it. The front end templates are all rendered using the Template Toolkit classes, and for those of you writing Perl web scripts without it, you're seriously missing out. This combination allows me the combined luxuries of using CPAN with a solid rapid application development framework, and I couldn't be happier.

More later, perhaps.

Today is a good day, as I'm slowly rolling out the (very) beta of the new www.whatthefuck.com. I'll be doing some testing on the production server today, and going to see what's broken tomorrow.

I'm actually rather lucky to have a small active userbase at this point -- it's just enough to consider it a beta pool, and let me know what's horribly broken before things go very wrong. As nothing is being overwritten, I can always roll back to original if things go awry.

Mr. Kawasaki, I hope you're right. You told me to just ship it, and I'm going to, dammit.

Just like an entry long before, here is whatthefuck.com in the current Catalyst rewrite. Two weeks until it is deployed to replace the current codebase, and I couldn't be happier. Bigger under the link.

While doing the big spruce up on my servers for Dytara/whatthefuck.com, I've had the opportunity to play with a few things that I never had the chance to play with. Since I moved the primary web server from FreeBSD to Linux (for shame!), I can finally play with some of the Danga/bradfitz tools that have come out over the past couple of years. Since most of the cool toys he releases rely on Danga::Socket, which relies on Linux epoll(), I couldn't use them on FreeBSD.

One of my favorites so far is DJabberd. It's a pretty elegant and small implementation of the Jabber XMPP messaging architecture, commonly implemented by jabberd and ejabberd. It's no secret that I've tried to implement a nice messaging system for wtf, but it has been aborted too many times due to things being too easy to break into or a pain the behind to integrate. I was hacking together a custom bahamut IRC server for use by a private client for wtf, but this may work out much better. In the distribution, I have built in interop with GTalk, LJTalk, and many other networks (meaning your whatthefuck.com IM account can talk to people on LJTalk or GTalk), instant authentication against my own user database, and a nice plugin architecture that will allow me to create hooks into services on the web site, right out of the chat client. I have an army of clients that support it, from Psi and Trillian on Windows to Adium and iChat on OS X, to about 300 clients on Linux. :D

Hell, I can even connect from a Newton and my Nokia 770.

Anyhow, if any of you are on wtf and want to just try it out and see what happens, feel free to login with username@whatthefuck.com and your wtf password. All of the standard config should apply -- port 5222, SSL enabled, can use anything as a resource. Your connect server is also 'whatthefuck.com'.

I've been toying with the fantastic idea of a "Catalyst Studio" type of application. Not saying I would or could write such a thing -- I likely could, it's merely a matter of a "serious lack of time".

A quick and dirty back story: I've been finishing off the newest revision of whatthefuck.com in Catalyst 5.7 on Perl. It's really quite a neat tool, really designed for large scale web applications that require the ability to scale, and will likely have many hands on it. It lacks the (relative) simplicity of your average PHP script, as well as the elegance and niceties that go with other object oriented web application servers such as WebObjects.

WebObjects, for those who don't know, is one of the modern founding fathers of MVC web application development. Large companies used WebObjects to create high availability, large scale applications and paid anywhere from $5,000 to $50,000 for the privilege to do so. WO was originally developed by NeXT Software, was written using Objective-C, and ran on NeXT Server software. As time went on, it ran on Solaris and Windows, and converted completely to Java between version 4 and 5. After Apple bought NeXT, WebObjects became a part of Apple Enterprise. Now, you can't write a WO app in Objective-C, and development is free with any copy of Mac OS X 10.4, and deployment is free with any copy of Mac OS X Server 10.4. Unfortunately, you can't even pay them money to deploy or develop on any other platform. It's simply end of life.

History lesson being complete, I can finally rejoin my train of thought, 200 miles down the track. WebObjects was really cool in that you never had to touch a database, and sometimes, never had to touch code. You would use an application called EOModeler to design a database schema, creating the data sets you need, designing relationships to other tables, and creating a data map. This is a lot easier than it sounds -- if you've ever organized a database, even an Excel spreadsheet, you can use EOModeler. Now here's the cool part. EOModeler will then create the tables in any of its supported JDBC databases, and generate Java classes for you to add, remove, modify, or delete data, so you never really have to write any SQL unless you absolutely have to.

The ORM features in Catalyst allow the same type of system to happen. You can tell Catalyst to generate some classes based on your existing MySQL, or Postgres, or whatever database, and you can use those classes to do the same add/remove/modify/delete as above. Except for the fact that you have to manually declare relationships within each class. The references to them are kinda odd as well, but I can go into that later.

The other issue with Catalyst is the fact that it's written in Perl, so there are twelve different ways to do the same thing. Catalyst is the perfect microcosm of perl culture. When I first looked at Catalyst, early in 2005, there was a certain recommended way of creating your application, and a certain database model (Class::DBI or Class::DBI::Sweet if you were on the edge), and most tutorials had you manually creating classes anyhow. Now, it's all DBIx::Class::Schema and RenderView, and this will probably all be replaced in 5.8. On top of that, many of the handy perl IDEs won't help you much with your Catalyst object ($c), or render the standard Catalyst tree very well. Java users have all sorts of fun tools, and WebObjects users have a whole IDE that work very well. Sure, we have TextMate or Komodo, but they aren't the whole shebang, and I want the whole shebang.

Maybe I will take on a project that works like EOModeler, but works with DBIx::Class::Schema and Catalyst. Maybe we can go from there.

Technorati Tags: catalyst, catalyst studio, webobjects, whatthefuck.com