Jayblog

Jaybill McCarthy, writing stuff.

Testing React Components with Mixins

I recently wrote a blog post about Testing React components with Mixins over on my company’s tech blog. Totally read it!


Lesser Known Emacs Key Bindings

C-y C-j C-j C-r - Yank the last kill into a buffer and eval it in a random programming language

M-$ M-c C-f C-f - Spell check in Hieratic

M-k M-k C-w C-t - Disable all wifi networks within a 1000 meter radius

C-x M-v M-v i - Track packages. All of them.

M-h M-m M-f - Hack nearest mainframe

M-d M-h C-j-j - Increase threat level of Homeland Security Advisory System

M-j M-d M-d - Lower Dow Industrial Average

C-o-o-o - Like currently marked region, then put ring on it

C-t C-x M-d - Disable electric fence surrounding tyrannosaurus paddock

C-o M-m M-8 - Interpret the contents of the current buffer as midi and make the ringtone of every mobile phone within range

C-x C-p M-f M-f - Display sports scores and lottery numbers for the following day

C-x C-f M-p C-M-p - Open a new buffer in an alternate universe

C-a C-k C-k C-y C-y - Duplicate the current line


PostGIS vs. earthdistance

I’m working on a web application which has markers on a graphical map. The markers are stored in a PostgreSQL database as a set of floats, one for latitude, one for longitude. This works fine for storing them, but getting all points within a certain distance of the current point is a bit of a challenge.

I had heard of PostGIS, which is an elaborate extension that “spatially enables” Postgres. It’s pretty amazing, actually, in that in can not only store points, but every bit of geometry needed to create a map. It’s a little bit tricky to install, but I did eventually get it working on my dev box. After reading through a bunch of the documentation, several tutorials and a bunch of StackExchange threads, I was no closer to being able to stick a point in the database, much less determine if it was inside of a certain area. It was at this point that I decided PostGIS was probably overkill for my application.

As it happens, if you all need to do is store points and then determine if said points are inside of a certain area, there’s actually an extension that’s part of postgres-contrib that does that for you called Earthdistance. The documentation is pretty short on examples, but fortunately, this guy wrote a very nice little tutorial that should get you up and running with minimal fuss.

I just wish I hadn’t blown two days going down the PostGIS route. Maybe this will save someone else some time in the future.


Sawsij Hits 1.0

Sawsij logo

A year ago yesterday, I launched a web framework called Sawsij written in Go. I’m proud to announce that after months of hard work, I’ve finally feel comfortable removing the “beta warning” and I’ve officially launched version 1.0.

Here’s an abridged list of the features:

  • An application generator that gives you a generic application you can build on top of (so you don’t have to start from scratch every time)
  • Can generate CRUD screens directly from database tables
  • A lightweight database abstraction layer (the “model” package)
  • Configurable routes
  • Simple Auth/ACL
  • Database versioning migration tool
  • The “model” package now supports both PostgreSQL and MySQL. Other databases/data stores can be used directly.
  • Uses Bootstrap 3 by default (can also be used without it)

I’ve updated the site, which now includes a gallery of sites made with Sawsij. Neat!

Exciting times!


The Making of Tugglebottoms: Part 7

NOTE: This post is part of a series that starts here.

Anyone paying attention probably noticed I’ve been absent for a week or two. I had to (and continue to have to) deal with some personal things, but I’m finally starting to get back on track.

Earlier today, I did some sketches of what the barge will look like:

sketch

And here’s the rendered comp of said barge:

sketch

I think it came out pretty well. The general idea was to make him look kind of bored, which I thought was funny in contrast to Tugglebottoms, who is very, very excited.

Hopefully by tomorrow I’ll have a new gameplay video to show off, but don’t hold me to that.


The Making of Tugglebottoms: Part 6

NOTE: This post is part of a series that starts here.

I know I’ve “gone dark” for a few days, but don’t let that fool you. I’ve been hard at work on this thing. I’ve implemented the title screen, the episode screen (mostly) and the level select screen (partially). I also have an extremely basic level sort of kind of working.

Would you like to see a short video of the thing working? Of course you would!

While this might not look like much, there’s quite a bit going on here. I implemented the screens as levels in LevelHelper, which made it a lot easier to get things pixel accurate against my design comps in Photoshop. The screen transitions are part of Corona. The title and episode screens were more or less completely laid out in the tool. The level select screen (other than the title) is generated dynamically, which begs another question: How are we storing the data to generate the thing dynamically?

Corona SDK comes with bindings for SQLite, which is widely considered to be the world’s most adorable SQL database. Under normal circumstances (read: In a server application), I wouldn’t touch the thing, but since this application is inherently single-user, it works like a champ. I check to see if the database exists when the game launchs and create it if it doesn’t. At the moment, it’s just one table with a row for each level, indicating if it’s locked or not and how many anchors you’ve achieved. The Corona bindings are pretty simplistic (there’s an exec() and a nrows(). That’s it.) and there is absolutely no database abstraction, but the interactions with the database are so simple and occasional that I don’t see abstraction as much of an advantage here.

Next Steps

Now that I can actually load a level, I’ll probably implement some kind of a “hazard” so that I can complete the win/lose mechanics. I’ll do pause, restart level and back to level picker at the same time.

On to Part 7


The Making of Tugglebottoms: Part 5

NOTE: This post is part of a series that starts here.

Not too much to report today, but I did finish (and update) some screens I thought I’d share.

Episode Select Screen (Updated)

Original sketch:

design2

Design comp:

Here, the only difference is that I added your total “anchor count” to the upper right.

Level Select Screen

Original sketch:

design3

Design comp:

I decided to go with 20 levels per episode instead of 12 (as seen in the sketch). The anchor counts for locked levels are wrong in the comps, of course, but you get the idea.

Next Steps

Still need to finish the “win” and “fail” screens, as well as the UI for the level itself. Then I’ll implement those. The Level and Episode screens came out a lot more…ornate…than the Title screen, so I will probably go back and make that a lot more interesting at some point. That point is not now.

On to Part 6


The Making of Tugglebottoms: Part 4

NOTE: This post is part of a series that starts here.

I have been hard at work creating images pixel by pixel, which is very rewarding but intensely time consuming. I’m really happy with the results I’m getting, though.

(Click on the design comp images for the larger “iPad” versions.)

Splash Screen

Original sketch:

design1

Design comp:

Episode Select Screen

Original sketch:

design2

Design comps:

The first image shows what this screen looks like before you’ve unlocked the second episode, the second shows it unlocked. I spent waaaaay too much time on that San Francisco panel (curved things are hard to pixel-ize!) but I think it came out pretty neat.

The sharp-eyed and marketing-savvy among you will notice that there’s room at the bottom of this interface to accommodate a 320x50 mobile banner ad, which I could use as a direct revenue stream, but will probably use as to promote the paid version and such, at least at the beginning.

There’s also room at the sides for forward and back buttons, which will come into play when there’s more than four episodes. (There will be more than four episodes, mind you.)

Creating the Artwork

Initially, I was drawing everything on one layer in Photoshop. That’s okay for simpler stuff, I guess, but as things get more complicated, grouped layers and Smart Objects save tons of time. I created smart objects for anything that was repeated, like the little brass knobs and even sections of the wooden rails (or are they rope?). I used them almost like tiles. I’m not sure I’ll actually implement them as tiles, but I think it works for the aesthetic I’m going for.

I went for a vaguely nautical look with the UI elements. I mean, it’s a game about a tugboat, for crying out loud. If there’s not water and brightwork in your tugboat game UI, you fell off.

You may be wondering why I’m taking the time to do all this UI work up front instead of the gameplay. Well, it’s my opinion that game UI is very important to the overall experience, and it can make a fun game into much more of an immersive experience. Games that overlook this feel really sloppy. I mean, I’m playing a game, not working on a spreadsheet. Make the UI fun to look at!

To that end, I plan to use the already-running physics library in the game to animate putting these UI elements on the screen, probably so they drop in and bounce or something. I have to play with that a bit.

One interesting technical note: In order to account for iPhone, iPad and “Retina” based versions of both, one needs to create three different versions of each and every sprite. I will now thank my lucky stars for Photoshop Actions, with which I have (largely) automated the process of creating image assets from my design comp files.

Next Steps

I still need to do comps for the level picker, the level screen (including the pause menu) and the win and lose screens. Then I can get on to the business of implementing those. I figure I’ll get the basic UI functional and then move on to level design and game logic.

On to Part 5


The Making of Tugglebottoms: Part 3

NOTE: This post is part of a series that starts here.

Screen Flow

Before I go too much further into game design, the application developer in me wants to wireframe the screens I’ll need. I figure I’ll get this out of the way now and then I can focus on the harder stuff, like, I don’t know, how you actually go about playing the thing.

I borrow pretty heavily from Ye Olde Angry Birds here, in that you start on a splash screen, then you pick an “episode” (more on that later) then you pick a level, then you play. While playing, you can pause, return to the level select screen, or restart the level. Then you either complete the level or fail, at which point you can continue to the next level if you win and return to the level select or restart if you fail. Pretty basic stuff, and pretty standard for the genre.

Overall, I want there to be as few actual words as possible, for two reasons: First is that I want the thing to be really intuitive, second is that I don’t want to do a ton of localization.

Splash Screen

design1

This is the first thing you’ll see when you launch the app. It’ll double as a loading screen if that becomes necessary. The button in the middle takes you to the Episode Select screen.

Episode Select Screen

design2

This is where you pick which episode you’d like to play. I figure I’ll probably go live with two episodes. The first episode will be part of both a free and paid version of the game, the second episode on up will only be available in the paid version. This will be the difference between the paid and free versions: The paid version will have way more content. I’ll also push updates with more episodes for paying customers over time, which should give the game more value.

Level Select Screen

design3

Each episode is made up of twelve levels that will get progressively more difficult. You start with only the first level of the episode unlocked. Selecting a level will start the game on that level. Each level will give you the opportunity to get three “anchors” which you obtain by successfully docking a barge. To unlock more levels, you need more anchors. You don’t need to get every single available anchor to unlock all the levels, so if you find a particular level particularly difficult, you can totally skip it. I figure this is a good balance: Really good players can aspire to 100% everything, while the less skilled get to see more of the content. I largely lifted this concept from the Mario 64 (and onward) star collecting game mechanic.

Level screen

design4

This is the actual gameplay screen. I’ll go into much greater detail with this later, but the idea is that Tugglebottoms starts on the right, facing left. You have to remove any obstructions from his path so he can go get a barge. He’ll get to the barge on the left and try to push it back to the right, again while you remove obstacles. The goal is to do this three times. If you get at least one barge in, you go to the Win Screen. If you don’t get any, you go to the Fail Screen.

The Win Screen

design6

Here’s what you see if you get at least one barge in. It shows you the number of anchors you’ve obtained and gives you the option to try again, go back to the level select or go on to the next level.

The Fail Screen

design7

This is what you see if you don’t get any barges in. You can either restart the level or go back to the level select screen.

Next Steps

From here I need to actually create some more graphic assets, which you’ll see in my next post.

On to Part 4


The Making of Tugglebottoms: Part 2

NOTE: This post is part of a series that starts here.

Choosing Technologies

So far, I’ve got a good skeletal game design and a pretty reasonable visual aesthetic. What I need to do now is figure out a) what my target platforms are and b) what development tools I’m going to use to build this thing. This turns out to be a more complex question than you might imagine.

After some cursory research, I’ve determined that while Android has greater market share, iOS users buy more apps. Also, Windows phone…exists. All kidding aside, I did consider targeting Windows phone. This is less crazy than it sounds. Sure, you’ve got a much smaller audience, but you’ve also got a lot less competition. There are almost a million apps a piece in both the App Store and Google Play. There are only twenty thousand in the Windows store, and of those only 2000 are paid. Ignoring most of this research, I decided that I want to target iOS first, followed very shortly thereafter by Android, because America.

So now that I know what platforms I’m targeting, I need to choose what development tools I’m going to use. Not long ago, this was an easy decision: iOS apps were developed in Objective C and Android apps were developed in Java There are purists out there who make a lot of noise about how “If Jobs had intended man to develop iOS apps in something other than Objective C, he would have put it in NeXTSTEP”. I do not care about this, even a little. I will do what makes the most sense.

I have quite a bit of Java background but my experience with Objective C is limited and all bad. I also don’t love the idea of writing all the code twice in two completely different languages. Fortunately, these days there’s quite a number of options for doing cross-platform mobile development. In fact, there’s a dizzying array of options, so much so that it’s really hard to decide. I came up with a basic set of criteria to help make the decision. I figure that any toolkit I use must:

  • support both iOS and Android (at least)
  • be able to take advantage of hardware acceleration
  • cost less than $500USD (an amount I picked out of the air)
  • not require royalty payments
  • have free trial of some kind so I can determine if it will do what I want it to do
  • have reasonable tools for creating sprites, levels and physics
  • use a programming language that will not enrage me (admittedly a tall order, for I am easily enraged)

Using these criteria, I was able to eliminate a number of options immediately. Here are the options I evaluated seriously:

iTorque2D

iTorque2D site

Cost: $128.00USD

I like the Torque2D toolkit. As I mentioned earlier, I had done some work with it for a PC platformer. The level editor is really nice and I was able to learn the API pretty quickly. It had to be eliminated for lack of an equivalent Torque2D toolkit for Android. I also understand it has performance issues with hardware acceleration. It’s a shame, really, this would have been by first choice by a wide margin otherwise.

Create.js and Phonegap

Phonegap site | Create.js site

Cost: Free(ish)

I recently did a project at my day job using the Create.js toolkit, and I like it a lot. It gets pretty close to turning HTML5 into Flash/ActionScript in terms of functionality. In theory (and I have not tried this) I could write my game in HTML5/Javascript and use Phonegap to create iOS/Andriod applications. There are two show-stopping drawbacks to this approach. First is that there’s no support for hardware acceleration (though there’s plans for it). Second is that there’s no built-in support for tile maps, much less anything like a level editor. This means I’d either have to write or adapt something to do this part, which doesn’t seem worth the effort to me given the features available in other toolkits.

The other big thing I wasn’t in love with about this approach is that Phonegap and Create.js are maintained by two completely different groups (though both have support from Adobe) and if I ran into issues, I might waste a lot of time trying to figure out where the problems were.

Impact.js

Impact.js site

Cost: $99.00USD

Impact.js is a really nice toolkit. You write your game in Javascript/HTML5 using canvas, test in a browser, and then package the thing up with their Ejecta Framework, which is basically a hardware accelerated implementation of the HTML5 Canvas. It also includes a nice level editor. I was really leaning in this direction, right up until I read this on the Ejecta page:

The JavaScriptCore API is private on iOS, which means you’re not allowed to link to it. Ejecta therefore comes bundled with its own version of the library to circumvent these restrictions. Several games made with (earlier versions of) Ejecta found its way into the App Store without a problem.

Hmm. I don’t know about you, but that sounds an awful lot like “We’ve bent the rules in a way that Apple hasn’t noticed yet.” Apple has been known to pull the plug on things for far less. I don’t think I can justify taking that chance.

It also seems like I’d have to find some other solution for Android (like Phonegap or somesuch) and that seems like a lot of extra work.

Corona SDK

Corona site | Game dev Helper site

Cost: Corona SDK $349.00USD + LevelHelper/SpriteHelper $49.00USD = $398.00USD

Corona is a very popular toolkit that has all the features I want along with really good documentation, a huge community and commercial support, should I want that. While the toolkit doesn’t include a level editor (though there’s one rumored to be in private beta) there’s another outfit that makes one called LevelHelper that more than suits my purposes. It also includes a companion tool called SpriteHelper, which makes doing sprite animations and rigging really easy.

I was able to download trial versions of both and I got them working pretty quickly, and within an hour or two I had a simple game working in the included simulator. The toolkit uses Lua as its programming language. This initially worried me, but after reviewing the sample code provided, I discovered my fears were unfounded. Lua, at least in the form of this SDK, is comically easy to use. They do as much of the legwork for you as possible, while still giving you quite a bit of control. There’s a cartoonish example of how much less code you need to do things in CoronaSDK/Lua than in the iOS SDK/Obj-C or in the Android SDK/Java. Take this with a giant grain of salt, but it does give you a good indication of the unneeded complexity they’ve tried to remove.

Conclusion

After weighing out the costs, features and processes required for each option, I decided to go with Corona SDK. My total cost for dev tools (when you add in the obligatory $99 for the iOS developer program that’s required to put stuff in the App Store) is $497USD, or $3 less than my development tools budget. Not bad!

Next Steps

I need to flesh out the screen flow and the game design, which I’ll focus on in my next post, which will totally have pictures.

On to Part 3


The Making of Tugglebottoms: Part 1

I recently had an idea for a game. I think it’s actually a pretty novel and clever idea for a game, so I decided I’d see what it took to develop it myself. This is the first in a series of blog posts that will describe that journey, from the original idea through to my ultimate success or failure, come what may. I’m going to do this right out in the open in the hopes that others might learn from it, whatever ends up happening. I also have a lot of things to learn, and it’s my hope that documenting all this might prove useful later.

I can hear you screaming at your computer. “What?! You’re not worried that people are going to steal your idea?” No, I’m not. Anyone capable of doing this sort of work very likely has their own ideas and isn’t going to bother stealing mine. Also, I subscribe to philosophy of the late Howard Aiken, who famously said, “Don’t worry about people stealing your ideas. If your ideas are any good, you’ll have to ram them down people’s throats.”

My Game Development History

I am not, by trade, a game developer. (Not yet, anyway. We’ll see where this goes.) I’ve largely worked on web applications, and even then most of that work was in the back and middle tiers. On my own time I’ve written two games that I didn’t finish: A Megaman-like platformer and a text adventure. For the former, I used an early incarnation of Torque2D. For the latter I wrote the engine myself in .NET using Mono. While both of these were fun projects that I poured a lot of time into, I wasn’t able to complete either of them. Both were very ambitious projects (more so than I realized at the time) and in the end I just couldn’t get them done. I’ve had a number of other false starts with games over the years, but nothing that resulted in any appreciable amount of code, much less anything working. This is all to say that while I’ve got some background and skills that will help me here, I’m hardly a seasoned pro at this the way I am in other areas of software development.

The Big Idea

I week or so ago, while doing something else, an idea popped into my head: A game where the object is to get a tugboat to pull barges into a port. Granted, it wasn’t much of an idea, but there’s no shortage of games (particularly in the casual market) based on less. I also figured I should probably develop it for mobile, as that seems to be where the action is these days and I thought the idea would lend itself pretty well to that format.

I mentioned the idea to my wife, Erin, who was skeptical but extremely encouraging. She even suggested a name, “Tugglebottoms”, which I immediately loved and decided to use. We talked at length about the idea (she’s an excellent sounding board) and came to a couple of conclusions about what makes for a successful casual mobile game. Of course when I say “successful casual mobile game”, I mean “Angry Birds”. Here’s what we landed on as “must haves”:

  • It must be really easy to learn. AB has one gesture and not a single word of instruction. (There are a few visual instructions, but that’s all.)
  • It must be fun and have a decent level of difficulty.
  • It must be ridiculously, unbearably, mind-bogglingly adorable.

Initial Game Design

Initially, I figured the player would control the tugboat and have to navigate a series of obstacles to get the barge across the harbor into the port. It occurred to me, however, that controlling a thing that must move across a tiny screen by touching (and thereby covering up) said screen is less than optimal. None of the attempts I’ve seen at overcoming this have impressed me. On-screen buttons reduce your already minimal screen real estate (and hey, we need that space for adorableness). Accelerometer-based control schemes tend to be really imprecise. Then it dawned on me: What if the tugboat moved by itself, and the player’s goal was to make sure his path was clear? You could have things that would obstruct the tugboat that the player would touch and interact with to clear the path. Drawbridges you had to hold open came to mind. Jet-skis that would come flying towards the tugboat, perhaps after losing their rider. Erin suggested a pod of dolphins that surround the boat that player has to distract by pointing elsewhere.

Graphic and Character Design

Visually, I really wanted to go retro, in particular, SNES-style. Some of the most fun I’ve had playing video games took place on that console. Super Mario World, in particular, is just pure joy. I wanted to make something that was an homage to that art style and color palette.

Having a very loose idea of what I wanted, I decided I’d just fire up Photoshop and start creating some concept art. Normally, I’d do this by hand with a pen and paper, but as I knew I wanted this to end up as pixel art, I thought it might be best if I just dove right into the medium. I found a couple of neat tricks in Photoshop that make it a pretty decent tool for pixel art. The most important thing is that you can open multiple windows with the same document in them. This means that you can zoom in to the pixel level in one window (at like, 1600%) while keeping another window zoomed in at 200%, allowing you to see what the result will look like in real time. I thought I’d start with the main character, Tugglebottoms, who is an anthropomorphized cartoon tugboat. After an hour or two of work one evening, here’s what I came up with:

Derplebottoms?

Oof. This image was immediately christened “Derplebottoms” and rejected. It was a good practice for doing pixel art in Photoshop, but it’s not cute and I’m pretty sure it would beg for its own death if it were real. No, this was going to require something way, way cuter. The following evening (and early morning) I came up with this:

Tugglebottoms 1

Now that’s more like it, I think. The problem, however, is that it’s enormous for a sprite that’s going to be used on a smart phone screen. I’d have to scale it down to make it work, which, when you’re talking about pixel art, is a lot more complicated that just shrinking the image.

A few days later, I came up with this:

Tugglebottoms 2

This one is only 32 pixels wide in its smallest form, which is perfect. I also think it’s even cuter than the previous iteration.

Before going any further, I thought it might be a good idea to do a quick screen mockup to get a general sense of whether or not this little guy is going to work. I did a quick pixel art background and threw him in an image the size of an iPhone screen, which I then brought up on my iPhone:

Screen mockup

I was pretty happy with this in terms of character design and execution, so I decided to run with it. I made some drawings to work out some of his facial expressions:

Face sketches

At this point, I had a pretty good idea of where I wanted the thing to go visually and I’ve got enough of a skeleton of a game design that I can start hanging details on it.

In Part II, I’ll start getting into how I arrived at my technology choices and what my target platforms are.

On to Part II


The Sawsij Framework

Sawsij logo

I’ve just launched a new web framework called sawsij. (pronounced “sausage”) It’s written in Go. I’ve tried to take all the things I liked and didn’t like about other frameworks (some of which I’ve written or helped write) and distill them into Just What I’ve Always Wanted™. I think I’ve managed to get pretty close.

Here’s an abridged feature list:

  • An easy to follow application structure
  • An application generator that gives you a generic application you can build on top of (so you don’t have to start from scratch every time)
  • A lightweight database abstraction layer
  • Configurable routes
  • Simple Auth/ACL
  • Database versioning migration tool

Give it a whirl and let me know what you think!


What Developers Mean: a Guide for Project Managers

If you’re a project manager new to software development projects, a lot of things that developers say can be confusing, particularly when it comes to the status of tasks. Use this handy guide to translate what your developers are saying into meaningful answers that will help you update your Gantt or burndown charts.

All of the following assume you’ve just asked a developer, “Hey, so, are you done with task X?”

Developer: “I should finish that up today.”

Meaning: I haven’t even started that yet.

Developer: “That’s all done, I just need to check in my code.”

Meaning: I haven’t even started that yet.

Developer: “That’s done, but there’s a bug I need to fix, so I’m going to hold off on checking that in.”

Meaning: I haven’t even started that yet.

Developer: “That’s done.”

Meaning: I will start that today.

Developer: “I should finish that up tomorrow.”

Meaning: I will start that tomorrow.

Developer: “I think I’m going to need until the end of the week on that.”

Meaning: I won’t start that until you ask about it again in a few days.

Developer: “I’m all set up to do that.”

Meaning: I haven’t even considered starting that yet.

Developer: “We should probably push that to the next [phase/sprint/quarter].”

Meaning: I have no intention of ever doing that. Ever.

Developer: “I’m waiting for [some other person] to get me [a thing] but I see no reason why I shouldn’t finish that up today.”

Meaning: That is not my problem yet.

Developer: “I’m almost done with that, but I’d like [another developer] to look at it.”

Meaning: That will be someone else’s problem by the end of the day.

Developer: “I have to do some research on that, but I should finish that up today.”

Meaning: Not only haven’t I started that, I don’t even know how to do it.

Developer: “I shouldn’t have any problem finishing that up today if you can just get me [some information].”

Meaning: I will be ignoring your emails today.


How to copy an entire directory tree in Go

UPDATE: It has been pointed out that Go actually does have built-in functionality for doing this. You can use the Walk() function in the filepath package. I’ll leave this post here because it’s not bad code and because it will serve as a reminder to thoroughly search the package documentation before inventing something.

Go has a lot of built-in functionality for dealing with files and directories, but it lacks an easy way to copy an entire directory tree. To this end, I’ve written the following bit of code. I’ll list it out first and then talk you through it:

copytree.go

package main

import "fmt"
import "os"
import "io"
import "io/ioutil"
import "log"

// Copies file source to destination dest.
func CopyFile(source string, dest string) (err error) {
    sf, err := os.Open(source)
    if err != nil {
        return err
    }
    defer sf.Close()
    df, err := os.Create(dest)
    if err != nil {
        return err
    }
    defer df.Close()
    _, err = io.Copy(df, sf)
    if err == nil {
        si, err := os.Stat(source)
        if err != nil {
            err = os.Chmod(dest, si.Mode())
        }

    }

    return
}

// Recursively copies a directory tree, attempting to preserve permissions. 
// Source directory must exist, destination directory must *not* exist. 
func CopyDir(source string, dest string) (err error) {

    // get properties of source dir
    fi, err := os.Stat(source)
    if err != nil {
        return err
    }

    if !fi.IsDir() {
        return &CustomError{"Source is not a directory"}
    }

    // ensure dest dir does not already exist

    _, err = os.Open(dest)
    if !os.IsNotExist(err) {
        return &CustomError{"Destination already exists"}
    }

    // create dest dir

    err = os.MkdirAll(dest, fi.Mode())
    if err != nil {
        return err
    }

    entries, err := ioutil.ReadDir(source)

    for _, entry := range entries {

        sfp := source + "/" + entry.Name()
        dfp := dest + "/" + entry.Name()
        if entry.IsDir() {
            err = CopyDir(sfp, dfp)
            if err != nil {
                log.Println(err)
            }
        } else {
            // perform copy         
            err = CopyFile(sfp, dfp)
            if err != nil {
                log.Println(err)
            }
        }

    }
    return
}

// A struct for returning custom error messages
type CustomError struct {
    What string
}

// Returns the error message defined in What as a string
func (e *CustomError) Error() string {
    return e.What
}

func main() {
    err = CopyDir("/home/jaybill/data", "/home/jaybill/backup")
    if err != nil {
        log.Fatal(err)
    } else {
        log.Print("Files copied.")
    }
}

Looking at the main() method, you can see how to use this code. You supply a source directory (which must exist) and a destination directory (which must not exist). CopyDir() returns an error if there is one.

CopyDir() is a recursive function which looks at the source directory, makes sure it exists and gets its permissions. It then creates the destination directory with the same permissions. It gets all the filenames in the source directory and then calls CopyFile() to (surprise!) copy the file. It checks each file to see if it’s a directory. If it is, it calls CopyDir() recursively, thus going through the entire tree.

Thanks to the participants in this thread that provided me with a starting point and most of the CopyFile() function.

This is also available as a gist if you want to grab it that way. Feel free to suggest improvements!


Does that belong in the database?

When standing up a data model for your application, you often start by considering the real-world properties of whatever it is you’re trying to model. That’s totally natural, considering what most applications do: Represent real-world problems and help the user solve them. There’s a hidden pitfall here, though, which is the tendency to have elements in the model that logically apply to the entities you’re modeling, but do not matter to the problem space.

I ran into an example of this recently in an application I was working on. There was a need to represent “sponsoring organizations” in an application that tracks monetary grants. When we asked our in-house product manager for a list of things we needed to track about sponsoring organizations, she replied with all of the normal things you might expect: Organization name, ID number, tax status, phone number and an address, including city, state and zip. Okay…nothing unreasonable there.

If you’ve ever had to deal with addresses in a database, though, you know that all bets are off when it comes to dealing with addresses in a way that they work internationally. As my colleague and I started down the path of a long discussion about the best way to store addresses (and the rest of the people in the room settled in for a nap), our resident Database GuyTM asked a seemingly obvious question:

“Do we actually need to store addresses?”

After a long pause and a lot of blinking, one of the product management people blurted out, “Well…of course! I mean, they have addresses…” The Database Guy calmly responded with a question no one could answer. “Okay, but where are we going to display that information in this application?”

We looked through all our requirements documentation. Addresses weren’t on any reports and they weren’t on any screen. Well, they weren’t on any screen except the one where you entered the address. After a some further discussion, we found out that our customers either don’t maintain this type of information, don’t need it, or store it elsewhere. “Address”, in this case, was a logical attribute of a sponsoring organization, but one that attribute had absolutely no meaning in the application context.

Someone then pointed out that in another application we sell, we do maintain address information, but we load it once when we deploy the application and “The customers know that the address information is usually out of date, so no one uses it.” I’d hate to think about what would happen if a customer didn’t know that and mailed something to an old or incorrect address! Unmaintained information is not only annoying to users, but it’s not hard to imagine a scenario where it could be costly or even dangerous.

I think the basic litmus test for determining if attributes belong in a database can be boiled down to this: If you don’t know where a particular data attribute is going to come out, then do not put it in, no matter how logical it may seem to do so.


Please decide if you're a producer or a consumer.

Front page of hacker news

There has been a lot of fuss today about the merits of learning to code versus not learning to code if you’re a non-programmer. There are interesting points on both sides of this argument and I won’t rehash them all here, though some resonated with me more than others.

I think we’re all having the wrong argument. This isn’t about learning to code or not learning to code. This is about consumption versus production.

Let’s compare this situation to something more generally well-understood: Books. Until relatively recently, if you had something to say and wanted to share it with a lot of people, you wrote a book. There are lots of examples of books that changed the world and how we look at it.

Writing a book is hard. Additionally, not everyone has something they want to share with the world. People who do have something to say are sure as heck going to learn how to write and publish a book. These people actively engage in the production of books. You don’t have to worry about advising them that they should or shouldn’t write a book. You couldn’t stop them if you wanted to.

Do you need to understand the book authoring and publishing process if you’re just going to read a book? No, you don’t. If you’re just going to consume books, you don’t actually need to understand the process of making them. Does it hurt if you do? Of course not. Does it give you greater insight into the work itself? It well might. It isn’t, however, a requirement.

The means of production on the web (at some level, anyway) is code. If you want to produce something new and useful for the world to see, you need to know how to write code. If you just want to consume web content, you really don’t need to know how to produce it.

I’m not here to tell you if you’re a producer or a consumer. That’s up to you. That said, I’m certainly not going to discourage a non-programmer that’s interested in coding from learning how. There are no shortage of well-documented benefits to doing it.

Do I think knowing how to code is a requirement to be a productive member of modern society?

No I do not.


Do not read this if you like installing Postgres the hard way.

I like Postgres. It’s feature-packed, high-performance, robust, mature and a bunch of other adjectives that describe good software. It’s also kind of obnoxious to install. Installing it totally by hand requires a lot of Arcane KnoweldgeTM. Probably because of that, OS packages usually get it wrong. (I’m looking at you, Ubuntu. Sure, there are PPAs that deal with these problems, but PPAs have a way of scaring the bareekus out of me, especially on production systems.

Recently, I discovered a 3rd-party installer package that works better than anything else I’ve tried. It’s called Enterprise DB, and it’s pretty great. If you’re on a machine with a GUI, it’s got a wizard that takes you through the process. If you don’t have a GUI (on a remote server, for example) the exact same wizard runs via interactive command line. I’ve found that just picking all the defaults lands you with a very sensible configuration in about a minute. It’s free (of course) and just works. It’s absolutely my go-to way of installing postgres at this point.


Is there a right way to write web apps?

I have been developing web applications for almost as long as it’s been possible to do so. No, really. I started doing this stuff in 1995. The web as we know it got off the ground in 1993. My entire professional life has been spent getting web servers to talk to web browsers in one form or another. I’ve built all kinds of web applications. I’ve built small ones, big ones and everything in between. I’ve built them for clients, I’ve built them for employers and I’ve built them for myself. I’ve worked in Java, PHP, C#, Ruby, Python and a number of more exotic things. In all that time, I have never found a language/platform combination that really felt…right.

I’ve gone to great lengths to try to assuage this feeling. I think I’ve tried every framework available in every language I’ve used. I even went so far as to write my own frameworks in one or two of them. (One of which is still in active use, actually…) I always get excited at first, then work with it for a while, then expose the glaring flaws. Sometimes it was the framework, sometimes it was the language, sometimes it was both. Sure, I got stuff done and I got paid and moved on to the next thing, but not without knowing that parts of what I worked on were just ugly. More than once, I’ve found myself using the fact that I delivered something (or that someone, somewhere did) as an apology for the development platform’s weaknesses. I’m just not going to do that anymore.

I guess this is the point in where I’m supposed to tell you about what a mess PHP is, or how unnecessarily abstracted Java is, or how Ruby is…well, you get the idea. I’m not going to do that. I could fill a station wagon with all the negative screeds that have already been written about what’s wrong with this way or that way of writing web applications. I have no intention of adding to that. What I’m going to do instead is outline what I want out of a programming language that I’m going to use.

Here’s my wish list:

  1. I want an elegant, modern language with clear, concise syntax.
  2. I want a compiler, not an interpreter.
  3. I want a good set of standard libraries.
  4. I want a lot of available packages and an easy way to install them.
  5. I want a thriving ecosystem of developers.
  6. I want something I can use to build a framework.

What I Chose: Go

After what can only be classified as an exhaustive search, I landed on Go. If you’re unfamiliar with it, Go is a language designed by some very smart people at Google. When I say “very smart people”, I mean nerd heroes of the highest order. Guys like Rob Pike and Ken Thompson, whose collective achievements include things like UNIX, Plan 9 and UTF-8. I think it’s fair to say that these guys know what they’re talking about.

Go grants every wish on my list and then some. I thought it might be worthwhile to explain how it does this, so I’ll just go ahead and do that, thank you very much.

An elegant, modern language with clear, concise syntax.

According to the brochures, “Go is a compiled, garbage-collected, concurrent programming language.” Well, that sure adds up to “modern” in my book! The syntax has a lot in common with C, but without any of the annoying stuff. There are pointers, but there is no pointer math. It is not, in the strict sense, “object-oriented”. At first glance, this bothered me, as I have lived in Object Land for my whole adult life. After you get your head around it, though, this is actually a huge benefit. Objects are a collection of methods and properties. Go separates properties from methods. Here’s an example of what I mean:

type Foo struct{
    Bar   string
    Baz   int64
}
func (f *Foo) GetBaz() int64 {
    return f.Baz
}

What this does is to establish a data type called Foo, with two properties, a string called Bar and an int called Baz. Below that is a method that can be called on a pointer to a Foo and returns its Baz property. Here’s how you might use that:

foo := &Foo{Bar: "Hello, World!", Baz: 23}
fmt.Print(foo.GetBaz())

If you ran this, it’d print “23”. Calling a method on a pointer to a struct is as easy as adding a dot. There are a number of advantages to making this separation. First off, there’s no complex type hierarchy. If your “thing” implements the methods of X interface, you can use it anywhere that accepts X. Done deal. This means that you can do things like implement several interfaces at once, and that your data types aren’t “carrying around” a bunch of extra stuff, like a method table or unnecessary overhead from parent types. Also, because two types that implement no methods would match, you can have functions that accept an interface with no methods and figure out what the thing is later. This gives you something not totally unlike generics (my most-loved feature of .NET).

You can also extend anything with a method that accepts a pointer to it without having worry about type hierarchy, because there is no type hierarchy. Pretty neat!

This is only one example out of a hundred that demonstrates how well thought out the language design is. Don’t take my word for it, though. Go take the tour and see for yourself.

A compiler, not an interpreter

Sure, interpreted languages are fine, but apologizing for less-than-optimal performance by extolling the ease-of-use of the language just doesn’t cut it for me anymore. That’s like saying it’s okay that your car has terrible gas mileage because it has a really nice steering wheel. Performance aside, I just like having a compiler. I like to know that my code is going to run before I run it. This is a personal preference, to be sure, but then again, this is my wish list, not anyone else’s.

Go’s “official” compiler (there are two official ones, I can only speak to gc, as I haven’t used gccgo) is very fast and provides very useful error messages. The compiler is so fast, actually, that it’s almost like working with a dynamic language. I’ve got a shortcut key set up in my IDE (well, my text-editor) that saves my files calls the compiler for me. There’s really no waiting around for stuff to compile, as it’s almost instant.

A good set of standard libraries

The standard libraries supplied with Go are awesome, particularly when it comes to writing web applications. They more or less thought of everything. There’s an HTTP server you can instantiate, a template parser, all manner of encoding stuff and even a standard sql library. (Drivers sold separately, but there’s one available for just about any RDBMS you can think of.)

A lot of available packages and an easy way to install them

There are a boatload of open source packages available in Go, which really speaks to its growing popularity, as the whole thing is like three years old. A quick glance a the Project Dashboard reveals there are packages to do just about anything the standard libraries don’t do alone. Assuming you’ve followed the directions for standing up your development environment, incorporating new packages into your code is as simple as:

go get code.google.com/p/example

Then adding an import statement to your code:

import "code.google.com/p/example"

You grab packages by the URL of where the source lives. Go’s “get” tool knows about all the major public code repository systems, including Github, Bitbucket and Google Code. Assuming you’ve got Subversion, Mercurial and Git in your path, it can automatically clone package repositories and put things in the right place for you.

A thriving ecosystem of developers

Perhaps not surprisingly, Go has a large, supportive community of very smart developers around it. The main golang-nuts mailing list is fantastic. I’ve never posted a question that went without a quality answer for more than a day. Additionally, most of the larger 3rd-party package maintainers have active mailing lists as well.

Something I can use to build a framework

I hate boilerplate code. If it’s possible to do something once and usefully reuse it later, I’ll take the hit at the beginning of the project and write something I can use over and over again. I admit that I have at times overdone this, but I’ve written a couple of web frameworks of varying sizes at this point, and I have a really clear idea of exactly what I want and what I don’t. What goes into a web framework is a whole blog post in and of itself, but to answer the obvious question: Yes. Go is awesome for writing web frameworks.

How do I know? Well, I just finished writing one. It took me exactly thirty days to do this, and that was far from full time. I had also never used Go before embarking on this project. As of today, it’s in production. It is, in fact, delivering the words you are reading right now. This blog is a Go application that uses my new framework. I could not be happier with it and I can’t wait to share it with you, which I’ll do very soon.

I feel like I finally found the right way to write web applications. Maybe it’s not right for everyone, but it sure is right for me and I’m willing to bet other people are going to dig it as well.


About Me (And This)

My name is Jaybill McCarthy. I have done a number of different things so far, but professionally I’ve spent a lot of time developing large-scale web applications. I’ve worked for a big company and a bunch of small ones you’ve probably never heard of. I’ve also done a lot of freelance work. I thought about listing out all the things I’ve done and worked on, but I’ve done that elsewhere already. Frankly, I hope what I write here will speak for itself and that it will be evaluated on its own merits rather than in the context of my work experience.

In this blog, I plan to write about my thoughts and experiences around developing and deploying web applications. Hopefully other people will find what I have to say useful. Failing that, they’ll find what I say absurd and we’ll have a heated public argument about it, which will probably be fairly entertaining, in a “watch bitter nerds fight over the minutiae that passes for meaning in their lives” sort of way. Failing that, the whole thing will go largely unnoticed, which is kind of sad to think about, so I’ll just “forget” to install Google Analytics and pretend I have thousands of daily readers hanging on my every word.

The main reason I’m starting this blog is that I’ve recently “turned a corner”, so to speak, as a software developer. I have abandoned old ways and embraced a bunch of new ones, which we’ll get into later. In the meantime, thanks for reading and stay tuned.

“What happened to that site you used to have?”

Jaybill.com has existed as a blog of sorts since 2001. I got a lot of traffic early on due to a thing I will not mention here. After that I had a number of false starts, but never seemed to really “get off the ground” as a blogger, mainly because I never had a consistent focus. I hope to change that by focusing on one thing (software) and updating regularly. We’ll see how that goes. I have high hopes.

There were a number of technical blog posts on the old site, most of them related to a way of doing web development I’m not doing anymore. I’m sure that any useful information on that subject can be found elsewhere at this point, so it’s of no great loss to the world that I took it down.



© 2001 Jaybill McCarthy