UNIX and the Emerging Web

After reading "The Art Of UNIX Programming" again, it's got me thinking about the nature of the technological explosion that has happened since the quaint days of mainframes and text-based consoles.

The two ideas that these legendary computer scientists cared greatly about and seem to be dissolving today are ownership and simple formats.

Ownership

Web-based services take a certain level of trust. The user gives control of their data to a service provider in trust. By doing so, there is a subtle loss of control by the user. We end up with bits of content all over the web that may or may not be accessible or editable by us.

There has been some recent development to build a unified "user space" like OpenID, but in my alternate history of the internet, user profiles would have expanded beyond the workstation into a worldwide user space. Up until now, the user space was determined by email accounts, but with the shifting landscape of email services this was not really a practical solution.

In my alternate history, instead of email accounts, user profile would be generated like IP addresses, and could be linked to email accounts or any other service. Maybe this just seems too big brother for the average user. It would enable much easier tracking of the web services and content that is linked to your profile. I think this would be incredibly convenient for managing your online history; the services you've used and the content you've generated.

Simple Formats

In the UNIX days, plain text was the format of choice for almost anything. Today, raw emails remain more-or-less human readable, but that's about it. Every new web service and application introduces their own formats and interfaces.

Maybe I'm just dreaming, but I think that with some work, text formats could work in the context of most popular web services. Twitter, for example, is simply a collection of text notes with the date as metadata. So why not integrate it with the desktop workflow?

The reason is that web browsers have been designed to be very isolated from the desktop, and the only way to break this isolation is with downloadable formats or task-specific applications that can bridge the gap. Again, email clients are the classic example of a task specic application that grabs a stream of information with a secret handshake and delivers content to the user.

What I envision here is a two-way RSS folder using a universal keychain and managed with something like Git. But what this requires is that the web service and browser be given read/write access to the folder. I think this model is fairly incompatible with the current state of web technology.

Summary

In some ways, I think that the values of the UNIX crowd are coming full circle with emerging web trends. ReadWriteWeb has been blogging for years about the potential of a web that's as easy to write as it is to read. This post is a good summary of emerging trends like structured data. But I think that to really acheive this flexibility, internet and desktop architecture needs to adapt and build robust standards to allow it to happen.

The problem isn't access to services, it's the completely sandboxed environment of the web browser. If we truly want to see the potential of web applications and services, we need a way to let the web in.

Browser Review: Arora

The Browser War is an ongoing battle with no obvious winners, although there is a seemingly never ending stream of competitors.

The big players at the moment seem to be converging on their basic feature set while edging out their opponents in a certain area. Firefox has extensibility, Chrome has speed, Safari has stability, Opera has security, and IE has users (yes, IE bashing is mandatory in any browser debate!).

Beyond these, there is an interesting subset of purpose built browsers: Flock does social browsing in an interesting way. Sea Monkey includes many tools specific to developers and advanced users. Dillo boasts an extremely small footprint and fast loading.

Arora

But the new browser I've been testing lately is Arora. It is still a very immature version, but I think some of the features are really well done. First of all, it looks really nice on Mac. This may seem like a silly comment, but most apps that have been developed with a "cross-platform" toolkit come out looking like cheap knockoffs with cartoon icons and ugly screen proportions. Even Chrome, with it's oh-so-clever tab bar and Firefox with it's monster icons. This is fine for Linux, which doesn't have clearly established UI conventions, but for Mac development: don't break the rules unless you have good reason.

In this sense, Arora does it right. Not that they use Mac icons, but they are minimalistic and clear, and the layout doesn't add visual clutter or break established conventions. It doesn't shout "Look at this groundbreaking layout!", and that's a good thing as far as I'm concerned. It let's you focus on the task at hand.

I downloaded the .10.1 version which had some serious bugs on the Mac: many SSL errors, no two-finger scrolling. But this is an alpha version so I gave it the benefit of the doubt. I installed the Qt SDK and built .10.2 from source and now it works great. Not really for the faint of heart, so hopefully there will be a Mac binary update soon.

Give me the Keyboard!

The first shock was that it was very keyboard friendly. This is like a +5 million points for me. The way it works is that when you're on a site, if you press the Command key all the links get letter popups beside them. You press the key and it goes to the link. How cool is that! Generally, all the links get letters based on the first letter in the link name, so if a site's top bar has links named "Home", "Reviews" and "Downloads", these will be mapped to the keys H, R and D. All other keyboard shortcuts work as you would expect on a Mac.

These guys have done their homework when it comes to keyboard accessibility. The only exception is using F11 for fullscreen, my macbook has F11 mapped to Volume-DN and fn-F11 mapped to Expose. Not a big deal, considering the wealth of things that they got right, even compared to the big browsers. For example, Cmd-L puts the cursor in the Location bar and Cmd-K jumps to the Search bar. In Safari, you have to Cmd-L to the location and then tab to the search for keyboard only input. Yes, it might be a small thing, but it's good attention to detail.

So far, there's really nothing keeping me from using this as my main browser, which is an amazing feat considering how lightweight and early in development it is. Page loading is fast and application loading is the fastest of any full-featured browser I've used. It renders with qt-webkit, which was a bit rocky to begin with, but has come a long way in a short time.

And you gotta love the polar bear on the globe icon!

Command Line Google

Google is available for the command line!

http://code.google.com/p/googlecl/

Install this and the gdata library and your good to go. This post is my first test of the blogger post command. I'm hoping that HTML publishes correctly. This is, of course, related to my previous post about command line productivity.

Now if I could install this on my Android phone...

New Templates!

If you've been paying attention, which you haven't, I've changed my site layout. Now it kind of looks like my www.andyvanee.com site and my amxor.blogspot.com site are kind of the same thing.

I did this by yanking the guts out of both and using inline HTML for everything. The template headers and layouts weren't very flexible in Google Sites or Blogger, so I finally scrapped them entirely and put my own header in the content section. I also found a little css hack to get rid of the Blogger nav bar. By what I've read it's not against the rules but, we'll see.

Also I've generated some nifty images for the headers by playing around with Gimp's IFS Fractal patch. What can't fractals do?

Applescript Mindbender

This is a crazy hack!!!

http://www.macosxhints.com/article.php?story=20060425140531375

How to have the same file contain both a shell script and a compiled AppleScript

% /usr/bin/osacompile -o shebang.scpt
on run argv
    argv
end run
^D

% chmod +x shebang.scpt
% echo \#\!'/usr/bin/osascript' > shebang.scpt
% ./shebang.scpt a b c
a, b, c

This puts a compiled applescript in the resource fork and a shell script in the data fork... mind exploded!

Why The Command Line?

Vaporware

The software arms race is heating up. At the forefront are apps, which are small, single purpose programs that are very stylish, but generally have very little functionality. The reason I say that they have very little functionality is because they are built as small standalone, self-contained programs designed for a very specific purpose.

The iPhone needs multitasking because every little bit of functionality is wrapped in it's own app. The average computer user has many windows open and many applications installed because we need many apps to complete one task.

In all these things, I believe that software developers have lost their way. The user is forced to multitask extensively and perform every task one step at a time.

The Killer App

For those that do not understand the UNIX command line, it looks like a deviously complex and foreign language. There are no visual clues about how to accomplish a certain task, no tooltips or interactive dialogs to walk you through.

What they don't see is thousands of purpose-built apps all designed to work seamlessly with one another and to be completely automated. Using a command line is not just a bunch of typing to achieve the same result as a mouse command line, it is a bit of typing to achieve what a thousand mouse clicks could do.

The standard question I get when people see me working in a fullscreen terminal is "Wow, are you a hacker?" No, I'm not a hacker in the sense you mean. I'm not decyphering encrypted passwords and finding loopholes in government databases or anything like that. I am a hacker in the sense that I'm not constrained by the functionality offered by graphical interfaces. I am a hacker in the sense that if I have repetitive or menial tasks, I'd rather let the computer do the work for me.

Learning the basics of the command line takes some work. Learning how to use the powerful text editors like Vim or Emacs takes some work. Learning how and when to use the versatile processors like Sed and Awk takes work. Learning how and when to tie these commands together into shell scripts or functions takes some work. But in all these things, it is work that pays off in higher productivity and far fewer hours spent repeating tasks that are better automated.

But the true value of the command line comes when you can effectively monotask. When you can focus on the task at hand and automate all the housekeeping. And when I say "housekeeping" I probably mean a lot more than you think.

For example, here's my current workflow for blogging:

1. blog why_the_command_line? >> opens a new file in my blog folder in the Vim text editor.
2. Type post using vim and the simple Markdown formatting.
3. %s/CLI/command line/g >> Replace all occurrences of 'CLI' with 'command line', cause I knew I'd be typing it a few times.
4. "publish why..." >> run the file through the Markdown to HTML formatter and email it to my blogger page.

I'm using two simple scripts here, one called blog and another called publish, and for most workflows that's all it takes. I now have a local copy in the easy to read markdown format and one on my blog in HTML format.

The value in this is partly the shortcuts, yes, but mostly the lack of context switching. There is no need for multitasking, because everything I need is a few keystrokes away, which allows me to focus. And that's a good thing.

Building Toolchains

A toolchain can be a very subtle thing.

We make them all the time, a lot of times without even realizing that we are doing it and very often without realizing that there might be a better way. In order to really use computers effectively, we must pay attention to the toolchain, or workflow that we are using.

A simple example could be task lists. Your toolchain could be as simple as: "Every todo list goes on a Sticky note". This works but it isn't terribly versatile. You have to manually clean them up when you are done with them, you can't access them from your phone or another computer, and there's no way of sorting them by date unless you sort them visually by placement.

This could be improved by using a spreadsheet or database like Excel or Bento. Then you have the option of adding priority, type and date fields and sorting them accordingly. You could then use Google Docs and make this spreadsheet accessible online. It's definitely better, but in my opinion it's a bit over the top just for a todo list.

My established toolchain is the task list in Google Calendar. It's simple; the only necessary info is a title which gets a checkbox beside it to check when the task is done. You can optionally add a due date, notes about the task and you can move them to another list or delete them when finished.

But the new toolchain that I'm playing around with is even simpler, it's simply a private Twitter feed. Instead of being limited to todo's, I post anything that I want to remember. I can delete the posts if it's a completed task or I can simply let it fade into the timeline. I can retweet if it's something important that might be getting stale and I can include links if it's related to something on the web.

This is a simple example, but I think it highlights a few important ideas. Anything you do with a computer can probably be done tens or hundreds of different ways. With that in mind, is your current toolchain the best for you? Some things to keep in mind when building a toolchain:

• Is the content properly accessible (over time and space)?
• Does the content need the draft/publish distinction?
• Is there a single endpoint (don't repeat yourself!)?
• Is there a concern about losing your data to a cloud service?
• Do you need a revision history?
• Is there other related tasks that can use the same toolchain?
• Is the endpoint shared, collaborative or private?

Many applications, including most cloud services, manage the data for you. This can be convenient for simple tasks, but can be troublesome if you want to extend or customize your toolchain beyond what the service offers. It's wise to think for a moment whether such programs should be the single home to your content, or whether they should be an endpoint of a larger toolchain.

When I started using blogger, I used the online editor, publishing as I went. After a little while I thought, "Hmm, it would be nice to have a local copy of my blog." It is possible to export your entire blog, but this is not practical on a day to day basis. The better solution is to use the cloud service as an endpoint, not the only point. By working on a local copy of my blog and publishing it to Blogger, I'm able to do some things that I can't with the Blogger tools. I now use Markdown formatting to simplify markup, and I can archive this version with version control using Git.

It's not a complex toolchain, but it's nice to know that I have a platform neutral copy of my blog that I can reuse however I see fit.

Apple's Flash Killer

There has been a lot of buzz lately about Apple's sudden and powerful push for HTML 5 adoption in place of Flash content. I think that the real issue is lost in most of the discussion. Here's my take on why they're doing it.

1. The "Silver Bullet" Toolchain

Apple has been moving a lot of it's development tools and platforms towards web-based models. Dashboard widgets are all built with HTML/CSS/JavaScript, embedded iTunes rich content such as the iTunes LP format are all thinly wrapped web pages. The only thing standing in the way of the web-based design flow becoming the ubiquitous silver bullet is rich, interactive content. For that you need Flash, and Apple is not willing or able to integrate Flash development into Dashcode's toolchain.

2. Integrated Advertising

Flash content bogs down page loads, and advertisers have paid big money to produce those bandwidth hogs. The average user doesn't come across enough useful Flash content to warrant it's use, especially on mobile devices. This is a big problem for advertisers who spend huge sums of money on dynamic content only to be blocked by a rapidly expanding portion of the market.

Apple's HTML 5 showcase is unambiguously directed towards advertisers. Apple is, in a sense, scrathching their own itch here. They are at the forefront of "cool" advertising, and they don't want to rely on a third party platform to deliver their slick visualisations.

3. Big Content

Apple has emerged as one of the major digital content providers in the world. They have done this by ensuring that content providers can deliver their content through Apple's platform in a safe and very profitable manner. Adobe has not offered content providers equal assurance with Flash content.

YouTube has already announced that they will be offering streaming movie rental options. I'm sure that Apple has plans to do similar things with streaming H.264 content, but they have no intention of doing it within a Flash container.

Thought Experiment: JavaScript POSIX Shell

Okay, this one is a bit of an odd idea. My thinking is that you build a shell that will work in a browser window. The 'filesystem' is created in the browser's localStorage. Only the basic IO system and OS facilities are hardwired, the rest is implemented through executable 'files' in the filesystem.

The general idea is to make a lightweight shell that is programmable and could do some really cool stuff.

echo "Hey, I'm in a browser terminal!" | twitter

I've already hacked together a demo just to see how all the parts would work. Here are my findings.

User Interface

A text area is editable or not, whereas a terminal window is only editable at the cursor point. I puzzled over this one for a bit, wondering how to mimic this behavior in a browser, and then I dropped it and just did what was convenient: one editable textarea for commands, one for output. This is what that looks like:

At first I thought this was kind of a lame hack, but as I use it a bit more it feels quite intuitive. It's like posting on Twitter; you type on top and the command gets pushed onto the stack. But, you might ask, what about programs that take over the full terminal window? Well, in this case you just pass the lower window to the program and keep the upper for commands to the program. In the Vim editor, and most others, you always have a status/command line used for issuing commands anyway, so why not keep it on top?

Calling Convention

This is the one place that I wanted to stay very UNIXy. The commands are called with arguments and have access to stdin, out and err and everything is a 'file', meaning everything is a key/value pair in localStorage.

Files

In my first draft, the files are just key/value pairs with the file name being the key and the text being the value. There are a few problems with this.

First, file permissions/types are not implemented, which is mostly a problem because there needs to be a mechanism to determine what are executable files. Remember: everything is a file; all the commands 'echo, emacs (!)' will be JavaScript strings stored in the browser. Any user can make their own executables.

Second, a file heirarchy would be nice. Timestamps would be nice.

I think the best solution is to format the 'name' string with all the metadata, which would get stripped when accessed by the Terminal. So a file like /bin/echo would become < name='echo' path='/bin/echo' mode='-r-x' > if we want to use XMLish formatting.

Summary

I really like this idea. When I first got it up and running I was a bit giddy. It is browser based, but I also want to make it very Internet based. I would like to make it possible to wire in any site that has a public API; Social Networking, Mail, RSS, the works.

Skratch Development Update

"Make one to throw away"

I forget who said this, but it has been good advice for my little web-based sketching app called Skratch. After the first version was up and running I took a step back and had a hard look at the overall structure.

Overall it wasn't too bad. It has been a work in progress to figure out how to modularize and encapsulate. In the alpha stages it was a bunch of global functions and variables going every which way. By version 0.1, I had realized that I needed to introduce some structure and so I grouped the drawing code into objects with attributes. Most of the event handling code was still in global functions. At this point I took a break and did some reading.

First I read Douglas Crockford's JavaScript: The Good Parts, and none of it really sunk in. It's an excellent book for the theory of how to use the language functions and how to avoid bad practices, but it doesn't really give the reader a sense of how they should write JavaScript. All of the examples are very short and specific to a certain language feature.

Then I read the jQuery source code and realized that I hadn't got it at all. It took me a while to really grasp what was going on here, and most of that time was spent trying to figure out what it was that I didn't understand. The thing that I didn't understand was simple but elusive: closure.

I have heard of closures as a programming concept a number of times but I never really grasped why they would be practically useful or how I would recognize one when I saw one. Reading the jQuery source, it took me a while to realize that this was a ~6000 line closure staring me right in the face. So I started coding to see how I could use an anonymous function to my benefit.

It's really quite simple: any vars, functions or objects declared inside a function are only accessible within that function. So if an anonymous function invokes itself, it's totally possible to not add to the global namespace at all. Nothing within that function is accessible unless it is attached somehow to the global namespace. But those functions that ARE attached to the global namespace can make use of all the private variables and state within the anonymous function.

This is the little testing function that I hacked together to prove this to myself:

(function () {
    var obj = {}, 
        privateData = 42,
        privateMethod = function (arg) {
            return "Private method: called with "+ arg + 
                    "\n" + "The holy grail: " + privateData + "  sees:" + obj.globalData;
        };
    
    //Priveledged: public methods which rely on private members.
    obj.protectedMethod = function (arg) {
        return "Priveledged method: called with " + arg + "\n" +
                    privateMethod("really " + arg);
    };
    
    // Public: members that are publicly accessible, rewritable, etc...
    obj.globalData = "I am the viewable data of the globalobject.";
    
    obj.generateString = function () {
        return "Global method: " + obj.globalData + "\n" +
                obj.protectedMethod("do this");
    };
    genericObj = obj;
})();

Only in the last line do I "publish" the object I've created, anything else is encapsulated in the closure and remains private.

Once I grasped this concept, it was immediately apparrent how it would be useful to my drawing app. I created an object, called "skratch" and attached only the functions and data that need to be accessible from outside to it; basically just the UI controls and a .setupCanvas method. The drawing engine and event handlers are now completely wrapped inside the closure.

This way of programming seemed kind of ugly to me until I looked at the DOM tree for my app. There was only one global object called "skratch" with a couple methods for triggering UI events and that was it.

So, I now know what closure is and what it can do for me and I think my scripts will be better for it.

Oh, and FYI, after reading the jQuery source, I couldn't help but include it in my app, even though all it does right now is make the toolbar go swoosh! Totally worth it!