Russell Beattie’s Weblog

I was just checking out Amazon Video, and I absolutely love it. Like Hulu, it works without any problems on my Ubuntu box because it doesn't require any special software to download besides Flash, which Linux has. (Thank you wonderful people at Adobe! I will finally forgive you for the Sklyarov thing.)

Years ago when I was at Yahoo!, I worked down the hall from their media lab, where they were working with some cable companies to create a "10 Foot Interface" for the Internet. That's the distance from your couch to the TV, if you don't know what I'm talking about. They had a whole office decked out as a mini-living room as a test center, and *really* comfortable meeting place.

What's surprising to me, however, is that it seems people aren't waiting for a common 10 foot interface to arrive, and are just simply using their big LCD monitor as their TV, or plugging their laptop into their TV, browsing the web for the video they want, clicking full-screen and then watching. College students today don't bother much with a TV and DVD player in their dorm room - why? They have their laptops or computer screens. And apparently, many of the rest of us do the same. But we can't all be watching old episodes of Heroes on Hulu on our laptops while sitting at our desks, are we?

I have an old computer just plugged into my TV and we watch all the DVDs or downloaded videos on that. I've pretty much given up on any of the various Media Center apps out there, and there's a ton. Why? Because none really understand the web much, and there's more and more content that's available via a web browser. So I just use a infrared bluetooth mouse on my lap navigate around the desktop UI on my TV from my couch. I've bumped up the size of the fonts, so in general, it works pretty well, but it can be painful sometimes if the web interface is made to be rendered only with smaller sizes.

Here's the thought, you know how more and more sites out there are adopting a m.* mobile version, tailored for smaller screens? Why don't the video sites out there start adopting tv.* versions of their interfaces, tailored to people using the site from their couch? And there could be browser-extensions you could download to modify your User-Agent to notify those sites that you were, in fact, viewing the site from far away, so they could adjust automagically. Finally, this seems like a great opportunity for someone to create a proxy that did something similar to what I did with Mowser - dynamically adapting popular video web-sites interfaces for TV viewing (the flash players wouldn't need any tweaking I don't think).

We've already seen a little of this with the popularity of watching video with the Wii browser - some sites dynamically adapted to a 10 foot interface, like StumbleUpon Video - but I'm thinking about a more generic solution for all the PCs out there being used in the same way. If there was *any* sort of standard around this stuff, then the big cable companies might even be willing to put web browsers with Flash right in the set top boxes themselves, and then there'd be a whole new version of the web out there to tap into.

Just a thought.

-Russ

Update: Daniel Gerges just pointed out that Free.fr, a French ISP, has been promoting "Telesites", which is pretty similar to what I was thinking of. I don't read French, so I can't find much info about it, but I found their homepage here. It seems like they're kinda promoting special TV-formatting for all sites, but I think the first ones should definitely be all the video sites out there: YouTube, Vimeo, Hulu, etc.

Actually, just thinking about it - I'm pretty sure that Microsoft Media Center "plugins" are all just websites as well, it's too bad they don't make them more public for general use.

-R

Since I have the attention of some Pythonistas, I have another question about Python. Here's a basic command line script for posting an update to Twitter, not using any libraries:

import urllib
import urllib2
import httplib
import sys

if len(sys.argv) != 2:
    print "Please enter message"
    raise SystemExit

msg = sys.argv[1]

username = "user"
password = "pass"

url = "http://twitter.com/statuses/update.xml"

password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, "http://twitter.com/", username, password)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
data = urllib.urlencode({'status' : msg})
req = urllib2.Request(url, data)
f = opener.open(req)

if f:
    print "Update OK!"
else:
    print "Error updating..."

I got the bulk of the code from this script, and just stripped it down to the very basics. If I had to write it all myself by using nothing but the documentation, I wouldn't have figured it out. The honest truth though is that I still don't understand most of it. The question is, is that really the most basic way to POST to a site that uses Basic Auth or is there something simpler?

Compare the above to what I'd do if I were using PHP:

#!/usr/bin/php -q
<?php

    $status = stripslashes($argv[1]);

    $username = 'user';
    $password = 'pass';

    $url = "http://$username:$password@twitter.com/statuses/update.xml";

    $params = array('http' => array(
                  'method' => 'POST',
                  'content' => 'status='. $status
               ));
     $ctx = stream_context_create($params);
     $fp = @fopen($url, 'rb', false, $ctx);
     $response = @stream_get_contents($fp);

     if ($response === false) {
          echo 'Failed to post';
     } else {
          echo 'Success!';
     }

And if I needed more power/reliability, I'd probably use PHP's integrated Curl support. I guess that's technically a third-party library, but not really as the docs are on the main PHP site and it's considered sort of standard for most PHP projects. Example:

#!/usr/bin/php -q
<?php

    $status = stripslashes($argv[1]);

    $postfields = 'status='. $status . '';

    $username = 'user';
    $password = 'pass';

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, 'http://twitter.com/statuses/update.xml');
    curl_setopt($ch, CURLOPT_USERPWD,$username . ":" . $password);
    curl_setopt($ch, CURLOPT_HEADER, false);
    curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_TIMEOUT, 10 );
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);

    $data = curl_exec($ch);

    $http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);

    if($http_code == 200){
        echo 'Success!';
    } else {
        echo 'Failed to post';
    }

So the question is, what's the standard Pythonic way of doing this stuff? Does it really need to be so complicated as my first code sample above? Looking through the web, in the documentation and in books, it's not clear to me if there's a more straight-forward way of doing it that I can understand easily and re-use quickly in scripts.

Thanks,

-Russ

Update: It looks like httplib2 is commonly used and recommended. Jacob gave me a sample POST using it in the comments and it's clean and works well. I modified the above code using it to this:

import urllib
import httplib2
import sys

if len(sys.argv) != 2:
    print "Please enter message"
    raise SystemExit

msg = sys.argv[1]

username = "user"
password = "pass"

http = httplib2.Http()
http.add_credentials(username, password)
response = http.request(
    "http://twitter.com/statuses/update.xml", 
    "POST", 
    urllib.urlencode({"status": msg})
)

if response:
    print "Update OK!"
else:
    print "Error updating..."

Cool beans.

-R

Jonathan was chatting to me over IM today about memes on Twitter. I happened to be watching the Roomatic stream the other day, and saw the "Little Known Facts" meme spread before my eyes. It was interesting because though later people started linking to the Twitter Search page with that term, at first it was just sort of being passed from person to person, but because I was simply watching "Palin" updates live posted from everyone, I got to see it from above, so to speak, as it happened.

Anyways, the Roomatic stuff is written in JavaScript and trapped in a browser, so I thought it would be nice to convert it into Python so that I could re-use it if I wanted to. It's not difficult, especially because the original Summize API had a "since_id" param that you can pass to it which restricts the results to only the updates that haven't been returned yet, so you never have to worry about duplicates. This makes the script brain-dead simple to write.

Here it is:

from xml.dom import minidom
import sys, time, urllib

if len(sys.argv) != 2:
    print "Please enter a search"
    raise SystemExit

search = sys.argv[1]

id = 0

while True:  

    url = "http://search.twitter.com/search.atom?rpp=20&q=%s&since_id=%s" % (search, id)

    xml = urllib.urlopen(url)

    doc = minidom.parse(xml)

    entries = doc.getElementsByTagName("entry")

    if len(entries) > 0:

        entries.reverse()

        for e in entries:

            title = e.getElementsByTagName("title")[0].firstChild.data
            pub = e.getElementsByTagName("published")[0].firstChild.data       
            id = e.getElementsByTagName("id")[0].firstChild.data.split(":")[2]
            name = e.getElementsByTagName("name")[0].firstChild.data.split(" ")[0]

            print "> " + name + ": " + title + " [" + pub + "]"

    time.sleep(3)

A few things about that script:

First, it took a lot longer to write than you'd imagine, because I don't know Python that well yet. Most of the examples online use some sort of third party library, which I think stinks for learning about how the basic standard libs work.

Second, the basic standard libs for XML suck. Or rather, just stuck in the DOM/SAX past. Pythonistas need to check out PHP's SimpleXML to see how a nice, clean, usable XML lib should work.

Finally, I actually refactored the above code a couple times to make it smaller once I had figured out what I wanted to do. Originally, I did normal DOM processing of iterating all the elements and checking for NodeTypes, then I went back and cheated and used the minidom's getElementsByTagName() everywhere instead, which made the script cleaner and shorter, but is also sort of a really nasty thing to do, IMHO. Like I said, I didn't want to use a third party lib like feedparser or the JSON stuff which would have made it cleaner. But honestly, for something this simple I really shouldn't need to either.

Python's lack of ending braces still freak me out. I'm starting to get sick of "unexpected indentation" errors already.

Enjoy!

-Russ

I went by the local library today here in Menlo Park, which is adjacent to a large park, the community rec center and the police station. In the center of it all, next to the parking lot is the bulletin board I snapped a picture of above. It's filled with notices of local events, businesses, services, festivals, etc. There's a similar one nearby in the local bookstore as well, which has probably even more notices tacked onto it.

What always strikes me about these boards is that there's *always* stuff on there that I didn't know about.

I'm on the computer every day, all day and yet I know almost nothing about what's happening in my local area. Not only is it basically impossible to stumble upon the information, like I can while walking back to my car from the library, but even actively searching for it is normally fruitless. I'm sure most of the items on that board have some sort of site out there, or information online - but where ever it is, it's not organized at all.

There's just all this information out there that's still completely "under the radar" of the web. Being a parent, I'm always searching for interesting things to do with my kid, and am always amazed at how much information still travels through word of mouth. Not being particularly social, and not being a "Mom" - as apparently there's an underground network for mothers-only it seems - I don't hear about a lot of it until it's too late.

I started thinking about this stuff around the Fourth of July - I tried to look up where the various fireworks displays were going to be and found almost nothing but outdated info - community calendars long out of date, news items from years past, etc. Happily the day before the holiday, the local newspaper published a list of displays online and I was able to find a show. Where they got all that info, however, is a mystery. It's indicative of most local event information - it's all out there, just not searchable online.

And speaking of papers - I was in the barbershop the other day flipping through the free local daily and it was filled with ads for shops I didn't know existed, all within walking distance of my home. I've written a bit about this before, but it still amazes me that all these businesses with money to spend on ads are still giving it to newspapers instead of the web. I'm sure that local shoe shop paid $200 or more for a 2x3 inch advertisement shoved in the corner of page 7 of the paper that I just happened to look at while waiting my turn in the chair. It seems like a lot of money for them to pay for that random chance encounter that the person who happened to see that ad also happened to need shoes *and* remember the name of the shop (I do, but sadly I don't).

This brings up the Yellow Pages. You'd think by now that the utility of the Yellow Pages - you know, the book of business listings they give you for free every year - would be something that is just as good if not better online. But I find that if I'm looking something up - say trying to find where an appliance repair place close by is - I get more and better results from flipping through a book made of dead trees, than using Yahoo! Local or searching on Google Maps. I saw some people complaining about them the other day, and I think they're nuts. Or maybe they just think they get everything they need online and don't even bother to look at a Yellow Pages any more. But go open one up right now and you'll be amazed, believe me.

I wonder why it is that we just can't get it together when it comes to Local stuff online. (Or is what I'm talking about "hyper local"?) Maybe this is just a problem for suburbia, where stuff is so compact, and so overlapping, that there aren't clear places to go to find all the information you need. I'm sure in the small town where I grew up and my parents still live, it's a lot easier to determine what the big events are every month. But it seems that for any decent sized city and the surroundings, there's just not enough info, or too much as well. Either what you're looking for is not available, or it's a needle in a haystack.

I guess this stuff is really a combination of two things - local information, and calendaring. But hey, I've tried to see Craigslist to find local event information, and it's really hit or miss. I've also tried to use Upcoming.org or Eventful as well, and there's tons of things missing. I've also tried subscribing to local news sources via Topix, and that doesn't seem to do it either.

Part of the problem is that their databases are incomplete, another is the restrictions of being displayed on the web. A huge wine festival in Mountain View and a weekly meeting of a knitters anonymous club both share the same font-size and weight in a list of hundreds of other similar items. It just doesn't have that same feel like the notice kiosk pictured above, does it? Big posters show bigger events like festivals or shows, and regular letter-paper printouts with pull off tabs cut at the bottom show less important things. There's a clear difference in importance just by size alone, but then older stuff gets pushed farther back as new items get tacked on top, and even older items get faded by the sun as well. And the more stuff going on, the more chaotic and colorful the board becomes, calling more attention to itself. There's a lot of usability on a simple kiosk that's just inherent in how it works, and yet nothing I've seen has duplicated it online. (The closest thing I can think of is the Million Dollar Homepage).

This brings me back to that notion of "stumbling" upon a notice like you can with a bulletin board, or by chatting with the other moms while dropping off your kid at school. There's stuff happening all the time in every community, and yet it's so easy to miss it all and one day (like last weekend) you'll turn a corner and discover that all of downtown Palo Alto is closed to traffic for a street fair. What the hell? I didn't hear about that!

I wonder what the solution to this is? I wonder how local information, businesses, updates, etc. can be better integrated into our daily web browsing? There's been services out there that can pinpoint your location by your IP address for years now, and yet it doesn't seem to matter much. Even if I explicitly ADD that information to an online profile, I don't get much for it. For example, I go to Facebook and I still see crazy-ass ads about getting dates tonight with unbelievably hot women that are at least a dozen years younger than I am, rather than an ad for a shoe store around the corner from me like in that local free daily newspaper. Why?

In terms of stumbling on new local info, I can't imagine that there will ever be a site that monopolizes that stuff, even Google only has a ~60% market share and they're considered pretty universal in terms of websites. Maybe there's a site out there that does everything I'm looking for, but I just don't know about it - and that's obviously part of the problem.

It just seems like there should be a simpler solution to this stuff. Sites have been trying to do local information since the 90s - CitySearch for example, which is still as useless now as it was then - with no success. I wonder if there's some key ingredient - like easily accessible location information - that's been missing, or if there's been too *much* effort put in and in fact, like Twitter, the key would be to remove complexity instead?

Here's an idea for a bulletin board online - how about a Million Dollar Homepage style page where anyone can take up a certain amount of space on the page for their flyer or pamphlet for free, editable like a Wiki for anyone to use. I have no idea how it'd be regulated, or how to keep out spammers (maybe no external links allowed?) but it'd be interesting to see if something like that could take off. I've seen "city wikis" before which are like open guidebooks, I wonder if there's anything like that for events and things? Well, even if there are things like that, I haven't used them yet, and most likely neither has my neighbors either.

Which is really the key, as they're the ones who knows what's going on as I sure don't.

:-)

-Russ

Update: Got a note from Roger about Flyerboard from PaperG, which is used on the Metro Boston home page. It's very cool - just like you'd imagine an online bulletin board to look like!

There's a general misconception as to what, exactly, the "cloud" really is. I think most people imagine it as just a large cluster of computers, hooked up together as a large database or map/reduce storage system sitting behind a load balancer so that it can respond to requests as scalably as possible. That's definitely part of it, but there's much more to the cloud than that.

The cloud can be intelligent.

Don't think of it as just a large disk drive in the sky where we store MP3s, address information and our email, and sync occasionally with our various computers. The cloud - or really each cloud as there's many of them out there - is made up of hundreds or thousands of individual computers, each with the ability to do intelligent tasks.

Right now these tasks are inward facing, simply doing activities like spidering the web and indexing the results (Google), or they serve as places to flexibly host external applications (Amazon EC2), or maybe do in fact do little more than serve as a central syncing storage space (Apple MobileMe). But those individual clouds could be the basis of so much more. I'm thinking of intelligent agents running on all those linked computers, doing tasks that we ask them to do, monitoring the internet for us, processing data and responding to our requests both on demand, and on schedule.

The problem is that so far we can only create full applications that are designed to be hosted on specific instances of the cloud, such as custom clusters, or hosted services like Googles App Engine or Amazon EC2. Because each cloud is siloed off from the others, there's no way to use those clouds as a single entity. You have to use various websites to access different functionality, and hope there's some sort of data interchange format that can help these services communicate.

However, I can see a future where a standard for secure scripting emerges, so that just like my music or files can be stored and retrieved on any server, my scripted tasks and applications could be stored and run on any server as well. This was impractical before there were clouds of computers running virtualized OSes on multi-core CPUs. Nowadays however, scripts are relatively easily sandboxed, and we're quickly learning (with examples from languages like Erlang) how to divide up those tasks to run efficiently on multiple processors as well.

Imagine instead of relying on random one-trick-pony "Web 2.0" sites which may disappear as soon as their funding runs out, you get to own and manage your cloud-based tasks and applications yourself. Less like installing WordPress on your own hosted server, and more like having your own Greasemonkey or cronjob-style scripts running on one or more computers in the cloud. We already have similar systems already - our email or IM, for example, is a process running on the server listening for messages for us from other servers, before then contacting our local client. Imagine instead of these services functioning as dedicated applications, they instead were autonomous and addressable agents running in the cloud on our behalf. It's quite cool to think about and is the basis, I'm sure, for many a science fiction novel or movie already.

This brings me to what spurred me to write about this now - the demo of Mozilla Ubiquity released yesterday. In it, they combine somewhat natural language with predefined scripts that run in the browser that help with various common desktop tasks. It's a great demo, which shows a trend to extend the command line/address bar/search box beyond just the simple request-response paradigm that the web has used up until now. But it has its limitations both in terms of platform, and data. If I leave my desk, I leave all that functionality behind (and my effort in learning how to use it). In order to be truly "ubiquitous", those scripts will have to run in cloud.

I can easily envision a standard - call it OpenScripting - where tasks, agents, bots, or even full-blown apps written in a certain way can be run in any of the clouds supporting that standard. The equivalent of JavaScript in a desktop browser with the DOM and browser APIs, but running in a server somewhere. The scripts would have to be portable and secure, so that for instance, an agent monitoring a website in another country could be moved to a computer at the edge closest to that site, reducing latency and improving performance without compromising the computer on which it runs.

Well, much of that is probably still fantasy and easily a quite a few years off, but to my main point about the cloud: It can be intelligent, and that intelligence is what's so exciting about cloud computing, making it in my opinion, not just a buzzword, but really the future of the web and computers.

-Russ

Since it was in the news as it got purchased by Amazon, I just joined Shelfari and added some books that were sitting on my bookshelf. I'm not sure how useful this will be, but feel free to friend me and we can swap book recommendations. Not that I read much nowadays, but I try to get a book in once every few months.

Some people collect books for some reason in personal libraries, and then lug them around from place to place forever, but I think that's insane. I'm a book "giver", usually trying to get rid of books as soon as I read them. A nice virtual bookshelf like this will do me perfectly, actually. (And a library card for my local library, of course, in case I want to check the books out again.)

:-)

-Russ

Just a quick update... The second Roomatic version I whipped up a couple weeks ago using Disqus as the back-end was a fantastic failure. It was interesting to try, but generally useless to me and everyone else, so I reverted back to the old version which polled Twitter Search for keywords and displayed them as a chat room.

Within a few hours I was immediately rewarded with the following email:

I use the instant twitter chat all the time, it's great. Can you embed this on a site?

Thx Cindy

Sent from Cindy's iPhone : )

That's very encouraging to get such immediate feedback, as all I did was tweet the rollback last night and I don't actually know Cindy. Also, I only made a minimal effort to make Roomatic iPhone friendly, so it's cool she sent it from her mobile. Creating a widget is a good idea, though embedding Roomatic is insanely simple, actually, just using an iframe like I did at the top of this post:

<iframe src ="http://roomatic.com/%23dnc08" width="500px" height="520px" style="border:none"></iframe>

The source URL just can end with any "room" (i.e. keyword, including hash tags) and it'll display within that iframe. I added a style to get rid of the 3D frame, and it looks good on Firefox right now, but if you don't see it correctly, leave a comment with your OS/Browser of choice.

I think I need to do some more work making Roomatic more useful as a live keyword tracker, as that's mostly what I use it for. I personally never log in and use it as a chatroom as I had originally intended, but for keeping track of stuff like the Olympics or Stevenotes, I think it actually works quite well. I'll have to mull over some enhancements to that core functionality and add them in.

-Russ

I remember years ago, William Gibson decided to stop blogging after a while because it was anathema to his day job of actually writing novels. Basically (and this is something I often say to myself), if you're blogging, you're not working. And for a writer, it's even more apropos, as the activity is *almost* exactly like your work - it feels like you're writing, you are moving the same muscles both mentally and physically, but at the end of the day you've been blogging, not producing a novel.

I'm starting to realize that using Twitter has a similar relationship to long-form blogging as well. Once you've reduced that blog-thought down to 140 characters and tweeted it, there's little urge to write any more about it on your blog.

For example, I could have Tweeted this whole post as:

"Tweeting totally takes away that blogging urge from me... Once I tweet about something, it's like it disappears from my mind completely."

That is a pretty good summary, though without context, and without any sort of explanation as to why I feel that way. You can follow up a tweet with others filling in details, of course, but it doesn't work particularly well I don't think. The question is whether for thoughts that don't need a much explanation, is that really good enough? Are all these extra words I'm writing now really just bloviating?

I can definitely tell you one thing I've noticed, and the reason I actually want to blog more - Tweets have no archival value of any sort. If you haven't noticed, I've stopped doing the daily Tweet summaries. Not only that, I deleted all of the old summary posts completely from the site, as I was completely sick of wading through them trying to find stuff I had written. When I first turned the script on, I thought maybe after a while I'd be able to go back and check the tweets and glean some interesting tidbits of something I was thinking briefly a few months ago, but both in whole and in part there was little value there, so I wacked them.

This brings me to my point. If tweeting has such dubious historical value, and takes away my urge to blog, is it something I want to continue to do? I don't know... It works well as a communication tool, that's for sure. But unlike a good IM conversation, I rarely expand on ideas much using Twitter. If I jumped into a chat, and wrote some opinion, usually I'd get a response, or a push-pull of opinions which might spur me to write a nice, long post clarifying my ideas about that topic. That post could spur comments or other blog posts, and at the end of the day if I'm lucky, there's been something of archival value created that myself or others can search for later and learn from. When I tweet though, the thought usually just seems to go out into a black hole, never to be seen again.

Anyways, I figured I'd write this all out rather than just tweeting it so as to not lose the urge. (Well, that, and I like bloviating, actually. So there.)

:-)

-Russ