We have a crisis of leadership in America because our overwhelming power and wealth, earned under earlier generations of leaders, made us complacent, and for too long we have been training leaders who only know how to keep the routine going. Who can answer questions, but don’t know how to ask them. Who can fulfill goals, but don’t know how to set them. Who think about how to get things done, but not whether they’re worth doing in the first place. What we have now are the greatest technocrats the world has ever seen, people who have been trained to be incredibly good at one specific thing, but who have no interest in anything beyond their area of exper tise. What we don’t have are leaders.

This is one of the finest things I’ve read in weeks.

A while back, I signed up for redbubble.com to buy a t-shirt or something. Right away I got this really nice (automated) email that actually seemed personal because it was actually “from” someone. It felt like I was communicating with a person rather than a faceless company. Why thank-you Jason Michaels for welcoming me to this community. But then I realized the ‘from’ address is ‘noreply’ and I felt duped.

I suppose this happens because of the way companies are organized internally:

• Marketing is measured by how many people they can engage in the company. So they like to send out emails.
• Customer service is measured by how many people DON’T engage with them. Any additional customer contact is an expense.

So the compromise is to send out emails, but do everything you can to keep them one-sided. I wonder how these companies are coping with their twitter/blogs/getsatisfaction ?

On a related note, MySpace no longer has a human face.

My own issue involved having a python based daemon running that does HTTP requests (using httplib) to another service. This daemon is restarted gracefully by sending SIGTERM, which it catches with a signal handler, finishes up what it was doing and exits. The problem arises if it receives a signal while in a system call, for example while receiving the response from an HTTP request. To correct behavior is to attempt the system call again, however the actual system call is abstracted away, so the caller, or even httplib can’t re-try.

The crux of the issue is the function readline() provided by a fileobject socket wrapper in socket.py

                self._rbuf = StringIO()  # reset _rbuf.  we consume it via buf.
                data = None
                recv = self._sock.recv
                while data != "\n":
                    data = recv(1)
                    if not data:
                        break
                    buffers.append(data)
                return "".join(buffers)

I’m not the first to find this, as this issue even has a patch. But, due to the “test needed” status, it’s been siting there getting no attention for quite a while. Well I want it fixed, so let’s try to write a regression test!

The first step was to apply this patch to an appropriate development branch:

  svn co http://svn.python.org/projects/python/branches/release26-maint python26
  cd python26/Lib
  patch -p0 < ~/socket.py.diff

Now it turned out, this didn't apply cleanly, as the patch was from an earlier version. But it was easy enough to fix.

Secondly, I need to a test case to Lib/test/test_socket.py
There is already a test case for normal behavior of fileobject, however causing a real socket to generate a EINTR isn't exactly easy. But I just need to test the error handling, this is unit test. Perfect case for using a mock object. Now there arn't any handy mock object libraries in the standard python distribution, so i'll just keep it simple:

        class MockSocket(object):
            def __init__(self):
                # Build a generator that returns functions that we'll call and return for each
                # call to recv()
                def raise_error():
                    raise socket.error(errno.EINTR)
                self._step = iter([
                    lambda : "This is the first line\nAnd the sec",
                    raise_error,
                    lambda : "ond line is here\n",
                    lambda : None,
                ])

            def recv(self, size):
                return self._step.next()()

Now when I create my test case, I'll just pass this mock socket in and call readline on it.

class FileObjectInterruptedTestCase(unittest.TestCase):
    """Test that the file object correctly handles being interrupted by a signal."""
    def setUp(self):
      ... create my mock socket ...

    def test(self):
        fo = socket._fileobject(self._mock_sock)
        self.assertEquals(fo.readline(), "This is the first line\n")
        self.assertEquals(fo.readline(), "And the second line is here\n")

Now to find out if this test case will allow this fix to be included......

So I was particularly excited to have these two events transpire today:

The iPhone Upgrade

ATT finally got their act together and enabled their premier site to allow online upgrades to the 3GS. Fantastic, thank you AT&T, you were only a couple of days late. Anyway, I placed the order, goes great, no problems. But then 2 minutes later I get an email:

Thank you for your recent purchase on the AT&T Premier Online Store. We’re pleased that you chose AT&T as your wireless service provider.

Unfortunately, we cannot validate your credit card number. Please call us at your earliest convenience with a different credit or debit card number so we can complete your order: 1-866-499-8008, Monday through Friday, 8 a.m. to 9 p.m. Eastern Time.

Oh great, 3 cheers for online ordering. Well I still want my phone so 2 minutes later I’m on hold listening to the same 8 bars of crappy jazz music playing over and over, interspersed with messages about how much they appreciate my business. Finally, after a full 10 minutes of hold I get to talk to a real person. Of course I have to tell them my phone number, again, I’ll never understand what the automated system does with the phone number I gave them earlier. She starts looking for orders and can’t find me.

“Sorry, it says you’re still elegible for an upgrade, so the order must have come through yet. Can you call back again in about an hour?”

Somehow I feel better sending this email reply the no doubt automated email address:

I just called, but they couldn’t find the order.
This is really poor customer service. I order stuff online all the
time, why don’t other companies have this hard of time. I’m trying to give you money!!

The Amp Assembly

My JBL Amp Assembly blew up a few weeks ago. Thankfully, I can buy a replacement part for only $120. So I called up the parts department today, part number in hand, excited to watch Jurrasic Park again and rattle the windows.

“Let me place that order for you…….Oh, sorry sir, my computer just crashed”

“Hmm ok, that sucks.

60 seconds later

“Sorry sir, I don’t think this coming back anytime soon. Can I call you back ?”

Why is this commerce stuff so hard!?!

It’s very impressive how quickly after the 1 year warranty on the electronics of my JBL Venue SUB12 decided to melt down. Not sure when it happened exactly, but one day I thought to myself, “self, there isn’t much bass going on here anymore, is the subwoofer still on ?”

RIP Subwoofer, Sept 2007-Sometime in the spring of 2009

The memory that will stick with me the best on this trip though is the hour or so Britt and I spent trying to track down a decent sweatshirt. I always assumed that Disneyland was designed for you to spend a whole lot of money once you got into the park. They have this nifty Fastpass system where you can get a pass to come back to a ride, so instead of waiting in line you can go shop.

HOWEVER, it was really difficult to spend money here. Nothing was quite right. Honestly, with so many shops open, all over the park, outside every ride, there was a real shortage of choice. Even though all the shops have a sort of theme, usually related to where in the park they are, 80% of the merchandise was the same in every one of them. 

I found two kinda cool unique t-shirts:

I really dig these retro graphics. However, I’m not going to wear them on a t-shirt. I want a print or poster with these images. But of course, even looking through the Disney Art Galleries, they don’t have these ones available.

Disney has a ridiculous amount of IP to draw upon to create merchandise. The rides and unique parts of the parks alone should provide for no shortage of cool ideas for unique gifts you could get only at Disneyland. Somebody is REALLY dropping the ball here. 

I do have to applaud Disney for there forward thinking embracing of online retailing. They have licensed lots of their Disney characters and images to Zazzle so you can create your own custom merchandise. But seriously, it should be EASY to spend A LOT of money at Disneyland. But it isn’t.

I’ll admit, I was a little skeptical when I first looked at it, but since I hadn’t indulged in early adopter syndrome in a while I thought I would just go for it. So far, I’m pretty impressed. I chose to read, for free, Cory Doctorow’s book Down and Out in the Magic Kingdom at the suggestion of a friend of mine. Couldn’t have been a more perfect choice. One could argue that Diamond Age would be more appropriate, but as I had already read that seemed like cheating. 

So, two things:

1. Reading on the Kindle is great. I’ve pretty much been carrying it around with me everywhere I go. It’s not as easy to slip into a pocket as a small paperback, but it’s way more convenient for readers such as myself that have trouble dedicating to one book at time. Surprisingly, after a few hours of reading, the distractions that I thought I wouldn’t get over, including the page turning (weird blinking, kinda slow refresh) actually disappeared in preference to the new found form factor convenience. I like to read in strange positions, and not having to turn physical pages and hold them open really works well.
2. Down and Out in the Magic Kingdom is an awesome book. I’m not even a sci-fi fan except for the occasional Neil Stephenson, so I was blown away how much I loved this book. Also, the fact that I downloaded it for free direct from the author is not bad either.

The Problem

Imagine, if you will, daily log files of about 1.2 gigs (about 2.4 million lines). These log files are repr() python structures (and very easily translate into json). The information in them isn’t very important, but let’s say for example they detail clicks on a website.

We slice and dice this information in several ways for different reporting purposes. All said and done, I think we process these logs 3 times every night. They take about an 30-45 minutes a piece. The general methodology is to run through the logs totaling up certain values, mapping page types to number of events, etc. Once the counts are generated, we insert them all into MySQL. For some of these we end up wanting other rollup sizes as well… daily is are highest granularity, but often we roll into weekly and monthly versions as well. In practice, inserting in to the database is often the slowest part and tends to adversely affect other processes also using the database. Let me re-iterate that: Inserting daily and monthly rollup data into the MySQL is annoyingly resource intensive. I’m not even trying to put the raw data in to report on the fly.

Not an ideal situation, but it’s working for now.

Couchdb Solution

My theory was that couchdb could provide all these reporting functions in a much more flexible way than these custom reporting scripts / relational db could do. The hope is that I could just load the raw data into Couchdb, write my views and I’d be good to go. The big question mark was if couchdb was fast enough to make this feasible

The Setup

After some discussion with some helpful people in the couchdb user mailing list, I arrived at the following setup and performance tweaks:

• couchdb 9.0a<whatever trunk is> (allegedly MUCH faster than official released versions)
• Latest version of Erlang (5.6.5, apparently 5.1, which seems to be the default ubuntu install, is REALLY SLOW)
• Effective use of _bulk_docs (sorta awkward way to do uploads in batches. I chose a batch size of 1500 lines)
• Generated my own sequential doc ids (auto-generated ids are quite slow as they are not sequential, and we are living in a b-tree world)

I’m using a quad core opteron 2ghz machine, 8 gigs for ram. Storage is an XFS raid volume, but i’m trying to get more details on this, I *think* it’s some external scsi raid array.

There was some question as to how parallel processing would affect speeds. There are few possible setups:

• Single data loader
• Multiple data loaders, same db
• Multiple data loads, different dbs, different machines (merge with replication ?)

I tried the first two, but the 3rd is a bit more complex. I’d like to try it, but then again I’m really looking for a solution that is “good enough”.

The Result

Single threaded

Base line (running through the logs without inserting into couchdb) was 4:36

it took 33 min, 26 seconds to load 2.4 million rows. On disk, this took 959 megs (which is smaller than the log file the data came from). So that’s about 1200 rows per second.

Dual Loaders

Base line was 3:23.

Inserting into couchdb, I got it to 19 min, 16 seconds, or about 2000 rows per second.

Note that compaction (the process of reclaiming deleted space, making the datastructure as efficient on disk as possible) resulted in no space savings. It did take about 6 minutes to run though.

Conclusion

Though loading data into couchdb is just the just the start, I feel reasonably comfortable with my results. If having the data in couchdb is as flexible as I’m hoping, it should be fairly easy to convert these multi-step reporting projects into something a little more manageable (and scalable).

As for using couchdb in general, I’ve been pretty impressed. The whole thing is refreshingly simple. The JSON/REST interface is super easy to build tools around. Installation wasn’t really that hard, even with needing to install most everything from source for performance reasons.

The community has been quite supportive and knowledgeable…. albeit small. This couchdb project isn’t taking the world by storm quite yet, but it’s making a lot of progress.

Updates on actually using this data to come…….

So here is the key learning. If you are planning to create some business or other form of entertainment, you will need quality at some point to succeed. But what is more important than quality in the beginning is some intangible element that makes your project inherently interesting before anyone has even sampled it. That initial audience will give you the luxury of time to create quality.

I’ve heard this principle described in different ways in the software world: “Ship early and often.” But it’s interesting to hear this so well generalized.

On a related note, while reading this blog post at work, my CEO stood up and announced he just received a signed Dilbert cartoon from Scott Adams. More evidence, as Scott Adams says, that life could be an illusion.