My own issue involved having a python based daemon running that does HTTP requests (using httplib) to another service. This daemon is restarted gracefully by sending SIGTERM, which it catches with a signal handler, finishes up what it was doing and exits. The problem arises if it receives a signal while in a system call, for example while receiving the response from an HTTP request. To correct behavior is to attempt the system call again, however the actual system call is abstracted away, so the caller, or even httplib can’t re-try.

The crux of the issue is the function readline() provided by a fileobject socket wrapper in socket.py

                self._rbuf = StringIO()  # reset _rbuf.  we consume it via buf.
                data = None
                recv = self._sock.recv
                while data != "\n":
                    data = recv(1)
                    if not data:
                        break
                    buffers.append(data)
                return "".join(buffers)

I’m not the first to find this, as this issue even has a patch. But, due to the “test needed” status, it’s been siting there getting no attention for quite a while. Well I want it fixed, so let’s try to write a regression test!

The first step was to apply this patch to an appropriate development branch:

  svn co http://svn.python.org/projects/python/branches/release26-maint python26
  cd python26/Lib
  patch -p0 < ~/socket.py.diff

Now it turned out, this didn't apply cleanly, as the patch was from an earlier version. But it was easy enough to fix.

Secondly, I need to a test case to Lib/test/test_socket.py
There is already a test case for normal behavior of fileobject, however causing a real socket to generate a EINTR isn't exactly easy. But I just need to test the error handling, this is unit test. Perfect case for using a mock object. Now there arn't any handy mock object libraries in the standard python distribution, so i'll just keep it simple:

        class MockSocket(object):
            def __init__(self):
                # Build a generator that returns functions that we'll call and return for each
                # call to recv()
                def raise_error():
                    raise socket.error(errno.EINTR)
                self._step = iter([
                    lambda : "This is the first line\nAnd the sec",
                    raise_error,
                    lambda : "ond line is here\n",
                    lambda : None,
                ])

            def recv(self, size):
                return self._step.next()()

Now when I create my test case, I'll just pass this mock socket in and call readline on it.

class FileObjectInterruptedTestCase(unittest.TestCase):
    """Test that the file object correctly handles being interrupted by a signal."""
    def setUp(self):
      ... create my mock socket ...

    def test(self):
        fo = socket._fileobject(self._mock_sock)
        self.assertEquals(fo.readline(), "This is the first line\n")
        self.assertEquals(fo.readline(), "And the second line is here\n")

Now to find out if this test case will allow this fix to be included......

The Problem

Imagine, if you will, daily log files of about 1.2 gigs (about 2.4 million lines). These log files are repr() python structures (and very easily translate into json). The information in them isn’t very important, but let’s say for example they detail clicks on a website.

We slice and dice this information in several ways for different reporting purposes. All said and done, I think we process these logs 3 times every night. They take about an 30-45 minutes a piece. The general methodology is to run through the logs totaling up certain values, mapping page types to number of events, etc. Once the counts are generated, we insert them all into MySQL. For some of these we end up wanting other rollup sizes as well… daily is are highest granularity, but often we roll into weekly and monthly versions as well. In practice, inserting in to the database is often the slowest part and tends to adversely affect other processes also using the database. Let me re-iterate that: Inserting daily and monthly rollup data into the MySQL is annoyingly resource intensive. I’m not even trying to put the raw data in to report on the fly.

Not an ideal situation, but it’s working for now.

Couchdb Solution

My theory was that couchdb could provide all these reporting functions in a much more flexible way than these custom reporting scripts / relational db could do. The hope is that I could just load the raw data into Couchdb, write my views and I’d be good to go. The big question mark was if couchdb was fast enough to make this feasible

The Setup

After some discussion with some helpful people in the couchdb user mailing list, I arrived at the following setup and performance tweaks:

• couchdb 9.0a<whatever trunk is> (allegedly MUCH faster than official released versions)
• Latest version of Erlang (5.6.5, apparently 5.1, which seems to be the default ubuntu install, is REALLY SLOW)
• Effective use of _bulk_docs (sorta awkward way to do uploads in batches. I chose a batch size of 1500 lines)
• Generated my own sequential doc ids (auto-generated ids are quite slow as they are not sequential, and we are living in a b-tree world)

I’m using a quad core opteron 2ghz machine, 8 gigs for ram. Storage is an XFS raid volume, but i’m trying to get more details on this, I *think* it’s some external scsi raid array.

There was some question as to how parallel processing would affect speeds. There are few possible setups:

• Single data loader
• Multiple data loaders, same db
• Multiple data loads, different dbs, different machines (merge with replication ?)

I tried the first two, but the 3rd is a bit more complex. I’d like to try it, but then again I’m really looking for a solution that is “good enough”.

The Result

Single threaded

Base line (running through the logs without inserting into couchdb) was 4:36

it took 33 min, 26 seconds to load 2.4 million rows. On disk, this took 959 megs (which is smaller than the log file the data came from). So that’s about 1200 rows per second.

Dual Loaders

Base line was 3:23.

Inserting into couchdb, I got it to 19 min, 16 seconds, or about 2000 rows per second.

Note that compaction (the process of reclaiming deleted space, making the datastructure as efficient on disk as possible) resulted in no space savings. It did take about 6 minutes to run though.

Conclusion

Though loading data into couchdb is just the just the start, I feel reasonably comfortable with my results. If having the data in couchdb is as flexible as I’m hoping, it should be fairly easy to convert these multi-step reporting projects into something a little more manageable (and scalable).

As for using couchdb in general, I’ve been pretty impressed. The whole thing is refreshingly simple. The JSON/REST interface is super easy to build tools around. Installation wasn’t really that hard, even with needing to install most everything from source for performance reasons.

The community has been quite supportive and knowledgeable…. albeit small. This couchdb project isn’t taking the world by storm quite yet, but it’s making a lot of progress.

Updates on actually using this data to come…….