slightlyodd

s3 timeout errors with duplicity on centos 5.4

Posted on 13 April 2010

This is a highly specific and geeky kind of post, but I couldn’t find anything about this problem anywhere else on the net. Maybe by writing it here, I might help some other poor bastard who is having problems with duplicity on CentOS 5.4.

Just briefly, I had some really old version of duplicity installed – god knows which repository it originally came from but it definitely isn’t in any of the base CentOS repos right now. Doesn’t seem to be on RPMForge either. I mucked around trying to install it from an RPM downloaded from the duplicity project page, but was running into problems with librsync dependencies.

Thankfully, I decided to go looking for a repo that had a more updated version of duplicity. It turns out the Fedora EPEL repository has duplicity-0.6.06 in it, which is the second most recent version. Better! However, this version still has the S3 timeout bug:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 463, in ?
    with_tempdir(main)
  File "/usr/bin/duplicity", line 458, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 390, in main
    globals.archive_dir).set_values()
  File "/usr/lib/python2.4/site-packages/duplicity/collections.py", line 476, in set_values
    backend_filename_list = self.backend.list()
  File "/usr/lib/python2.4/site-packages/duplicity/backends.py", line 802, in list
    for k in self.bucket.list(prefix = self.key_prefix, delimiter = '/'):
  File "/usr/lib/python2.4/site-packages/boto/s3/bucketlistresultset.py", line 30, in bucket_lister
    delimiter=delimiter)
  File "/usr/lib/python2.4/site-packages/boto/s3/bucket.py", line 205, in get_all_keys
    body = response.read()
  File "/usr/lib/python2.4/httplib.py", line 460, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.4/httplib.py", line 509, in _read_chunked
    value += self._safe_read(chunk_left)
  File "/usr/lib/python2.4/httplib.py", line 555, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python2.4/httplib.py", line 971, in read
    s = self._read()
  File "/usr/lib/python2.4/httplib.py", line 947, in _read
    buf = self._ssl.read(self._bufsize)
socket.sslerror: The read operation timed out

The good news is that a patch exists for this bug (#505739) and it has been committed to the latest version of duplicity. Until the EPEL repo is updated with the latest package, it’s as simple as grabbing the diff file from that thread, navigating to /usr/lib/python2.4/site-packages/ and then running “patch -p0 < wherever_you_put_the_patch.diff“.

Then voilà! S3 backups with duplicity shouldn’t fail with that timeout error any more.