imapsync migration

May 21st, 2009, 5:29 pm PDT by Greg

A quick technical note… I just want to get this down so it’s findable later.

I was trying to move my mail from FASnet to the new Zimbra server. The Zimbra wiki suggests imapsync to migrate mail from one IMAP server to another.

When I tried that, imapsync insists on spidering every folder on the source server to see what’s there. Since FASnet is set up in its own particular fashion, that means it will look through every file in my home directory. That’s a lot of load on the IMAP server and something that caused the system admins to ask me nicely to stop.

To prevent the spidering, I replaced this line in the imapsync script:

my @all_source_folders = sort $from->folders();

with this:

my @all_source_folders = ();

As long as you specify a --folderrec (or --folder or --subscribed) and no --include, it will still work just fine. My command line was:

./imapsync --host1 imap.css.sfu.ca --user1 ggbaker --host2 imapserver.sfu.ca --user2 ggbaker --noauthmd5 --folderrec mail --exclude old --prefix1 mail --prefix2 fasnet 2>&1 | tee log

Everything seemed to work, and it looks like I have my email moved over. The line in the imapsync summary “Total bytes skipped: 36444” worries me a little. I wonder what those bytes were.

Alaska-bound

May 19th, 2009, 9:46 pm PDT by Greg

Having comfortably adjusted to pacific time, Kat and I are headed out again. This time is much more low key: we leave for an Alaska cruise on Saturday.

My parents came along with this idea some time last year. They have done the Alaska cruise before and wanted to take us with them. Fine by me.

I guess I don’t have much to say about the cruise other than “we’re going”. With the China trip two three weeks ago, I haven’t had much mindshare for the cruise. Plus, there isn’t much to plan: get on the boat and figure it out from there.

I’m planning to rent a 70–200mm lens for the cruise (since I don’t have a decent long lens). That should keep me occupied for a while, and get some nice pictures of the glaciers, etc. That was lens #2 on my recent Lens Wish List. I already bought lens #1, the 4.5mm fisheye. This is my attempt to stop myself from actually buying the zoom: I can play and get it out of my system.

Geotagging

May 15th, 2009, 1:28 pm PDT by Greg

Astute observers (or people I told about it) will have noticed that in our gallery of China pictures, clicking the links on the left “View Album on a Map” and “View in Google Earth” do cool stuff.

I promised I would write about how I did that, so I will. There are obviously other ways to geotag photos, but this is what I did.

The Gear

Before the trip, I had the realization that I didn’t have to get some kind of GPS receiver that connected to my camera: any GPS data was enough, as long as it was timestamped. The photos I take are timestamped, so if the GPS data is too, I can connect the two and figure out where pictures were taken (assuming the GPS and camera are close to each other).

So, before the trip, I picked up a eTrex Venture HC. It claimed to be high-sensitivity, so that sounded good. Also, it was on the cheaper side of GPS receivers and it records the track data I needed.

I had the GPS on a lot over the trip, and I think I ran through 4 pairs of AA batteries the whole time. That’s quite reasonable to my mind.

The cameras were my Rebel XT and Kat’s SD800. Basically, it can be any Digital camera with its clock set reasonably accurately.

On the Trip

When we were about to arrive somewhere, all I had to do was flip on the GPS and throw it back in my camera bag. Then, take some pictures. When leaving, turn the GPS off.

The GPS records its position (“tracks”) until I turn it off. (It’s important with a Garmin GPS to not “save” the track: that throws away critical time info.) Just turn it off when done.

Processing

The data can be pulled off the GPS to a GPX file with its own software or GPSBabel. Photos come off the camera in the usual fashion.

I couldn’t find anything I liked to get the GPS data and the camera time stamps together, so I did what I always do in these situations: I started writing Python. The job was basically to read the GPS data, read the timestamps from JPEG files, interpolate the GPS data, and write the position data back to the JPEG.

I threw what I have on SourceForge as “Geotag Merge“. I haven’t “released” it yet, so you have to grab the Subversion repository if you want to play. Sooner or later, I’ll find some best-practices for packaging Python-based applications and I’ll do a beta release.

I added the Gallery2 GPS module to my gallery to make it all work.

An Example?

Okay, look at this picture which, according to its time stamp, was taken at 2009-04-19 14:20:08 (Beijing time) = 2009-04-19 06:20:08 UTC. Looking in the GPX file extracted from the GPS, I see these entries:

<trkpt lat="40.360190626" lon="116.013850048">
  <ele>829.274414</ele>
  <time>2009-04-19T06:19:48Z</time>
</trkpt>
<trkpt lat="40.360201774" lon="116.013833955">
  <ele>829.274414</ele>
  <time>2009-04-19T06:20:11Z</time>
</trkpt>

Like all good XML, this is minimally-human-readable: two observations separated by 23 seconds, with latitude, longitude, time, and elevation.

So, I deduce that the picture was taken in between these locations (and in fact very close to the second). A quick linear interpolation, and we decide that the picture was taken at 40°21’36.72″N 116°0’49.81″E (and 829.27 metres above sea level). This is then written back as part of the image’s EXIF tag, and it can be picked up by any geotagging-aware photo viewer.

Pictures!

May 2nd, 2009, 6:13 pm PDT by Greg

We have posted pictures from our trip! They are organized by city, which seemed to make the most sense to me.

Note that they are (almost) all geotagged, so you can click the left-side links “View Album on a Map” or “View in Google Earth” on the album (or any subalbum) to see the pictures geographically. This was done with a handheld GPS and some cleverness. The tags seem to generally be within about a few metres of the true location where the photo was taken.

I’ll explain all of the geotagging stuff in a later blog post.

Edit: By the way, I rolled over the odometer on my Rebel XT on the trip: I took img_9999.jpg, then img_0001.jpg in the mini-three gorges. Ten thousand pictures on that camera over its lifetime. It now officially owes me nothing.

We’re back!

May 1st, 2009, 8:35 am PDT by Greg

We got back to Vancouver yesterday afternoon, safe and sound. That is all.

I temporarily blocked access to our “china” gallery while we get the photos up. Don’t want anybody getting an incomplete experience. 🙂