iCalendar – Quirks From the Trenches

While working on Trello2iCal, an awesome webapp that schedules Trello cards on your calendar by their due dates, I worked a lot with the iCalendar protocol (or iCal). It was actually the biggest source of bugs and misbehaviors, me taking a very close second. It didn’t help that Apple decided to name their calendar app iCal adding noise to search results, which were scarce to begin with.

iCalendar itself has two version defined by two RFCs: RFC 2445 from 1998 and RFC 5545 from 2009. It’s a text-based protocol, which contains some metadata and a list of “components” that represent events, todos, alarms, and other calendar related structures. Each of these components is a list of “content-lines” which are mostly “key: value” lines. When you put these calendars online and constantly update them, like Trello2iCal does, it’s called a feed (much like an RSS feed). Here is an iCal feed with two events:

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sveder.com/trello_to_ical//EN
BEGIN:VEVENT
SUMMARY:Event One
DTSTART;TZID=UTC;VALUE=DATE-TIME:20120805T090000Z
DTEND;TZID=UTC;VALUE=DATE-TIME:20120805T090500Z
DTSTAMP;VALUE=DATE-TIME:20120918T230955
UID:4fe6645de5a5b15c4f51136ftrello_to_ical
DESCRIPTION:Card URL -nsome url
END:VEVENT
BEGIN:VEVENT
SUMMARY:Event Two
DTSTART;TZID=UTC;VALUE=DATE-TIME:20120129T100000Z
DTEND;TZID=UTC;VALUE=DATE-TIME:20120129T100500Z
DTSTAMP;VALUE=DATE-TIME:20120918T230955
UID:4f00df4fcf0b6aef7001bd90trello_to_ical
DESCRIPTION:Card URL -nsome_url
END:VEVENT
END:VCALENDAR

If I was a smartass, I would really question the decision to invent yet another textual protocol instead of using XML which is quite suitable for this. Maybe if the protocol was XML based it would have been easier to implement, thus making it more popular, which would have resulted in a bigger knowledge base and better tools. One tool I do have to point out that saved me a lot of time is the iCal validator (thanks Ben Fortuna!), but it only checks whether your iCal feed is valid according to the spec, not with the real world quirks below.

Below, I talk about the issues and quirks I ran into and their solutions or workaround, in no particular order.

  • iCalendar clients, especially Google Calendar, are very bad at refreshing their data. I’ve had reports of users waiting for over 24 hours for their calendars to sync up. There is a workaround but if you’re doing active planning, this is unacceptable. It’s a pretty easy issue to solve for Google Calendar – just add a button that a user can click to refresh, but I guess it’s not a priority. Outlook and Apple Calendar are a bit better, syncing a few times an hour, but AFAIK lacks a refresh option as well. I wish the protocol had a way to tell clients the maximal time between syncs.
  • Google Calendar refused to load iCalendar feeds before I put a robots.txt file, allowing Google bot access. I don’t think that makes much sense as this isn’t a bot but a protocol client (which is probably best called an “agent”).
  • While developing, I put version 0.1 as my iCalendar version mistaking it for the version of my app. Every iCalendar client accepted my feed except Apple Calendar who silently rejected it. Apparently, the only value they consider valid is version 2.0.
  • Apple Calendar was also the only one that didn’t parse the feed without a DTSTAMP field.
  • All Day Events – There are many methods that don’t work or are wrong:
    • WRONG: Adding one day to a date time object is not right – just make a 24-hour meeting. It doesn’t matter if you start at midnight and end at midnight either.
    • WRONG: Having the start date time the same as end doesn’t work either as it produces undefined behavior.
    • WRONG: Removing the time component and setting the end day and the start day as the same day – this is an “event on the day”, not a full day event. This should be used for birthdays for example.

    The RIGHT way to do it is to remove the time component and set the end day for a day later. If not for the first point, this would have been easier to test, but when refresh is a 5-step process, it becomes tedious.

    A mess of iCal Calendars and Events.

    A mess of iCal Calendars and Events.

  • New lines in the description – in this case Outlook is the outlier where encoding the line ends as “=0d=0a” (this encoding is probably a relic from Outlook Express in which this was the way to encode ASCII characters in emails) worked. The RIGHT way to do it is using “n”. No, not the ASCII n which is 0x0d, but the actual ASCII slash character “” followed by the letter “n”. Fun.
  • I was using the iCalendar python library. It is well maintained but looking at the code, it is pretty gnarly. It also doesn’t support Unicode in a sane way making me modify it in a few places that I’m now retroactively opening bugs for in their github repo.
    A current issue I’m having – both Apple Calendar and Outlook have no problem parsing the unicode I’m serving, but Google Calendar insists to turn them into question marks. I’m pretty sure the problem is somewhere in my HTTP headers, but I still didn’t solve it.

To be honest, I hope I didn’t come off as too critical of the protocol. Passing complex data like calendar events is a very hard problem – you need to take into account time zones, multiple people, multiple fields, recurring events, whole day events, free-busy times, event updates, event deletion, and a myriad of other scenarios. iCalendar is a solution, which is better than none, and a lot of the quirks can be blamed on calendar clients, but I still find this whole environment very unfriendly and mediocre at best.

Google Drive and Android Hackathon

Last week I attended a hackathon organized by Google Israel with some special guests – Google Drive’s Nicolas Garnier and Rich Hyndman and Nick Butcher from the Android team. The event started with a series of lectures on the first day and continued with a whole day of hacking on the second day and ended with the demos and the winner announcements.

The lectures were nice, but the two lectures I caught were pretty lackluster containing, mostly the same “themes” Google is pushing on Android developers this year. If you’ve seen some of the I/O lectures or have been to an I/O reloaded event you probably already know most of them (Project Butter, new notification features, compatibility with various screen sizes etc.). On the other hand the Hackathon part was amazing. About 30 developers showed up on the second day to code some Drive and Android apps, which is quite a lot considering it was a workday and a day before another major conference in Israel (Geekcon – the subject of my next two posts).

The work (hack?) room at the start of the day.

The work (hack?) room at the start of the day.

Before I start talking about the project I contributed to I want to highlight a few other teams and their innovative projects. There were plenty of teams to demo, so much so it took more than an hour, and there were three winner teams that won a goody bag with awesome Google swag. Unfortunately I don’t remember all of them but I’ll try to reach out to the guys that were there and create a list of people and projects.

Notable Projects

Password manager over Drive – A team of guys from onavo (one of Israel’s most successful startups) were probably the most impressive team. They really know how to work with each other and they coded furiously to create a secure password manager Android application that keeps your passwords safely encrypted on Drive. There is nothing more to say beside that they were one of the winners.

Onavo winning one of the swag bags

Onavo winning one of the swag bags

gdg-booktrunk – An interesting project that keeps track of the progress you made reading ebooks in you Drive folder and provides statistics and motivates you to read more. This project was started by Roman who I considered joining just for his sheer enthusiasm.

DogFooder

I arrived right on time for the team pitches in the second day so you can say I had the luxury of choosing the project I wanted to work on. I thought I’d hop between a few of them, but I ended up being stuck on some authentication issues for way too long and only worked on one project – I helped my fellow GTUG Organizer Ran Nachmany on a great idea named DogFooder. It is basically an artifact repository with great Android integration including push notification. The basic use case is having a web service that you send your APKs and metadata (like versions and release notes) to. The web service saves this to Drive and issues a push notification for subscribed devices, like QA’s devices or beta tester’s devices. They can then choose to download the APK and install it easily and quickly. This is a major pain point for mobile developers – propagating new APKs to test devices, and big kudos to Ran for thinking of this solution. In Zimperium we usually send the APKs but we have to send them to Gmail accounts because our mail server doesn’t allow attachments as big as zAnti (which is pretty small at 7mb). Only recently we started using a build server, but that still doesn’t solve the notifications and propagation problems. You can see how this can be very useful to mobile developers.

I joined Ran and wrote the web service and Drive integration but due to some Drive API quirks I only finished the code about an hour after the demos started, so we basically integrated it and jumped on stage two minutes later. Only by some incredible miracle did the demo work perfectly (typically – when we tried it again off stage something went wrong). We were also one of the winning teams, and we split the swag.

My half of the SWAG for being one of the winning teams.

My half of the SWAG for being one of the winning teams – a coffee mug and water bottle

I’ll save the technical details for another post because we still have some work to do before it is fit to be online, but hopefully we’ll get it into some alpha shape and put it on github soon because the guys at onavo really want to have this product, and so does Zimperium.