01.22.08

Camino 2008 Week 3

Posted in Camino at 2:31 am by Smokey

Week 3 was another big week for us…but maybe “hectic” is a better word for it.

• First off, we released Camino 1.6 Beta 1 and Camino 1.6 Beta 2 this week (well, sort-of).
  • We “soft-released” Beta 1 on Thursday afternoon (EST) to software update and planned to publicly announce the release on Tuesday. Late Friday afternoon we were able to connect the dots about some common crashes people had started reporting on Wednesday. Since we knew at that point a fix was available, we decided to ship Camino 1.6 Beta 2 ASAP with the fix for the crash in order to prevent another crashy beta. Thanks to all the team for the hard work that made this possible (more on that below)!
  • Beta 2 will be officially and publicly announced soon, but if you’re using Alpha 1 or a recent branch nightly, please use the “Check for Updates…” option in the Camino menu to let software update do work its magic. We’ve done some testing and it works (with a few pain points) in our personal set-ups, but the more testing we get, the better. Please let us know if you experience any problems.
• Mark Mentovai served as our fearless and tireless build and release engineer, as always. In addition, he spent some time working on early issues we’ve spotted in the software update process. He also devised a solution that will solve the problem where symbols are being stripped erroneously, which currently makes it extremely difficult for us to read crash logs; Stuart and I reviewed that patch and hopefully it will land soon.
• Stuart Morgan spent more time this week working on our preference panes, polishing the Web Features and Tabs panes. He also worked on cleaning up code related to the History menu and the site icon cache.
• Sean Murphy continued to work on follow-ups to the new OpenSearch support this week.
• Peter Jaros wrote patches for a couple of follow-up bugs from this summer’s AppleScript enhancements.
• Bryan Atwood made it through the super-review process with the patch to allow Camino to store (and access) credentials for multiple accounts on the same site, but some last-minute issues are keeping the patch out of the tree for now.
• Jeff Dlouhy continued working on tab drag-and-drop and posted a new WIP patch on Friday.
• I spent time doing reviews, checking in patches, and fixing bugs on the website again this week. I also took on the task of backing out, checking in, and watching the tinderbox for test results as we worked on an odd startup time regression related to changing the capitalization of “for” in the “Check for Updates…” menu item. Ultimately, we’ve decided that Interface Builder 2.5.6 (which ships with Xcode 2.5 on Mac OS X 10.4 and 10.5) creates .nib files that can be slower to load. (If the multiple-versions-support in Apple’s developer tools weren’t annoying enough already, this bug just compounded the problems.) In between, I also found time to update my patch for web-based feed readers, which is now up for super-review, and to create a new Universal version of the CaminoViewsPalette framework for our localizers.

Camino 1.6 Beta 2 by the (wall-clock) numbers

In the tradition of other “by the numbers” posts, here is a brief run-down of the time it took to release Camino 1.6 Beta 2 to software update. All times US Eastern Standard Time (EST).

18 Jan, 3:08 PM: Stuart mentions he’s gotten “two reports now of people with persistent gmail crashes,” which adds to the scattered noise about random branch crashes we’ve been hearing since late Wednesday.
18 Jan, 4:20 PM: Sam goes to set up Talkback reports for Camino 1.6 Beta 1 (which we had released to software update on Thursday afternoon), and I start comparing crash the logs that we have.
18 Jan, 4:35 PM: Sam announces this Gmail/random crash is already the “topcrash” for 1.6 Beta 1, and he informs us of the Core bug that covers the crash.¹
18 Jan, 4:39 PM: After verifying that the Core bug in question was the one causing the crashes we had crash logs for, we begin debating the course of action to take.²
18 Jan, 4:48 PM: We decide to release and begin debating what to call the new release.
18 Jan, 4:52 PM: We’re calling the release Camino 1.6 Beta 2, and I start updating the Beta 1 release notes.
18 Jan, 5:02 PM: Mark super-reviews the release notes and notes that it’s 5 PM on Friday.³
18 Jan, 5:05 PM: I check in the release notes while Mark changes the version numbers on the existing Camino 1.6 Beta 1 mini-branch and lands the Core patch on the mini-branch.
18 Jan, 5:20 PM: I file a bug on renaming the 1.6b2 flags in Bugzilla to 1.6b3 flags and a bug on updating the appcast and the website for the coming release.
18 Jan, 5:27 PM: Our release tinderbox cb-xserve01 begins building Camino 1.6 Beta 2.⁴
18 Jan, 6:12 PM: cb-xserve01 announces that the build is complete and has passed basic tests (42 minutes elapsed).
18 Jan, 6:14 PM: The 1.6b2 build is staged on ftp.mozilla.org.
18 Jan, 6:15 PM: Mark files the bug to get the release into Bouncer, mozilla.org’s release mirror management system.
18 Jan, 6:21 PM: We have verification the Gmail, Blogger, etc., crashes have ceased in the 1.6b2 build.
18 Jan, 6:33 PM: Mark leaves work and goes home for the weekend.
18 Jan, 6:37 PM: I have the appcast and all the website changes made locally.
18 Jan, 6:45 PM: Mozilla Corp build engineer Nick Thomas informs us the release is in Bouncer; we now have to wait for the changes to propagate to the mirrors.
18 Jan, 7:21 PM: Nick Thomas informs us that Bouncer is ready for our release.
18 Jan, 7:23 PM: I push the website and appcast changes to the live site.
18 Jan, 7:24 PM: Sam downloads 1.6 Beta 2 via software update!

Total time elapsed from the time we decided to release a new version to fix the Core crash until Sam downloads the release via software update: 3 hours 36 minutes.

Special thanks to Mozilla’s Nick Thomas for expediting the Bouncer work for us on a Friday before a long weekend (preventing hundreds, if not thousands, of crashes that would have happened in Beta 1 just this weekend) and to our own Mark Mentovai, who stayed at work for an extra hour and a half on Friday evening⁵ to get the build-and-release part of the 1.6 Beta 2 release done.

        

¹ Ironically, the bug had been filed nearly three years ago by a Camino user, but it had been rare until another Core change on Tuesday caused the crash to become fairly common on certain websites. ↩

² Since the patch to fix the crash had been appropriately reviewed and had landed on the MOZILLA_1_8_BRANCH that morning and since we had not officially announced Camino 1.6 Beta 1 (only “soft-released” it to software update), we had the option to pick up the fix for the Core bug and do a quick (re-)release to avoid a crashy beta. ↩

³ About half of the Camino team is based on the US East Coast (with the other half split between the US Midwest, California, and Europe), and those with real jobs at open-source-friendly companies usually stick around IRC and do checkins and release work only during business hours. ↩

⁴ The period between the checkins and the start of the build is filled by tagging the source code and configuring the tinderbox to make an official release. ↩

⁵ See footnote 3 above. ↩

Permalink