We launched Phase I of Active.com Realtime in order to prove a few concepts - namely A) that we could create an extensible infrastructure that efficiently taps into the activity happening on Active.com and B) that there was interest in this sort of thing. Before we launched, we knew the technology was sound, but we were pleasantly surprised to the positive reaction external and internal audiences had to the prototype. So, we embarked on Phase II with the aim of improving design and incorporating different types of data.
I'm happy to announce the availability of Active.com Realtime, Phase II at http://labs.active.com/realtime. Active.com Realtime is to showcases what people in a given city are searching for, registering for, and the results their viewing on active.com right now. The goal of the site is to help you make a decision about what event you might like to participate in by showing you what's popular in your area.
Here's a screencast explaining Active.com Realtime's features:
I read a blog post last year from Mountain Goat Software about the importance of feedback devices in an Agile workspace. They serve to reinforce a sense of "one-ness" when the team gathers together to solve mutual issues such as Production fires, or when builds are broken.
Here's what one of our teams started doing when emergency issues arise in our Production environment:
1. The Scrum Master for Team Ramrod (yes, it's named after a transformer) posts a sign at the entry point to the team's sitting area indicating to a would-be visitor that the team's busy fighting a fire.
2. And, in the spirit of visual queues, we put a virtual fire on the projector wall in the team's sitting area:
I put together a presentation this past week about building small, fast websites with Ruby and Sinatra, and then deploying them on Heroku. I had less-technical users in mind with this presentation. If you're a hardcore developer you have no problem setting up a web server from scratch. This is geared more towards designers who can make beautiful sites but aren't used to making them dynamic or pushing them public to the world.
We launched our iPhone app last week and it was greeted with less than stellar reviews. One reviewer complained of bugs, another about the lack of saving tab order, another about sluggishness. I wanted to explain the non-traditional approach we took when developing this app and what we're doing to improve the experience for those less-than-satisfied customers.
We wrote the first version of ActiveReader against a beta release of Titanium--0.8.3. Development of Titanium itself was happening very rapidly and new SDKs were being released every few weeks. Documentation was okay, but they had a thriving community which the developers participated in regularly. If you had a problem you could usually get it answered within a couple of hours. However lots of functionality was still missing from the API, and bugs were being found on a daily basis.
So, what have we done to get the app more stable? We were able to migrate the app from the 0.8.3 framework to the 1.0.0 framework over the past week and it is currently being reviewed by Apple to be added to the app store. We expect that to happen early next week at the latest. The loading times for the app are orders of magnitude faster. Previously you would be staring at the splash page for 10 seconds or more while waiting for articles to come down from the cloud. Now the loading screen is up for 1-2 seconds and you see a list of articles immediately. Viewing an article is faster as well.
We also removed offline access to articles for the time being. This is another of the major performance problems with the current version and we want to revisit the database storage issue in-depth. We felt that lack of offline access wasn't a deal-killer to getting this new version out the door in the hopes that it addresses most people's concerns.
It may turn out in the end that Titanium just won't fit our needs as a rapid iPhone application development environment. The Light Engineering Group's mission is to find new technologies that help us get to the market faster than ever and we hoped Titanium would be another way to do this. While it's true that we are able to develop apps quickly, it may not be worth the tradeoff of stability and lack of support. We may end up going back to the drawing board and instead build the app out from scratch using the good old fashioned tools that Apple intended for us to use in the first place. Time will tell!
We appreciate everyone that's downloaded the app so far and given great feedback on Facebook and Twitter. We won't leave you guys hanging -- we'll get you a great reader one way or another!
Active Realtime is the newest addition to our product incubator labs.active.com. Every day thousands of people are searching and registering for things to do at Active. The idea behind Active Realtime is to provide a window in to actual user activity, giving a sense of whats happening right now at Active. We also aggregate popular activities on a local level, which reveals favored events and activities in a region. We also have the opportunity to look at trending topics over time, which will perhaps yield new insight into what is keeping us Active.
Presently data for Active Realtime is provided by our newly launched search engine . In the future we hope to include realtime data from other parts of Active, such as event registration openings or community posts. Keeping this in mind, we designed a technical infrastructure for Active Realtime that would allow us to add additional sources of data in the future, as well as allow other applications at Active tap into the information.
To accomplish this, we used XMPP Publish Subscribe, an addition to the venerable IM protocol. Using XMPP PubSub, we are able to keep Active.com Search and Active Realtime loosely coupled, so each service can be independently maintained and scaled. Additionally, we developed a Search Logger, which acts independently of the Search site. The Search Logger acts as the Publisher, and pushes search data over the XMPP bus. Any interested part of the Active infrastructure may subscribe to receive these updates, and the Realtime application is just that. The Realtime architecture includes a Ruby daemon that acts as Subscriber, registering an interest for search activity and receiving data accordingly. This data is recorded into the Realtime database, which then can be displayed on the Realtime site. A diagram showing the architecture is included to the side.
The payoff for investing the time to create this architecture, rather than just querying the search infrastructure directly, is twofold. First, it provides a simple and scalable way for future datasource additions to be made to the Realtime application. Secondly, it de-couples realtime with each of the data publishers, allowing all systems to function and scale independently, without having an impact on one another.
I hope you enjoy this window into Active. The data on Active Realtime is generated by Active users, so perhaps another like minded person will help you to find a great local activity to participate in.
Active just officially released our first iPhone app—ActiveReader. This app lets you read all the articles from Active.com in a mobile-friendly format. It also stores those articles on the phone itself so you can still read them, even if you're offline. Articles are organized by activity like running, cycling, baseball, or you can search our site for articles.
This is only the first of many new mobile apps you'll be seeing from us. Stay tuned!
At 1:30 PM PST on 2 March, 2010 we flipped the switch and started serving www.active.com through Akamai. We then watched our average load time and bandwidth consumption stats decrease as the DNS change propogated across the world. The end result has been significant. Below are just some of the improvements we've noticed.
Active.com homepage load time reduced from 6 seconds to less than 1 second
Community homepage load time reduced to an average of 2 seconds
Datacenter bandwidth consumption reduced by at least 30%
Concurrent sessions on our webservers reduced by 45%
Hi, I'm Rob and I'm a Software Architect on the Light Engineering Group here at Active. LEG, as we call it, was modeled after the LED group at LinkedIn. LinkedIn was looking for a way to rapidly prototype new technologies and architectures with Ruby on Rails. Their first product was Bumper Sticker which at one point was the mostly highly trafficed Facebook application in existence! You can find many of the fruits of LEG's labor at http://labs.active.com
The Technical Stuff
The server itself is an Amazon EC2 instance running in their cloud infrastructure. This allows us to quickly react to any scaling needs by simply spinning up a new instance of the same server and deploying the code to it. Nearly half our division's products are now running on EC2 and we're moving more there every day. We've found it be an fast, secure and cost effective way to get our applications live.
Stay tuned for more exciting projects from the LEG team!
Following up on my post from Friday, below is a video showing what our daily Scrum meetings are like. Scrum meetings are normally 15 minutes, and you'll see we go into a "16th minute" conversation, where where we talk about topics peripheral to the daily Scrum meeting construct:
We took a good look at the way we developed software in 2009 and made the decision to implement fundamental changes to our process to improve product quality and focus in 2010. In 2009, we operated under a waterfall model - Design --> Build --> Test --> Deploy --> Operate. Some of us liked to think we were following a "Waterfall/Agile Hybrid", and this thinking was brought about from the fact that we'd combined elements of Agile into our "Build" phase that had carried over from a failed attempt to transition to Scrum in 2008. The only elements I could think of that carried over were burn down charts and daily standup meetings. But nobody liked the process as it produced a lot of waste (context is inevitably lost when work items are transitioned through the phases) and the business wasn't getting the product it wanted.
Enter Scrum in 2010.
We enlisted the help of an external company, Rallydev, to show us the way; to enlighten us on how to do Scrum and how we might build a better product. My aim, more than anything, was to convert the masses. I wanted them to become believers in the process, because if they believe they'll work to iron out the kinks during execution. But before Rallydev came to help us, we had to convince the executives that Agile was the way to go. I'd heard horror stories of Agile transitions not working because of a lack of executive buy in. So I brought in a buddy of mine and his CEO to talk to executive management within my division. My buddy, and his CEO, had successfully transitioned their company from waterfall to Agile, and they were exuberant about the outcome. After a great conversation my executives were convinced, and I bought 11 copies of Lean Software Development and had all of the senior leaders (including myself) read it. Soon we were all using phrases like "fail fast" and "conceptual integrity" and were eager to see how lean software development would impact our teams.
I put a business case together to invest in training, and in mid-January Rallydev came to educate us on the principles of Scrum. I wrote about that a few weeks ago. I flew in all of my North American Development Managers and invited all of the General Managers and future Scrum Masters to the RallyDev sessions. They gave us a great general overview of scrum and we spent one day walking through a Sprint planning session, which was key. My one complaint is that I was looking for something more perscriptive on how to do Scrum with off-shore resources. They gave us a few suggestions, but we were left to figure it out on our own, and I've detailed how we do it below. My team left the Rallydev session believing in Agile, as far as I could tell, and we were primed to make the switch to Scrum.
Since January, we've deployed Scrum to 5 of 7 teams, with 2 of those teams based in China. I came to the conclusion that one of the best ways to foster collaboration is to maximize the amount of hours team members are in contact with each other by aligning teams geographically. We created the 2 China-based teams with resources that had previously been embedded with North American teams and had never worked together (even though they sat 15 feet away from each other in the office in China). We also had to find Scrum Masters for these teams, and it was recommended to me by our China office coordinator that I use our China-based QA leads for the position, which I promptly did.
So, in China, we're running with 2 semi-autonomous Scrum teams who liase with a Product Owner in North America on a daily basis. They follow the exact same process that we do here in North America. And this frees us up from day-to-day management of off-shore resources, as they're managing themselves.
During my research, the most helpful thing I read was not Lean Software Development, which is more philosophical in nature, but Scrum and XP from the Trenches - an ebook that spells out exactly how one team does Scrum from a tactical point of view. Fittingly, I'd like to provide information on how it is that we do Scrum here at Active.com.
But before I begin I'd like to set the stage:
We have 7 development teams, 5 teams in North America, 2 teams in China.
Every North American team has at least one resource that works remotely, from home.
We have 9 resources that do QA. 2 of them are team leads based in North America. The rest are based in China.
Integrating Team Members
Team members must sit together. Period. Close collaboration is a key feature of Scrum, and it's amazing how well a thin, 6 foot high cube wall can subdue collaborative efforts. We made everybody move so that they were sitting together after Scrum training sessions. Remote team members are required to be on a Skype video call when there's only one remote member on the team, or a Skype call and tokbox video chat when teams that have more than 1 remote member (Skype only supports 1 to 1 video chatting). We tried a lot of video conferencing software, and tokbox, thus far, is the best for the money (i.e free). The verdict's still out, though, as teams that have the Skype/tokbox combo find the combo cumberson. I'd love to have the funds to pay for a Polycom V700 for each of my teams, but I'd also love to cruise around on a flying carpet. I've gotta live in reality. Here are a few photos of the video feature in play with my teams.
Our smallest team consists of a Scrum Master in San Diego, a Development Manager in Georgia and a Developer in North Carolina. I made them try the video setup for 1 sprint, and they hated it. Agile wants teams to have some control over their destiny, and since the team decided they wanted to discontinue video chat in their retrospective, I decided not to press the issue. But the other teams love it, and each voted to continue with video chat. Remote team members have also commented that they feel more like their part of the team as they're involved with small talk and can get a feel for what's going on in the office.
My next step with remote workers, however, is to build an Embodied Social Proxy, and we're already assembling a prototype.
Developers can test, but Testers cannot develop. As such, we don't have QA personnel permanently assigned to Scrum teams. During each spring, QA brokers determine where best to utilize their QA resources. For example, if team A is developing important, high-revenue features that require a lot of specialized attention, the QA lead will assign the bulk of his/her resources to that team leaving the others to do their own QA. Or the QA lead might choose to re-assign his/her resources each day. The point is that it's up to the QA broker to determine where to apply QA effort. Their responsibility is to attend all Scrum meetings and Sprint Planning sessions for their respective products and orchestrate their teams (which, again, are entirely based in China) on a daily basis through a Skype/tokbox call.
The diagram above shows the alignment of one QA team to 3 Scrum teams.
Most of the team operate in 3 week timeboxes (Sprints). The first day of each Sprint is always the Sprint Planning meeting. This is when the Product Owner (the guy with the vision and who's on the hook for P&L) determines which user stories the team will work on. Each team has its own Sprint Planning meeting, and in some cases Product Owners attend multiple planning meetings in a week as they have multiple teams. The Product Owner comes to this meeting with a prioritized "backlog" of user stories. And user stories are always in the following format:
As a [who] I want to [what] so that [why], for example
We've significantly reduced the waste we otherwise incurred through over-documenting requirements by using user stories. One of the big challenges for us was to get Product Owners to construct a backlog of user stories that met this format. The idea is that they're independent, negotiable, valuable to the end user and estimable. The key here is the negotiable part, where the user story is just vague enough that the Scrum team can negotiate scope to get the user story to fit into the Sprint. We use the term "barely sufficient" a lot, meaning let's do the least amount of work required to produce a shippable feature that we can learn from, and it's an art to learn how to negotiate correctly.
Sprint planning meetings follow the same format for all teams, that that format is:
Determine Delivery Team Capacity – each member of the delivery team (scrum master, QA, developers, product owner) comes to the meeting knowing what days off they’ll need during the next iteration. The goal is to determine how many hours each resource can dedicate to the sprint. Note, “focus factor” is then applied to the team’s capacity. For example, if a team of 4 is available for 480 hours over a 3 week period, a focus factor of 50% brings capacity down to 240 hours. I highly recommend we start with a focus factor of 50% for each team.
Team Estimation (see attachment) – The Product Owner explains his top priority user stories to the delivery team. The team then collectively decides whether each story feels S, M, L, XL, or XXL. Relative sizing stops once the team feels like enough work is represented to fill the time box. The delivery team then selects a user story with a size of 2 or 3 story points to use as a baseline. If the team has already done a sprint, they can use a previously-sized user story. If this is the first sprint, the team must use the poker planning technique on one of the user stories in the “S” category from the Relative Sizing step. The idea is to size all of the user stories that were relatively sized. The sizing unit is “story points” and is one of 1,2,3,5,8,13, 20, 40, 100.
User Story Decomposition – The delivery team then dives into each story and determines what tasks need to be completed before the story is “done”. Each task is given an estimate in hours and someone from the delivery team volunteers to take them on. As each task is determined and sized, the task owner’s capacity (from step 1) is reduced. When a given person’s capacity is 0 this is a signal that he is no longer in a position to commit to additional work. The team can then commit to the sprint plan as it stands or other team members can sign up for the remaining tasks until they too reach capacity.
Re-Sizing – Based on what was learned during User Story Decomposition, the team may feel that its initial story point estimates are now inaccurate. The team, through poker sizing, will re-estimate the size of each user story in the iteration using poker planning techniques.
Commit to the Sprint Plan -- After the team has gone through each of the stories (sizing, task breakdown, estimate tasks, determine task owners), ask for a "Fist of Five" (vote for consensus) on whether the team can commit to delivering the items within the sprint. If all members of the delivery team show a 3 or higher, the sprint is ok to proceed. If not, the team needs to work through the sticking points until a positive consensus is reached.
Backlog Sizing – If time permits, the team should run through the Team Estimation step for the next high-priority user stories in the backlog. The idea is to have baseline sizes (story points) for items expected to be worked on during the proceeding 2 sprints.
These meetings typically take between 4 and 8 hours. We end up writing all of our user stories and tasks on notecards, and they get placed on a board on a wall near the Scrum team:
We also create a virtual board on a paid website called Scrumy, which I highly recommend.
This happens even with our China-based teams. Our Sprint planning meetings happen over a 4 hour Skype/tokbox meeting with the Scrum team and Product Owner. It starts around 4pm PST and 8am in China. We've started off with a reduced focus factor of 25% for our China-based teams (focus factor is the percentage of time that each person dedicates toward the sprint. If a person works 8 hours a day, 2 of those hours are dedicated to the Sprint each day) to accomodate inefficiencies with communcation back with the Product Owner in North America due to the time difference, and because the process is relatively new.
The China-based teams also have a task board on a wall with sticky notes.
Each team has a 15 minute scrum meeting each day (usually in the morning). The China-based teams have it at 9am their time (5pm PST) so that our Product Owners can attend. We keep it short, each team member talks about what they did yesterday, what they'll do today and if they have any roadblocks. Follow-on conversation is left to the "16th minute", and people have the option of leaving for that conversation. At the end of every Scrum meeting, the Scrum Master updates the burndown chart and moves tasks along the task board.
Demo and Retrospective
The second to last day of the Sprint is dedicated to the demo and retrospective. This is a 2 to 4 hour meeting where the Scrum team demos the user stories they've developed to stakeholders and answers any questions they might have. A lot of good conversation usually comes out of this. After the demo, we kick into a discussion about the Agile process. This discussion covers:
What worked well - things we should keep on doing
What should change - things we should stop doing
New ideas - things we should start doing
Acknowledgements - kudos to people on the team
The retrospective is super interesting to me, as it allows me to observe how others are taking to Agile. This is a very iterative endeavor for us, and we're always "inspecting and adapting" as the Rallydev trainers put it.
We're still new at this, and we haven't deployed any significant enhancements to Production under this new Agile process, yet. But as far as process in and of itself goes, the teams and the Product Owners love it. Teams have bonded strongly together and they've become closer to the business by working directly with Product Owners. And the business always knows what it's getting through involvement and demos.
We're making a few changes this year, and one of those changes is a wholesale shift to Agile Software Development. And when I say "we", I'm talking about my division, Media + Marketing. We enlisted the help of Rally Software to lead us to the promised land. But in preparation, I had the Product Managers, General Managers, VP and Development Managers read Lean Software Development by the Poppendieks (husband and wife team). After that, everybody read Scrum and XP from the Trenches - a highly pragmatic guide to how one shop actually does Scrum. Scrum and XP from the Trenches is a great read.
This week, Rally came in for a two day kick off session - Agile 101. The idea was to evangelize to a team experienced with waterfall methodologies to make them believers. The picture on the left is of our training room. Even our breaks were time boxed. From where I sit, the session was successful.
So now we're left, on our own for the next two weeks, doing full-blown Scrum with one of our 7 teams. Rally's going to come back for a retrospective to help us refine our approach, and then we launch with the other 6 teams.
One of those 6 teams is based entirely in China (well, the Product Owner is here in North America). I'm flying out there in a few weeks to train them and make them excited about all of this.
If our transition works we'll be rolling out less features this year. But they'll be more focused and simple. Stay tuned.
I wrote last month that I would be speaking at the Business of APIs Conference on 16 November. It was an awesome venue, and Mashery really put on a good show. If you're interested in re-living my speech, check it out below:
I'm looking for an iPhone developer to help us step up our presence in the Apple App store and on iPhone mobile devices. I need somebody who can come up with a compelling visual design and make use of our APIs available at http://developer.active.com/docs/ to build this application. If you are or know of a good iPhone app developer, send me an email, jeremy dot thomas at active dot com.