Skip navigation

NEED HELP?|

Active Product Development

5 Posts tagged with the cloud_computing tag

Active Realtime_1296838692026.png A few months ago Amazon contacted me about our usage of Amazon Simple Notification Service (SNS), a messaging platform that enables topic-based messaging between applications. At the time we were one of the biggest consumers of SNS, and they were curious about what we were doing with it.  I pointed them to http://realtime.active.com and explained our realtime architecture to them.  Intrigued, they showed a few of their engineers and later decided to write a whitepaper about Active.com Realtime and SNS.

 

Now, finally, that whitepaper is available on amazon.com.  Check it out at http://aws.amazon.com/solutions/case-studies/active-com/.  Here's an excerpt:

 

Active.com was looking for a way to analyze a user’s click-stream in  near real-time to deliver pertinent trending information in a timely  manner. One of the fundamental ways that Active.com enhances user  experience on its website is by understanding and anticipating user  needs- surfacing relevant content dynamically to users whenever  possible. This is reflected in the “Popular Near You” feature on the  homepage, or the “Events Near You” feature on the channel pages, such as  active.com/running.

 

I've gotta give props to Kevin over at Amazon for driving this whole thing.  And of course, I've also gotta give props to the two guys who built Realtime, Brian Levine and Rob Cameron!

1,061 Views 0 Comments Permalink Tags: active.com, cloud_computing, amazon, realtime

http://awsmedia.s3.amazonaws.com/logo_aws.gifWithin M+M we've deployed myriad products to Amazon Web Services, including photos.active.com, advantage.active.com, ihoops.com, mobileapps.active.com, widgets.active.com, realtime.active.com, search.eteamz.com and labs.active.com.  We've found that, with each product, we have to run through the same checklist of items to ensure continual up-time.  Some of these items are related to application architecture, while others are related to infrastructure:

 

Cloud Checklist

  1. Caching - Serving pre-compiled content is always the best way to reduce I/O and have a fast website.  Memcached and Apache's mod_cache are frameworks we commonly use.
  2. Load Balance Across Availability Zones -- AWS has a 99.95% uptime SLA, but only if applications are deployed across multiple availability zones.  As such, nodes in a pool should run in two or more availability zones.  And, as if it needs to be said, applications should be load balanced.
  3. Data Backup - The cloud doesn't magically solve data retention issues (but it does help).  Amazon's RDS offering is a great option when not wanting to deal with setting up backup policies.  Otherwise, these must be put in place just like they would in any given data center.
  4. VIP Monitoring - Like most companies with products in the cloud, we run some of our products in a proprietary data center, and others on Amazon.  But our monitoring system acts as a central repository reporting status on all systems.  As such, monitoring must be setup on VIPs (i.e. www.ihoops.com) and nodes, with alerting configured should anything act up.  Node monitoring is enabled by allowing traffic through from the source ip address of the monitoring system through Amazon's security group configuration.
  5. CPU and Memory Monitoring -  CPU and memory utlization should also be monitored.  UDP port 161 should be opened to the monitoring system to record SNMP traps.
  6. Human Support Process - When something goes wrong, somebody's gotta get paged.
1,483 Views 0 Comments Permalink Tags: architecture, cloud_computing, amazon, infrastructure

Active Realtime

Posted by BrianActive Mar 10, 2010

Active Realtime is the newest addition to our product incubator labs.active.com.  Every day thousands of people are searching and registering for things to do at Active.  The idea behind Active Realtime is to provide a window in to actual user activity,  giving a sense of whats happening right now at Active.  We also aggregate popular activities on a local level, which reveals favored events and activities in a region.  We also have the opportunity to look at trending topics over time, which will perhaps yield new insight into what is keeping us Active.

realtime-architecture2.png

 

 

Presently data for Active Realtime is provided by our newly launched search engine .  In the future we hope to include realtime data from other parts of Active, such as event registration openings or community posts.  Keeping this in mind, we designed a technical infrastructure for Active Realtime that would allow us to add additional sources of data in the future, as well as allow other applications at Active tap into the information.

 

 

To accomplish this, we used  XMPP Publish Subscribe, an addition to the venerable IM protocol.   Using XMPP PubSub, we are able to keep Active.com Search and Active Realtime loosely coupled, so each service can be independently maintained and scaled.  Additionally, we developed a Search Logger, which acts independently of the Search site.  The Search Logger acts as the Publisher, and pushes search data over the XMPP bus.  Any interested part of the Active infrastructure may subscribe to receive these updates, and the Realtime application is just that.  The Realtime architecture includes a Ruby daemon that acts as Subscriber, registering an interest for search activity and receiving data accordingly.  This data is recorded into the Realtime database, which  then can be displayed on the Realtime site.  A diagram showing the architecture is included to the side.

 

 

The payoff for investing the time to create this architecture, rather than just querying the search infrastructure directly, is twofold.  First, it provides a simple and scalable way for future datasource  additions to be made to the Realtime application.  Secondly, it de-couples realtime with each of the data publishers, allowing all systems to function and scale independently, without having an impact on one another.

 

 

I hope you enjoy this window into Active.  The data on Active Realtime is generated by Active users, so perhaps another like minded person will help you to find a great local activity to participate in.
2,622 Views 1 Comments Permalink Tags: active, architecture, widgets, leg, product_development, cloud_computing, labs, realtime, xmpp

Active Widgets

Posted by RobCameron.2.16b Mar 1, 2010

Hi, I'm Rob and I'm a Software Architect on the Light Engineering Group here at Active. LEG, as we call it, was modeled after the LED group at LinkedIn. LinkedIn was looking for a way to rapidly prototype new technologies and architectures with Ruby on Rails. Their first product was Bumper Sticker which at one point was the mostly highly trafficed Facebook application in existence! You can find many of the fruits of LEG's labor at http://labs.active.com

 

This morning we officially launched our new Active Widgets site at http://widgets.active.com This was a quick two week project to create an embeddable window into Active.com's thousands of registerable events. The idea being that the blog you write about running/cycling/baseball can show a list of upcoming marathons/track events/Little League games in your area. The widget is customizable as far as what activities it displays, where it displays them, whether or not the user can change those settings, as well as the size and color of the widget itself. Once your done just click a button and you're given a couple of simple Javascript tags to drop onto your site wherever you want the widget to appear. And since the widget is written out onto your page like any other HTML, you can further customize it using your own CSS.

 

widget.png

 

The Technical Stuff

This customization site (known as The Configurator) is a Sinatra application written in Ruby. There is no database—the data store is a single YAML file which defines a widget and what about it is customizable. The widget itself is written in Javascript and after minification with the YUI Compressor, the whole thing is 14K (for comparison, the Google homepage is 47K).

 

Since we planned on having the widget be embeddable on any website we couldn't use Ajax to pull in the event listings. Ajax is subject to strict cross-domain security policies set by your browser—you can only access servers in the same domain that the page itself was served from. This is quite a problem since search results are coming from http://search.active.com  The workaround for this issue is to make server calls with <script> tags and then wrap the resulting code in a function call known as a callback. When the <script> tag is added to the page the browser immediately executes it as valid Javascript. Since this Javascript contains a function call (the callback) it looks a function you've already defined with the same name and calls it, passing in the returned data.

 

The server itself is an Amazon EC2 instance running in their cloud infrastructure. This allows us to quickly react to any scaling needs by simply spinning up a new instance of the same server and deploying the code to it. Nearly half our division's products are now running on EC2 and we're moving more there every day. We've found it be an fast, secure and cost effective way to get our applications live.

 

Stay tuned for more exciting projects from the LEG team!

1,900 Views 0 Comments Permalink Tags: events, widgets, leg, cloud_computing, amazon, ec2, labs, ruby, sinatra

We recently rolled out a private Beta for our new Search solution.  Not only is it a good functional test for us, we're also experimenting with Amazon EC2 and S3 for hosting and data storage.  And from what I've seen so far I'm impressed with Amazon and the approach they've taken to cloud computing. 

 

Amazon has made it dead easy to provision new servers.  They've created a collection of webservices to integrate to for starting and stopping instances.  It seemed odd to me, at first, that I'd have to manage my infrastructure through SOAP calls.  But client-side tools like Elasticfox and now Amazon's own AWS Console make it easy to manage.  There's also a good selection of Windows and various Linux flavors to choose from when setting up servers (we're running mostly Ubuntu 8.1.0 in Labs).

 

We wanted to make our Labs infrastructure extensible so we can new pilot applications quickly, regardless of platform.  To do this we're setting an Apache server on an EC2 Ubuntu host that routes http://labs.active.com to the appropriate app.  Requests to the base URL will be sent to a CMS/wiki that describes the various things we're working on.  Search, which currently resides in the root, will soon be available at http://labs.active.com/search.  Any future pilot we rollout will be available, then, at http://labs.active.com/{pilot-name}.

 

We're also setting up "global" memcached and MySQL servers, so that any application deployed to labs might benefit from these services (we love memcached by the way). 

 

Each pilot application will be responsible for load balancing its requests.  The Search application uses HA proxy running on a dedicated Ubuntu server to route requests to a collection of Apache instances running mod_rails distributed across Ubuntu "workers" (we can scale horizontally here, pending load, by adding more workers).

 

So far (and it's been about 1.5 months) we haven't had any Amazon-caused downtime.  Personally, I think this is a game changer.

2,546 Views 2 Comments Permalink Tags: cloud_computing, amazon, ec2, infrastructure, labs