HuffPost (formerly The Huffington Post) portal was launched May 9, 2005, and very soon it became extremely popular.
While one reason for the apparent success of this portal is that their contributors are popular journalists and well-known personalities from politics, business, and entertainment, the other reason is that the development team has been able to provide a technical solution that was up to the challenge.
To build a news portal capable of sustaining the heavy load, including peak load, and convenient for both visitors and authors of the entries.
Challenge – making definite entries appear among the results of running search engines queries, SEO optimization, simultaneous maintaining of a large number of users, quick generation of pages that contain dynamic and static data.
The portal is based on the MovableType platform, which has been modified to minimize the number of database queries and to improve performance. To load a server in order to process pictures, CSS, and other statistic content, Amazon S3 is used. To ensure stable and efficient work under heavy and peak loads we have used the memcache technology, which is a general-purpose distributed memory caching system that allows speeding up dynamic database-driven websites by caching data and objects in memory to reduce the number of times the database must be read. For subsystem where database queries were slow but memcache couldn’t be used because it wasn’t a persistent storage, the team used Redis NoSQL server.
Also, Akamai caching system was used to dramatically decrease the number of incoming requests approximately in 1 000 000 times, this system allows flexibility and functionality required to easily configure each page and create different caches for it.
QArea developers have implemented rather serious new functionality for this site: our own custom app, based on Google Maps API was created. It operates with a large number of images that were generated by our team. The app is fully capable of withstanding enormous volumes of data (~1 million rows) and can support a large number of users at the same time.
Several data storages were used by our developers in order to properly set up the comments system: first, they used MySQL with “nested set” data model, then comments system was moved to MongoDB and implemented as a web service. The new solution easily works with more than 100 000 000 records database.
Websocket Front-end Solution was used for reliable message delivery and adding latest news and comments to already loaded pages, aligned with Erlang-based server implementation. The system was configured for sending 150 000 messages per second.
A powerful statistics system has been implemented in the back-end: there is a statistics of tracking of entries, statistics of visits referred from different search engines. There is also statistics of the sites referring to huffingtonpost.com and general tracking of keywords/key phrases which can be used to refer the users to the site.
QArea’s responsibilities also included:
- Black-box testing
- Manual testing
- Automated functional testing based on Selenium
- Regression testing
- Documentation creating
The site has been featured on the main page of Yahoo several times. The site lives perfectly with 200+k visitors per hour. Also, we have created an iOS client for The Huffington Post news system.