专注于Magento开发与外贸SEO推广
Magento主题|Magento插件|Magento教程

日访问量2亿PV的YouPorn 色X网站架构

  • Erick Pickup, lead developer at YouPorn.com, presented their architecture in a talk titled Building a Website To Scale given at the ConFooconference.  As you might expect, YouPorn is a beast, streaming three full DVDs of video every second, handing 300K queries every second, and generating up to 15GBs of log data per hour.Unfortunately, all we have are the slides of the talk, so this article isn’t as technical as I might like, there’s no visibility at all on the video handling for example, but we do get some interesting details.

    The most interesting takeway is that YouPorn is a pretty conventional LAMP stack, with a NoSQL twist as Redis now replaces MySQL in the live datapath. Reminds me a little of YouTube in its simplicity.

    The second most interesting takeaway was the great switchover. Common wisdom says never rewrite, but in 2011 YouPorn rewrote their entire site to use PHP + Redis instead of a complex Perl + MySQL based architecture. And by all accounts the switchover went well. The site is 10% faster and they moved over 6 years of legacy data with no down time.

    Read on to learn more about the YouPorn architecture…

    Stats

    • Launched in 2006, acquired in 2011.
    • 100 million page views a day in 2008
    • 300K queries / sec
    • 100 Gb/s – 3 full  DVDs streamed every second
    • Logs 8GB-15GB of data per hour

    Stack

    • Originally written in Perl (Catalyst application with a DBIx::Class backend)
    • Now PHP-FPM (FastCGI Process Manager)  – is an alternative PHP FastCGI
    • HAProxy
    • ActiveMQ
    • Varnish
    • Redis
    • Nginx
    • MySQL
    • Syslog-ng
    • Symfony2 – a PHP Web Development Framework.

    Architecture

    • The great switchover. The entire site was rewritten in 2011 to use PHP instead of a complex Perl architecture and Redis instead of MySQL and ActiveMQ.
      • Targeted 200+ million daily requests
      • Moved  6 years of legacy data without downtime
      • New site is 10% faster.
      • The process took longer than expected:
        • Had to decide on which technologies to use
        • Longer than expected learning curve
        • Moving and restructuring data from MySQL to Redis
        • Staffing
    • HAProxy – provides load balancing, intelligent load distribution, and health checks
      • Maintain two pools of servers: a write pool with a fail-over to backup-Master and a read pool will servers except the master.
    • Varnish – reverse proxy helping speed up the site and reduce server load. Used also for cache management, edge side includes (ESIs), and health check on web servers.
    • Syslog-ng – collects data on page views and is used for view counters and related videos.
    • Nginx – high performance web server running PHP-FPM and acts as a external CDNs for static files like CSS, images and JS.
    • Symfony – fast and feature rich, comes with a large set of libraries.
      • Considers Symfony2 + Redis = Rapid Development + Scalable Website – a well balanced equation.
      • Loves dependency injection.
      • Choosing a web framework doesn’t mean killing performances as long as you use a solid architecture in front of it.
    • ActiveMQ – message bus designed for large sites, used for writes to MySQL and Redis.
      • They found it too rigid for a site undergoing constant changes and the gains did not justify maintaining a separate Java infrastructure.
      • Would like a do-over since the problems may have been more the developers than the tech.
    • Redis – a fast, open source, KV store, now the primary data source.
      • Updated in real-time.
      • Sorted sets are used for all lists
      • Pipelining,  executing several Redis commands with a single atomic command, is key for performance
      • After the switchover additional Redis nodes were added, not because Redis was overworked, but because the network cards couldn’t keep up with Redis.
      • All reads come from Redis
      • MySQL is used to allow the building new sorted sets as requirements change.
      • Append-only-file (AOF) format is used for incremental backups and is IO pain free
      • Consider using Redis as temporary cache in front of a Redis persistent store as a way to improve performance and reduce load.
    • MySQL – in a support role to feed Redis
      • Highly normalized because it’s not used directly for the site.
      • Some tables have 100+ million rows.
      • Used to populate Redis lists with new features
    • Both GIT and SVN are used, which doesn’t work well.
    • Doctrine is used in their CMS. Great tool that has saved weeks of development.
    • 网站2006年上线,2011年被收购
    • 2008年即达到每天1亿PV的浏览量
    • 每秒30万请求
    • 每秒推送100Gb的视频,相当于3张DVD的内容
    • 每小时生成8GB-15GB的日志

    架构

    • 之前用的是Perl
    • 现在用的PHP,以FastCGI方式运行于PHP-FPM
    • HAProxy
    • ActiveMQ
    • Varnish
    • Redis
    • Nginx
    • MySQL
    • Syslog-ng(https://github.com/balabit/syslog-ng)
    • Symfony2

    关于框架优劣之争

    经常见到关于框架优劣的争论,比较性能,比较功能完备性,等等。我们的观点是,能让你获得最高开发效率的框架,就是合适的。

    YouPxxn的例子说明,Symfony2在一个相对完善的技术架构里,完全可以支撑巨大的访问量。

原文http://highscalability.com/blog/2012/4/2/youporn-targeting-200-million-views-a-day-and-beyond.html

 

未经允许不得转载:Magento|Magento2|magento开发|magento插件|magento主题|magento模板|magento商城|magento维护 » 日访问量2亿PV的YouPorn 色X网站架构
分享到: 更多 (0)

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址

Magento开发|Magento咨询QQ283475615

联系我们