• 欢迎访问magento4you博客,我们提供magento建站、技术咨询、IT运维、支持培训、二次开发服务及模板定制,联系QQ:283475615
  • Magento入门教程 二次开发 模板定制 联系QQ:283475615
  • magento运营 SEO SEM 数字营销 广告竞价 联系QQ:283475615
  • 如果您觉得本站对你有帮助,那么赶紧使用Ctrl+D 收藏吧!

日访问量2亿PV的YouPorn 色X网站架构

PHP magento4you 5年前 (2017-06-28) 4110次浏览 0个评论
  • Erick Pickup, lead developer at YouPorn.com, presented their architecture in a talk titled Building a Website To Scale given at the ConFooconference.  As you might expect, YouPorn is a beast, streaming three full DVDs of video every second, handing 300K queries every second, and generating up to 15GBs of log data per hour.Unfortunately, all we have are the slides of the talk, so this article isn’t as technical as I might like, there’s no visibility at all on the video handling for example, but we do get some interesting details.

    The most interesting takeway is that YouPorn is a pretty conventional LAMP stack, with a NoSQL twist as Redis now replaces MySQL in the live datapath. Reminds me a little of YouTube in its simplicity.

    The second most interesting takeaway was the great switchover. Common wisdom says never rewrite, but in 2011 YouPorn rewrote their entire site to use PHP + Redis instead of a complex Perl + MySQL based architecture. And by all accounts the switchover went well. The site is 10% faster and they moved over 6 years of legacy data with no down time.

    Read on to learn more about the YouPorn architecture…


    • Launched in 2006, acquired in 2011.
    • 100 million page views a day in 2008
    • 300K queries / sec
    • 100 Gb/s – 3 full  DVDs streamed every second
    • Logs 8GB-15GB of data per hour


    • Originally written in Perl (Catalyst application with a DBIx::Class backend)
    • Now PHP-FPM (FastCGI Process Manager)  – is an alternative PHP FastCGI
    • HAProxy
    • ActiveMQ
    • Varnish
    • Redis
    • Nginx
    • MySQL
    • Syslog-ng
    • Symfony2 – a PHP Web Development Framework.


    • The great switchover. The entire site was rewritten in 2011 to use PHP instead of a complex Perl architecture and Redis instead of MySQL and ActiveMQ.
      • Targeted 200+ million daily requests
      • Moved  6 years of legacy data without downtime
      • New site is 10% faster.
      • The process took longer than expected:
        • Had to decide on which technologies to use
        • Longer than expected learning curve
        • Moving and restructuring data from MySQL to Redis
        • Staffing
    • HAProxy – provides load balancing, intelligent load distribution, and health checks
      • Maintain two pools of servers: a write pool with a fail-over to backup-Master and a read pool will servers except the master.
    • Varnish – reverse proxy helping speed up the site and reduce server load. Used also for cache management, edge side includes (ESIs), and health check on web servers.
    • Syslog-ng – collects data on page views and is used for view counters and related videos.
    • Nginx – high performance web server running PHP-FPM and acts as a external CDNs for static files like CSS, images and JS.
    • Symfony – fast and feature rich, comes with a large set of libraries.
      • Considers Symfony2 + Redis = Rapid Development + Scalable Website – a well balanced equation.
      • Loves dependency injection.
      • Choosing a web framework doesn’t mean killing performances as long as you use a solid architecture in front of it.
    • ActiveMQ – message bus designed for large sites, used for writes to MySQL and Redis.
      • They found it too rigid for a site undergoing constant changes and the gains did not justify maintaining a separate Java infrastructure.
      • Would like a do-over since the problems may have been more the developers than the tech.
    • Redis – a fast, open source, KV store, now the primary data source.
      • Updated in real-time.
      • Sorted sets are used for all lists
      • Pipelining,  executing several Redis commands with a single atomic command, is key for performance
      • After the switchover additional Redis nodes were added, not because Redis was overworked, but because the network cards couldn’t keep up with Redis.
      • All reads come from Redis
      • MySQL is used to allow the building new sorted sets as requirements change.
      • Append-only-file (AOF) format is used for incremental backups and is IO pain free
      • Consider using Redis as temporary cache in front of a Redis persistent store as a way to improve performance and reduce load.
    • MySQL – in a support role to feed Redis
      • Highly normalized because it’s not used directly for the site.
      • Some tables have 100+ million rows.
      • Used to populate Redis lists with new features
    • Both GIT and SVN are used, which doesn’t work well.
    • Doctrine is used in their CMS. Great tool that has saved weeks of development.
    • 网站2006年上线,2011年被收购
    • 2008年即达到每天1亿PV的浏览量
    • 每秒30万请求
    • 每秒推送100Gb的视频,相当于3张DVD的内容
    • 每小时生成8GB-15GB的日志


    • 之前用的是Perl
    • 现在用的PHP,以FastCGI方式运行于PHP-FPM
    • HAProxy
    • ActiveMQ
    • Varnish
    • Redis
    • Nginx
    • MySQL
    • Syslog-ng(https://github.com/balabit/syslog-ng)
    • Symfony2






极客公园 , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权
转载请注明原文链接:日访问量2亿PV的YouPorn 色X网站架构
喜欢 (0)



表情 贴图 加粗 删除线 居中 斜体 签到


  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址