NASAβs Jet Propulsion Laboratory
Government
Swipe, like, share, repeat. Going viral isnβt an exact scienceβbut at BuzzFeed, quizzes and political exposΓ©s alike can make it around the world with just a few clicks. For hundreds of millions of users, the leading digital media publisher is the first stop whether you need a break from work or the latest breaking news.
Managing a cross-platform news and entertainment network of articles, lists, quizzes, videos, original series, lifestyle brands, and world-class reporting requires a flexible infrastructure, one which can support multiple deployments per day. Behind BuzzFeed.com is a distributed team of developers and engineers located from California to London to Argentina, with tech users, IT teams, and BuzzFeed News reporters building and running code.
At the start, these teams deployed manually, pushing code to production themselves. The number of clicks quickly multipliedβand not just on the website. Engineers were spending more and more time on frustrating, manual tasks, and less on building tools that allowed them to focus on critical code. βThe mindset we carry within our team is that we always want to automate ourselves into a better job,β explained Director of Engineering Andrew Mulholland. βWe want to make sure that the task weβre doing manually today becomes mostly automated. We focus on building tools that serve as an enabler for developers.β
With an automation-first approach, Mulholland turned to solutions already in BuzzFeedβs tech stack. GitHub Enterprise had been used within the company by developers and journalists alike for years, but quickly became the foundation for a modern development workflow built on collaboration and monorepos. βOne of the great things about BuzzFeedβs culture is that, if you see something broken, and you offer up a pull request to fix it, people will be happy to review the code and approve it,β he said. βMonorepos make that even easier. You have access to all code by default and this serves as an enabler for learning.β
Beyond the tech team, users across the organization have access, including some of the Buzzfeed News team. The same transparency that applies to the companyβs code applies to their reporting as well: data journalists publish their data to a separate, public GitHub repository, whether itβs Jupyter Notebooks or stand-alone code. βWe want to try to show our work as much as possible and not take readersβ trust for granted,β said Mulholland. βWhen weβve done analysis for a story, we make the tools that we used for that analysis available so people can run and verify it if they want to.β
With GitHub, this process of open code by default is seamlessβanyone within BuzzFeedβs GitHub organization can open a feature request in another service. Known as innersourcing, Mulholland explained how code reuse and collaboration has allowed BuzzFeed developers to contribute to improving BuzzFeedβs infrastructure. For example, when an engineer needed custom autoscaling, the infrastructure team wasnβt available to develop it. But since the developer was able to access the teamβs code, they were able to self-service and suggest the code changes themselves. βThey came to us with some requests for improvements, and we said that if they helped, weβd be happy to accept that code. So they added the code in, and itβs now widely used across dozens of different services.β
Every second counts in an outageβespecially for one of the worldβs most-visited websites. In one instance, a service outage occurred while the original development team wasnβt available; however the problem was able to be quickly resolved. βBecause the application was in a monorepo in GitHub and we have a high degree of standardization between our services, one of our infrastructure engineers was familiar enough to make changes and solve the issue.β The next day, the original team approved and merged the changes.
One of the most important roles for the infrastructure team at BuzzFeed is serving its internal developers. And local development environments are key to that. βBeing able to have developers bring up a service locally on their machines is crucial to ensuring a quick service when they run it,β Mulholland said.
BuzzFeed created libraries and templates for BuzzFeedβs supported languagesβPython, Golang, and Nodeβto enable developers to quickly start building locally. To save even more time, the team used GitHub as a core part of their highly-automated CI/CD workflows. Now, developers can immediately package and push code. βAs soon as your branch is pushed up to GitHub, that will kick off a build in our CI system. Then once it has been reviewed and merged, it will be automatically deployed to production.β
By moving from manual deployments to automation with GitHub, BuzzFeedβs teams deploy 30 percent more frequently, sometimes up to 300 times per day. Now thereβs more bandwidth for other top prioritiesβincluding security and open source.
Open source is important to BuzzFeed; as a user of many open source technologies the company wanted to give back to the community tooβand so in 2018 opened sourced its single sign-on (SSO) solution. Before sharing the SSO project as open source on GitHub, the team worked to improve the readability and structure of its code. βWhen youβre under pressure to get things done, thereβs an element of technical debt that creeps up,β Mulholland explained. βWe had our SSO project internal for a year before making it open source. Since making it open source meant making the code public, it forced us to raise the bar and fix our code internally, and created a great learning opportunity for our developers.β
Participating in open source brings better codeβand credibility. βA lot of people obviously know us for our buzzy content, quizzes, massive food brand Tasty, and BuzzFeed News, but now people understand that, βOh yeah, thereβs good software engineering happening there as well.β Thatβs been very helpful when weβve been hiring.β
By making the code to one of BuzzFeedβs most critical projects open source, itβs also made it eligible for the companyβs bug bounty program. βWith SSO being open source itβs easier for researchers to find issuesβand thatβs a good thing,β said Mulholland. βWe then encourage the use of our bug bounty program to ensure we can reward researchers for responsible disclosure as we triage and remediate it.β
Initially, the bug bounty program was a struggle. As he explained, the all-in approach created another backlog: βIn theory everyone was responsible for the triage of tickets, but by having everyone responsible, it meant that really no one was responsible. Often non-critical reports took a long time to be triaged.β
Having worked at companies with large security organizations, Mulholland knew a siloed approach wasnβt in step with BuzzFeedβs values. βIt creates a very much us-versus-them mentality, and that doesnβt fit well how we do things. We wanted to create a more collaborative, inclusive approach.β So instead they built a security program that was uniquely BuzzFeed, coming up with a rotation plan that assigned an engineer from a different team each week to be on triage. βOur mindset was that by sharing the burden across the tech organization, it wouldnβt be too much of a burden to any individual or any one team.β It workedβthe rotation meant that someone was always responsible and the meantime to triage and bounty dropped from weeks to days.
The improved process also brought in more bug reports, which Mulholland welcomed: βSuddenly we were getting way more reports because security researchers got the signal that we care about security, and we were paying bounties quickly. They realized it was worth their while, so they spent more time looking at BuzzFeed.β Mulholland and his team even began working with researchers periodically, inviting them to preview new features for potential vulnerabilities.
In Mulhollandβs view, using GitHub has introduced more secure features into BuzzFeedβs infrastructure, along with some much-needed changes. The team just recently migrated BuzzFeed.com article pages onto a new React-based front end. βGitHub played a key role in that, with its collaborative features being a boon to development,β he noted. βBy moving to the React front end, itβs more modular. Itβs going to enable us to work to evolve the user experience to make it easier to find the content that you want to read, and easier for us to enable new features to put around the reading experience.β
After all, from security to CI/CD to personality quizzes, itβs about connectionβand finding creative solutions that keep everyone moving forward. βGitHub has become so essential to the developer experience,β Mulholland said. βIf we can keep enabling our engineers like we did using continuous deployments with GitHub, it saves our team a considerable amount of time. And those time savings are a boost to productivity.β
Product
Industry
Location
Number of seats
From flexible hosting to dataβpowered security, get everything your team needs to build at their best.
Start collaborating with your team on GitHub
Most popular
The basics for individuals
and organizations
Most popular
Advanced collaboration for
individuals and organizations
* Discounted first-year pricing is for new yearly customers for up to 100 users (Contact sales for pricing for more than 100 users). After first year, billed at $48 per user/year.
Most popular
Security, compliance,
and flexible deployment
* Discounted first-year pricing is for new yearly customers for up to 100 users (Contact sales for pricing for more than 100 users). After first year, billed at $252 per user/year.
Want to use GitHub on your own? Check out our plans for individuals