Meta Empties 5 Data Centers to Create Giant AI Cluster
Custom Robots Power Massive Campus-Wide Forklift Upgrade
In what may be history’s largest forklift upgrade, Meta has emptied out 5 production data centers to create a massive AI cluster of NVIDIA GPUs.
The operation highlights the intense pressure at Meta to create more capacity for AI capacity, which has also seen the company deploy GPU clusters in tents.
Meta shared details of the unusual data center upgrade in a presentation by Yee Jiun Song, the VP Engineering, Infra Foundation at Meta at last week’s AI Infra Summit in Santa Clara.
“We moved thousands of racks between data centers as part of this operation,” said Song. “And in order to do this quickly, we had to redesign the loading docks in our data centers, build brand new robots to move these 1,000-pound racks, and even design crateless packaging for these racks to speed up the moves.
“We quadrupled the amount of networking in these buildings, which involved pulling out and replacing hundreds of meters of network fiber, and digging new trenches to connect the five buildings together,” he added. “And we did all of this in a matter of months.”
The end result was a cluster of 129,000 NVIDIA H100 GPUs, which at the time was the largest cluster Meta had built.
The Meta Campus Model
Meta builds some of the largest data center campuses in the world, typically consisting of between five and nine large buildings, each of which includes up to 1 million square feet of data halls.
In recent months, Meta has super-sized its AI infrastructure ambitions.
The reason? Training AI models requires enormous amounts of computing and storage, which must live in close proximity to one another to support low-latency movement of data.
As Meta begin deploying larger GPU training clusters, it found that it had run out of capacity to support its training ambitions.
“It was clear we had to go beyond a single building,” said Song. “We didn't have five or more empty data centers conveniently located next to each other. But because the demand for AI compute was so intense at the time, we actually emptied out five production data centers that were located next to each other to produce the power we needed to build a large cluster.”
There’s lots of good reasons Meta had never done this before.
“It's extremely expensive to take down data centers that are in production, because these are expensive investments, and you want them to keep running and doing useful work,” said Song. “In our case, these data centers were serving live workloads, and we have to take them down as quickly as possible, while not causing any user perceived outages.”
Song didn’t share the exact date or location of the campus. But he noted that there are bigger campuses to come.
Speed, Scale and New Thinking
Song noted the recent announcements of two massive data center projects:
Prometheus, a 1 gigawatt data center campus in New Albany, Ohio.
Hyperion, a 5 gigawatt project being built in Richland Parish, Louisiana.
“So you might have seen this crazy visual,” said Song, noting the graphic shared by Meta CEO Mark Zuckerberg (and later reshared by the President). “The scale at which we're building these super clusters is truly mind mind boggling.”
Speed of deployment is also prompting new strategies for Meta, which recently began deploying GPU clusters in tents to bring them online faster.
The tent deployments - along with upgrading 5 buildings at once, and 5 gigawatt campuses- all reflect the creativity needed to meet the infrastructure demands of the AI arms race.
“Having built out infrastructure for years, we thought we'd learn everything that was to learn about scale,” said Song. “But honestly, AI is kicking our butts and teaching us that we know nothing.
Song said his job “gives me a front row seat to how quickly things are moving.”
“I've never seen hardware and infrastructure move and change so quickly. The next few years are going to be even more crazy than the last few years with the rapid change and innovation that's happening. And I can very confidently say that there has never been a more exciting time to be working in infrastructure.”