240,000 uploads! *Friends* memes have overwhelmed Linux

Sometimes when I browse communities, what I love clicking on most isn’t the hot posts, but those seemingly inconspicuous corner features. You think it’s just a small sign by the roadside, but when you walk closer, it may connect to an entire unmarked path behind it. This Discourse crash felt a lot like that: a 1.6MB Friends meme image was used 246173 times, and in the end it somehow consumed a total of 377G of space, directly dragging down the Linux system it depended on.

The meme itself is quite recognizable too. It comes from a screenshot of the moment in Friends when Rachel cheers after learning there’s going to be a soap opera–themed rooftop party; the actress is Jennifer Aniston. The problem wasn’t the size of the image, but Discourse’s “secure upload” mechanism. When files move between different security contexts, the system automatically generates new copies with random SHA1 values, so even if the content is completely identical, it is still treated as a new file.

Following that line of thought, you’ll find this was far more serious than simply “the cache wasn’t cleared.” Discourse had actually long been aware that duplicate files take up a lot of space, so it had made optimizations, trying to keep only one copy of duplicates as much as possible through hard links. But the real trap hidden off the map was the ext4 file system, which has a limit on the number of hard links a single file can have: at most 65000. More than 240,000 repeated uploads obviously far exceeded that ceiling, and the original solution hit a dead end there.

Once that limit was hit, the whole thing went from saving space to frantically paying the price. In addition to the first full backup, the system also performed about 181,000 extra full file backup downloads, and that was what gradually crushed the server. The most interesting part of this incident is that it wasn’t some ultra-complex business logic that blew up, but a meme image everyone assumed carried “basically no burden” that tipped the whole thing over—like driving across a plain only to get stuck in a patch of mud that wasn’t even marked on the map.

The fix proposed by the Discourse team in the end actually feels a lot like opening a new path around a mountain in an open-world game: when the file system returns the EMLINK error for “too many links,” the system first makes a local copy of the file, sets that copy as the new “primary file,” and then continues creating hard links based on it until it hits the limit again. This way, it no longer bets on the boundaries of any one file system. The official team also said the new mechanism is compatible with all file systems and requires no extra user configuration.

If you spend time in tech communities or run a forum yourself, this incident is highly instructive. It’s a reminder that the truly dangerous thing is often not the Boss on the big map, but the little button in the corner that got clicked more than 200,000 times. Compared with many community tools that “go live first and patch the holes later,” Discourse at least explained the whole chain of issues clearly this time and also provided a practical new solution. As for how community users will remember this incident, my guess is it’ll probably come down to one line: Jennifer Aniston can now do more than provide reaction memes—she can also casually help stress-test a server.