Company news

Meta crawlers flood creator's website, exhausting hosting resources

San Diego children's book author Chelsea Paulus said Meta crawlers made more than 550,000 visits to her website, exhausting hosting resources and leaving the site unstable. She used host-provided code and Cloudflare human verification to temporarily block them, but the issue remains unresolved.

Published · Updated · Source: IT之家 · 机器人筛选 · AI translation
Illustration
IT Home reported on Nov. 3 that San Diego children's book author and illustrator Chelsea Paulus told Business Insider that Meta crawlers made more than 550,000 visits to her website, exhausting its hosting resources and forcing her to troubleshoot the failure with hosting provider SiteGround. Paulus said she received a warning from the host on Oct. 24 that 80% of her allocated CPU resources had been used, then found multiple access requests per second in her security panel, with the IP belonging to a Meta node in Dallas and identified as "Meta External Agent." She modified robots.txt and configured a firewall, but the rules took time to take effect; code the host provided for a hidden configuration file only worked temporarily, and in the middle of the week another crawler, "Meta Web Indexer," maxed out hosting resources again. A Meta spokesperson said website operators have the right to control crawler behavior through tools such as robots.txt and that the company respects rules set by site owners. A screenshot of a SiteGround email shows that its server-side protection was already limiting crawler requests, but as of Monday Paulus's website remained unstable. She removed the website link from her Instagram profile and tried Cloudflare's free human verification tool to block the crawlers. She said she cannot afford to upgrade her traffic quota because crawlers bring no revenue, and called on Meta to establish constraints. The above information comes from the original source and was organized by AI.
Read the original source

Related news