There's a common belief that you can stop bad bots by editing robots.txt. It's worth understanding why that doesn't work — because relying on it leaves your site exposed.
robots.txt is opt-in
robots.txt is a set of instructions that well-behaved crawlers (like Google) choose to follow. Malicious bots — the ones scraping your content, probing for vulnerabilities, or abusing your bandwidth — simply ignore it. Worse, robots.txt publicly lists the paths you'd rather hide, which can act as a map for attackers.
What actually stops abusive bots
- Firewall rules and rate limiting at the server or WAF level.
- Blocking by IP range, user-agent, and behavior — not by request.
- Challenge pages for suspicious traffic.
- Ongoing tuning as bots rotate IPs and disguise themselves.
It's a moving target
Block one bot and another shows up minutes later with a new approach. That's why bot defense is a maintained service, not a single configuration you set and forget.
When to call in help
If your site is already down, hacked, or eating bandwidth, every hour of guesswork costs money. Ximbalo runs a full diagnostic, finds the root cause, and gives you a clear repair estimate before any work begins.
Book a consult or request a $250 assessment from the homepage — we get you back online and hardened against the next attack.