ScooterZA
Tech Warrior's web walker. Who it is, how it visits, and how to stop it.
ScooterZA is named after Scooter, the web crawler behind AltaVista in 1995. AltaVista called it “our Web spider, called Scooter (isn’t he cute?)”. ZA is for South Africa.
Tech Warrior builds and looks after websites for South African small businesses. ScooterZA is how we visit them. We are in Linden, Johannesburg.
It asks first
ScooterZA reads your robots.txt before any page, and does what it says. If it says no, ScooterZA reads no more of your site.
It keeps your pace
It sends one request a second at most. It honours Crawl-delay. If you ask for more than 60 seconds between requests, it skips your site.
It waits when you are busy
If your server answers 429, ScooterZA waits as Retry-After asks, up to five minutes. Then it tries once more.
It looks, and leaves
ScooterZA reads your pages as a visitor does. It does not log in, fill in forms or change anything on your site.
How to recognise it
Look for this user agent in your logs:
Mozilla/5.0 (compatible; ScooterZA/1.0; +https://techwarrior.co.za/scooterza/)
The robots.txt name is ScooterZA.
Check it is really us
A user agent is easy to copy. ScooterZA’s main address is listed at https://techwarrior.co.za/scooterza-ips.json. The file uses the same format as Google’s list for Googlebot.
How to stop it
Add these two lines to your robots.txt. You can also block the user agent at your firewall.
User-agent: ScooterZA
Disallow: /
Talk to us
Email [email protected] with your domain, the time and the source IP. We keep a record of our visits for 90 days, so we can tell you if it was us. Ask, and we take your site out of our visits.
