Our web requests, and how to stop them
If IT Master appeared in your server logs, this is what it was doing, why, and how to ask it to stop.
Who is making the request
IT Master is a content engine operated by CodeNest in London. It publishes articles for its customers' websites, and a few of its jobs need to fetch web pages. Every request says who it is in its user agent and links here. We do not scrape sites to copy their content, we do not train AI models on what we fetch, and we never send forms, create accounts or buy anything.
The user agents, and what each one does
- ITMasterBot/1.0 (link verification): before we suggest a site for a link to one of our customers' resources, it reads that site's public pages to check the page exists and what it links to. It reads robots.txt first and honours any rule for the ITMasterBot token, keeps the result for 24 hours, and makes a small number of requests per site.
- ITMasterBot/1.0 (site audit on owner request): an audit of a website whose owner asked us for one when they signed up. It reads that site only.
- itmaster-linkcheck, ITMasterClickGuard and ITMasterLinkRetarget: one request to a URL that an article we published already links to, to confirm the link still works or has moved, so readers are not sent to a dead page. They never follow links onward.
- ITMasterRenderCheck and ITMasterIndexRescue: requests to our own customers' pages, to confirm a page shows what we delivered and can be indexed.
- ITMasterSEO/1.0 (site-owner-authorised inventory): reads our customers' own robots.txt, sitemaps and a handful of their pages each day, with the site owner's authorisation, to keep our record of their site accurate.
How much we fetch
Very little. The checks on published links run at most once a day per URL. The outreach checks read a few pages per site, spaced out, and stop when robots.txt says no or a site answers with an error. If you see more traffic from us than this page describes, that is a fault, and we want to hear about it.
How to stop it
To stop link-verification visits, add a group for our token to your robots.txt: the line User-agent: ITMasterBot followed by Disallow: /.
For anything else, or to be excluded entirely, email hello@itmaster.uk with your domain and we will stop. The link checks exist to keep links to your site accurate; if we stop checking, we stop linking.
In one sentence
Every request we send names itself and links to this page. A request that does not match what is written here should not exist.