DomainlessBot

DomainlessBot is the crawler for Domainless Search, a search engine with its own index of the web. It visits public pages so they can be found in search results, which link back to your page.

How it behaves

• It follows your robots.txt: the rules for DomainlessBot, or your * rules when there are none for it.
• It never asks a site for more than one page a second, and waits longer if your Crawl-delay says so.
• It skips any page marked noindex (a robots meta tag or an X-Robots-Tag header), and slows down or stops when your server answers 429 or 503.
• It reads your sitemaps to find pages, and only fetches public pages: no logins, no forms.
• It keeps the text of a page for search snippets. It does not keep images.

It identifies itself as:

Mozilla/5.0 (compatible; DomainlessBot/0.1; +https://domainless.fun/search/bot.html)

Limit or block it

To keep DomainlessBot away from your whole site, add this to your robots.txt:

User-agent: DomainlessBot
Disallow: /

To slow it down instead:

User-agent: DomainlessBot
Crawl-delay: 10

Changes apply the next time it reads your robots.txt, before it fetches anything else.

Remove a page

To have a page removed from Domainless Search, send us a report with its address. Reports are private.