Saturday, September 26, 2026 For AI agents: /llms.txt
Poliety
Building the business layer of the agent internet.

PolietyBot

The crawler behind Poliety Concierge: what it fetches, why, and how to control it.

What it is

PolietyBot fetches public web pages to build source-grounded knowledge bases for Poliety Concierge customers. It crawls a domain only after the domain's owner has verified control of it and asked for the crawl. It is not a search engine spider and does not roam the open web.

How it behaves

  • User-agent: PolietyBot/1.0 (+https://poliety.com/bot)
  • Respects robots.txt by default. A customer may override it only for a domain they have verified they own.
  • Paces requests per host and honors a bounded page budget per crawl.
  • Fetches over HTTPS where available, follows a limited number of redirects, and never executes page JavaScript.
  • Stores fetched content with a content hash so customers can see exactly which version of a page a fact came from.

How to block it

Add this to your robots.txt:

User-agent: PolietyBot
Disallow: /

Status

PolietyBot crawls only domains their owners have verified with Poliety Concierge. The first such domain is poliety.com itself: the crawler read this site to build Poliety's own Concierge. On this site it deliberately skips /news/, declared in /robots.txt, because news content belongs to editions, not to business facts.

Contact

Questions or reports about crawler behavior: dburks@poliety.com. Security reports: see /.well-known/security.txt.