GEFTBot

GEFTBot is the crawler operated by GEFT. Its single purpose is search: helping AI agents find web results that match every condition a user asks for, and present them with a link back to the source site. Content collected by GEFTBot is not used to train AI models.

For example, when a user's request carries several conditions at once — a date, a place, a price limit — the goal is to find results that satisfy every one of them, and to send the user to the relevant source pages.

What GEFTBot does

GEFTBot performs one passive discovery crawl of a site, and does not come back to recrawl it. From what the site already shows publicly, GEFT works out the site's facet set (the conditions its listings can be narrowed by, such as date, location, price, or category) and its query grammar (how the site expresses those conditions in its own addresses and parameters).

GEFTBot reads:

• pages reachable through the site's own public links and sitemaps;
• structured data the site publishes, such as schema.org markup;
• the filter and search links the site itself exposes.

GEFTBot does not:

• submit forms, log in, or try filter or parameter combinations the site does not itself link to;
• access paths disallowed by robots.txt;
• collect personal or payment data;
• purchase, book, or change anything on the site.

Trying filter combinations to learn how a site behaves is active probing. GEFT does this only after a verified site owner explicitly authorizes it.

How to identify and verify GEFTBot

The user agent is there for readability in your logs; anyone can copy it. The signature is the identity: a request is from GEFTBot only if its signature verifies against a key in the directory above. Treat unsigned traffic that claims to be GEFTBot as not GEFTBot.

How to control GEFTBot with robots.txt

GEFTBot will read robots.txt before crawling and obey it, including Crawl-delay. Rules addressed to GEFTBot take precedence over rules for *. If a site has no robots.txt (the file returns 404 Not Found), GEFTBot treats the site as having no crawl restrictions and crawls it within the load limits below. If robots.txt cannot be retrieved because of a server error or a timeout, GEFTBot will not crawl the site.

To block GEFTBot from your whole site:

User-agent: GEFTBot
Disallow: /

To block part of your site and slow GEFTBot down:

User-agent: GEFTBot
Disallow: /account/
Crawl-delay: 10

Load

GEFTBot will stay within the limits a site sets, back off when a server slows down or returns errors, and keep retries limited so they do not multiply load. GEFTBot will not rotate IP addresses, imitate a browser or another crawler, or attempt to solve challenges to get past bot protection.

Claim your site

After the discovery crawl, GEFT contacts the site with an invitation to claim it. Claiming is optional, and you do not need to install anything: no code to deploy, no endpoint to expose, no WAF rule to write.

If you want your site to be AI-ready on your own terms, you accept the invitation and verify that you own the domain. After verification you can:

• see what GEFT discovered — the facet set, the query grammar, and the evidence behind each finding;
• set GEFT's permissions for your site across seven axes: which data and capabilities, which agent identities and purposes, which access path, freshness, load limits, economics, and retention and indexing.

A setting is not a request we take under advisement; it becomes a machine-checkable permission that the runtime checks before every request. Anything you have not explicitly allowed is denied. Restrictions always apply; expanding access, such as allowing active probing, requires your explicit authorization.

To start a claim, email hello@geft.ai from an address at the domain concerned. More on the model: geft.ai → Publishers.

Operator

Contact

For questions about GEFTBot, or requests concerning the crawling of your site, contact hello@geft.ai. Please include the domain concerned and, where applicable, the addresses of the affected pages.