Help Center

AI crawlers and AI search: what bots get from your site

AI companies send bots to read your public pages. The Crawlers and AI search section of the plugin lets you choose what each kind of bot gets. Bots only ever get what a logged-out visitor sees.

You find it under MemberPress AI > AI agents > Crawlers and AI search. The section shows On or Off beside its name.

The switch

The switch is Give AI a clean text copy of your pages. When it is on, AI bots can get a plain text version of each page, without menus, ads or scripts.

While it is off:

  • Every visitor, bot or person, gets your normal page.
  • Addresses ending in .md do not exist.
  • You cannot block AI bots from this screen.

When you turn it on, you choose a response for three groups of bots. Every group starts on Full page, so bots see no change until you pick another choice.

The three choices

  • Full page: your normal web page, as a visitor sees it.
  • Text copy: a clean text version of the same page. Easier for AI to read.
  • Block: no page. The bot gets a refusal.

The three groups

  • AI that collects pages to train models: GPTBot, ClaudeBot, CCBot, Bytespider.
  • AI search engines: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
  • AI that opens a page for a person: ChatGPT-User, Claude-User, Perplexity-User. These visit when someone asks an AI assistant to read one of your pages.

To set them, choose a response for each group, then click Save bot choices.

Give one bot a different choice

You can give a single bot a different choice from its group. You can add up to 50 of these exceptions.

  1. Under Add a bot, pick the bot.
  2. Click Add exception.
  3. Choose the response you want for that bot.
  4. Click Save bot choices.

How this works

  • Only Block stops a bot. A bot set to Full page can still ask for the text copy: by adding .md to a page address, by opening /llms.txt, or by asking for text in its request.
  • Members always see the normal page. A signed-in member gets the normal page. A .md address gives everyone the logged-out text copy.
  • Hidden text is included. Text hidden only by CSS, such as a members-only box set to display: none, is in the text copy and already in your page source. Protect it with a MemberPress rule, not by hiding it.
  • Caches can get in the way. These choices apply only to visits that reach WordPress. A page cache or CDN can give a stored page to any bot, even a blocked one. Let bot visits pass through to WordPress, or make the cache vary on the User-Agent header.

What this is not

This setting is about bots that read your public pages. It is separate from the assistants you connect to your site, which Using MemberPress.ai from your own AI client covers, and from the public catalog.

FAQ

Will blocking a bot remove me from AI search?

The screen says only this: Block means no page, and the bot gets a refusal. Blocking applies to that bot or that group of bots, and nothing on the screen says anything about search results or traffic. If you want AI search engines to keep reading your pages, leave their group on Full page or Text copy.

Can bots read my members-only content?

Bots only ever get what a logged-out visitor sees, so content behind a MemberPress rule is not in the text copy. The exception is text you hid with CSS, such as a members-only box set to display: none. That text is already in your page source and is in the text copy too. Protect it with a MemberPress rule, not by hiding it.

I chose Block but a bot still got my page. Why?

These choices apply only to visits that reach WordPress. A page cache or CDN can hand a stored copy of the page to any bot, including one you blocked. Let bot visits pass through to WordPress, or make the cache vary on the User-Agent header.