Browse apps

Guides

How to Whitelist Googlebot, Ad Crawlers, and Webhooks on Shopify

TL;DR

Storefront geoblocking and datacenter rules can hurt SEO and ad campaigns if they block Googlebot, AdsBot-Google, or Meta crawlers that browse from US data centers. LabX Fraud Filter evaluates a 25-crawler allowlist before any block rule, while server-to-server payment and Shopify webhooks never run storefront JavaScript at all.

When merchants set out to whitelist Googlebot, Stripe, and Meta crawlers on Shopify, they are usually preparing to turn on country blocking, datacenter filtering, or anti-scraper rules—and want to make sure those defenses do not break their organic search rankings, paid ad campaigns, or payment processing. It is a smart question to ask before activating any traffic filter.

A poorly designed geoblock or bot script can quietly damage a store in two ways. First, if you run a domestic store outside the United States and block foreign visitors without an exception for search engines, you will block Googlebot—which crawls predominantly from US data centers. Second, if you block cloud hosting providers or automated user-agents without allowlisting ad verification crawlers, Google Ads (AdsBot-Google) and Meta (facebookexternalhit / meta-externalagent) can fail landing-page checks and flag your ad URLs.

At the same time, merchants migrating from self-hosted platforms like WooCommerce or Magento often worry that blocking IP ranges or countries will drop incoming Stripe, PayPal, or fulfillment webhooks. In this guide, we explain how search crawlers, social/ad bots, and server-to-server webhooks actually interact with Shopify, how rule precedence works in LabX Fraud Filter, and how to customize your allowlist so bad traffic stays out while essential services pass through. For a complete overview of visitor filtering, see our Shopify traffic and bot blocking guide.

Why do geoblocking and datacenter rules break SEO and ads when configured poorly?

To understand why naive traffic blockers cause false positives on search engines and ad platforms, look at how modern crawlers connect to your Shopify storefront:

  1. Search crawlers browse from foreign and datacenter IPs: As Google Search Central documents, Googlebot crawls primarily from IP ranges located in the United States and executes JavaScript using a headless Chromium rendering engine (Web Rendering Service). If a merchant in Germany, the UK, Canada, or Australia enables a “Block all except my country” script that checks IP location before checking the visitor’s user-agent, Googlebot is treated like an unauthorized US visitor and shown a block screen or redirected away.
  2. Ad quality and catalog crawlers run from cloud servers: Before approving paid campaigns or shopping feeds, Google’s crawlers and fetchers (AdsBot-Google and Storebot-Google) and Meta’s webmaster crawlers (facebookexternalhit, Facebookbot, and meta-externalagent) inspect your product landing pages. Because these crawlers originate from data centers and declare automated user-agent strings, a blanket bot or datacenter block that lacks a good-bot allowlist can trigger “Destination not working” disapprovals in Google Ads or broken Open Graph previews on Facebook and Instagram.
  3. Overlay delays and <head> wiping hurt indexing: Some legacy blocker scripts hide the entire storefront behind an opaque white loading screen until an external GeoIP server responds, or wipe the document <head> when a rule triggers—stripping canonical URLs, meta descriptions, and JSON-LD product schema during rendering passes.
Automated Service TypeCommon User-Agent / Connection PatternWhat Breaks If Blocked on StorefrontHow LabX Fraud Filter Protects It
Organic Search CrawlersGooglebot, Google-InspectionTool, Bingbot, Applebot, DuckDuckBot (often from US data centers)Pages drop from Google and Bing index; URL Inspection in Search Console failsEvaluated at step 1 of storefront precedence (Allowed bot list) before country, ISP, or datacenter rules
Paid Ads & Shopping CrawlersAdsBot-Google, Storebot-Google, adidxbot (Microsoft Ads)Google Ads landing page disapprovals; Merchant Center price/availability mismatchesIncluded in the Search Engines allowlist group and allowed before any block or redirect rule
Social Media & Catalog Previewsfacebookexternalhit, Facebookbot, meta-externalagent, Pinterestbot, Twitterbot, LinkedInBotBroken link previews, missing Open Graph images, and Meta ad destination errorsIncluded in the Social Media allowlist group and allowed automatically
AI Search & Training BotsGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, BytespiderAI search assistants cannot cite product pages (or, if unwanted, scrape copy for training)Allowed by default in AI Crawlers; can be unchecked individually in Settings > Mitigator
SEO Audit ToolsAhrefsBot, SemrushBot, DotBot, Screaming FrogSite audits fail when run by your own agency (or competitors spy on your backlinks)Allowed by default in SEO Analytics; can be unchecked anytime to block competitor SEO crawlers
Payment & Order WebhooksServer-to-server HTTP POST callbacks (Stripe, Shopify Payments, PayPal, 3PLs, ERPs)Never affected by storefront scripts because webhooks do not run browser JavaScriptCompletely unaffected—storefront rules run in the browser and Checkout Protection validates buyer form fields

Do you need to whitelist Stripe, payment gateways, or webhook IPs on Shopify?

Merchants searching for a webhook IP allowlist on Shopify or asking how to whitelist Stripe IPs are usually carrying over habits from self-hosted servers (where an Nginx firewall or Cloudflare WAF rule can accidentally drop POST /webhooks/stripe callbacks).

On Shopify, architectural separation makes manual webhook IP allowlisting unnecessary when using LabX Fraud Filter:

  • Storefront rules execute only in a visitor’s browser: As we explain in our comparison of storefront blocking vs checkout blocking on Shopify, Shopify hosts your storefront on its global CDN and serves page HTML before any third-party app script runs. LabX Fraud Filter’s storefront rules (IP Address, Location, Region, ISP, User Agent, Referrer URL, Tor, and Datacenter) run inside the Fraud Guard theme app embed in the visitor’s browser.
  • Webhooks never load your storefront theme or run JavaScript: When Stripe, Shopify Payments, PayPal, Klarna, or a fulfillment service sends a webhook, it makes a direct server-to-server HTTP request to Shopify’s payment infrastructure or your app’s backend endpoint. It never opens a browser tab, never renders Liquid theme templates, and never executes fraud.js. No storefront country, IP, or datacenter rule in LabX Fraud Filter can intercept or block a payment gateway webhook.
  • Checkout Protection checks buyer form data, not webhook IPs: LabX Fraud Filter’s server-side Shopify fraud prevention layer runs as a native Shopify Cart and Checkout Validation Function on Shopify’s servers. While it cannot be bypassed by buyers who disable JavaScript or call the Cart API directly, it only evaluates checkout attributes entered by the shopper (Email, Phone, Name, Address, Zip, Country, and Customer tag). It never blocks payment processor callbacks or backend order webhooks.

How does the 25-crawler allowlist and rule precedence work in LabX Fraud Filter?

To guarantee that geoblocking countries on Shopify or blocking datacenter IPs never locks out search engines or ad crawlers, LabX Fraud Filter enforces a strict evaluation order inside the storefront guard (evaluate.js):

  1. Admin bypass link (bypass): If the browser holds your verified ?fg_bypass= token from Settings > General settings, the guard allows the session immediately so you can test rules without locking yourself out.
  2. Allowed bot list (good-bot): When Allow verified good bots is enabled, the guard checks the visitor’s user-agent against the allowed crawlers before looking at any bot switch, Tor/datacenter flag, country allowlist, redirect rule, or IP block rule.
  3. Explicit Allow rules (allow:<type>): Any rule saved with the allow action beats every subsequent block or redirect rule.
  4. Auto-block unknown bots (bot): When enabled in Settings > Mitigator, blocks generic scraper and HTTP library signatures (bot, crawl, spider, scrap, curl/, wget, python-requests, python-urllib, go-http-client, java/, okhttp, headlesschrome, phantomjs, puppeteer, playwright, selenium) and browsers reporting navigator.webdriver === true.
  5. Tor and Datacenter switches (tor, datacenter): Blocks visitors flagged as Tor exit nodes or cloud/hosting ASNs by the session’s cached GeoIP lookup. (Remember that datacenter blocking is not consumer VPN blocking: it stops server-hosted bots without blocking real shoppers on residential VPNs.)
  6. Suspicious crawler checks (suspicious:headless, suspicious:unknown-ua): Opt-in checks for non-browser user-agents and headless automation signals (navigator.webdriver or zero plugins, zero touch points, and zero languages combined).
  7. Country “Block all except…” mode (block:country-except): Blocks visitors outside your allowed countries—but only when the visitor’s country is positively known.
  8. Redirect rules (redirect:<type>) and Block rules (block:<type>): Evaluates your custom rules across IP, Location, Region, ISP, User Agent, Referrer URL, Page, and Product.
  9. Allow by default (default): Any visitor that does not match a rule passes through normally.
Allowlist Category in Settings > MitigatorBuilt-In Crawlers IncludedMatched User-Agent Tokens (Lowercase)
Search Engines (10)Googlebot, AdsBot-Google (Ads quality), Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider, Yeti (Naver), PetalBot, CocCocBotgooglebot, google-inspectiontool, storebot-google, adsbot-google, bingbot, adidxbot, applebot, duckduckbot, yandexbot, baiduspider, yeti/, petalbot, coccocbot
Social Media (4)Facebookbot, Twitterbot, LinkedInBot, Pinterestbotfacebookexternalhit, facebookbot, meta-externalagent, twitterbot, linkedinbot, pinterest
AI Crawlers (6)GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), Bytespider (ByteDance), Google-Extended (AI training)gptbot, oai-searchbot, chatgpt-user, claudebot, claude-web, anthropic-ai, perplexitybot, ccbot, bytespider, google-extended
SEO Analytics (4)AhrefsBot, SemrushBot, DotBot (Moz), Screaming Frogahrefsbot, semrushbot, dotbot, screaming frog
Web Archives (1)Wayback Machineia_archiver, archive.org_bot

How do you customize which crawlers are allowed or blocked in Settings?

Not every merchant wants all 25 crawlers browsing their catalog. While almost every store needs Search Engines and Social Media allowed, you might want to stop competitor backlink tools (AhrefsBot, SemrushBot) or AI training crawlers (CCBot, Bytespider, GPTBot) while stopping competitor price scraping on Shopify.

You can customize the allowlist in a few clicks (see annotated screenshots in our bot and content protection guide):

Step 1: Open the Allowed bot list in Settings > Mitigator

  1. In the LabX Fraud Filter navigation, click Settings and select the Mitigator tab on the left.
  2. Scroll to the Allowed bot list card at the bottom of the page. The status badge shows either All allowed, <N> bots allowed, or None allowed.
  3. Click Manage to expand the crawler configuration panel.

Step 2: Keep Allow verified good bots enabled

Leave the master switch Allow verified good bots turned On. If you switch it off, LabX Fraud Filter displays a warning banner: “Search engines will be blocked — With this off, your store can be dropped from Google and Bing. Only turn it off if you understand that consequence.”

Step 3: Choose Allow all or select individual crawler groups

  • To allow every verified crawler automatically: Leave Allow all checked. This waves through all 25 built-in crawlers across every category, including future additions.
  • To allow Googlebot and Meta while blocking AI or SEO spy crawlers:
    1. Uncheck Allow all.
    2. Keep Select all Search Engines (10) and Select all Social Media (4) checked so Googlebot, AdsBot-Google, storebot-google, Bingbot, Applebot, and Facebookbot (facebookexternalhit / meta-externalagent) always pass.
    3. Under AI Crawlers (6) or SEO Analytics (4), uncheck any crawler you do not want on your store (for example, Bytespider, CCBot, AhrefsBot, or SemrushBot).
    4. Scroll up and make sure Auto-block unknown bots and/or Block datacenter traffic are toggled On, then click Save. Once a crawler is unchecked from the allowlist, its "bot" user-agent token or datacenter ASN will be blocked by your mitigator switches.

What other safeguards keep your store fast, indexable, and easy to test?

Beyond user-agent allowlisting, LabX Fraud Filter includes five engineering safeguards that protect your search rankings, Core Web Vitals, and internal QA workflows:

  • <head> preservation when automated clients are blocked: If a bot that is not on your allowlist gets blocked on the storefront, renderBlockPage checks whether the client looks automated (looksLikeBot). When true, it clears only document.body and leaves your theme’s <head> untouched so canonical tags, meta descriptions, and structured data are never stripped if an unrecognized SEO tool visits your page.
  • Rules travel in an inline Shopify metaobject: Your compiled ruleset is stored in a Shopify metaobject (app_config) and rendered directly inside the page HTML. Non-location checks (including the Allowed bot list, IP rules, user-agent rules, and referrer rules) evaluate in microseconds with zero external network requests.
  • 30-minute cached GeoIP lookup with an 800 ms fail-open timeout: Location, region, ISP, Tor, and datacenter rules make a single request per browsing session to /apps/fraud/geo (cached in sessionStorage for 30 minutes). If the lookup takes longer than 800 ms or if our server is ever unreachable, the request aborts and fails open—allowing the visitor or crawler through rather than risking a false block.
  • Loading overlay is Off by default for optimal LCP: In Settings > Blocking template, the Loading page effect toggle is Off by default so your storefront paints immediately without hiding content behind a blank overlay while waiting for GeoIP resolution. (See our block page and admin bypass guide.)
  • Honest browser vs server boundaries: Keep in mind that storefront rules and content protection (right-click, text selection, and shortcut deterrents) execute in the visitor’s browser; a script or crawler that fetches raw HTML without running JavaScript (curl or disabled JS) will not trigger browser blocks, and OS screenshot shortcuts cannot be blocked by any website. Conversely, if you use a browser-based synthetic monitoring tool or remote QA team in another country, give your team the rotatable Admin access link (?fg_bypass=...) or audit recent blocks in the Analytics and Visitor logs (documented in our Analytics and Visitor log guide) inside LabX Fraud Filter.

Frequently asked questions

Why does geoblocking foreign countries on Shopify risk blocking Googlebot?

Googlebot and AdsBot-Google crawl primarily from IP addresses in the United States, even when indexing stores in the UK, Europe, Australia, or Asia. If a storefront geoblock restricts US or datacenter traffic without checking an allowlist of verified search crawlers first, Googlebot sees your block page or redirect instead of your product catalog.

Do I need to whitelist Stripe, PayPal, or Shopify webhook IP addresses in a storefront blocker?

No. Storefront rules in LabX Fraud Filter run inside the visitor browser via the Fraud Guard theme app embed, and Checkout Protection runs as a Shopify Cart and Checkout Validation Function on buyer form inputs. Server-to-server webhooks from Stripe, Shopify Payments, PayPal, or ERPs send HTTP POST requests directly to backend endpoints without loading storefront HTML or executing browser JavaScript.

How do I keep Meta and Google Ads crawlers allowed while blocking scraper bots?

In LabX Fraud Filter, go to Settings > Mitigator and keep "Allow verified good bots" enabled. The built-in allowlist of 25 crawlers—including Googlebot, AdsBot-Google, Storebot-Google, Bingbot, Facebookbot (facebookexternalhit and meta-externalagent), and Pinterestbot—is evaluated at the very top of the precedence order before any bot, datacenter, or country rule.

Can I allow Googlebot and Bingbot while blocking AI training bots or SEO spy tools?

Yes. In Settings > Mitigator, click Manage next to Allowed bot list, uncheck "Allow all," keep the Search Engines and Social Media groups checked, and uncheck specific AI Crawlers (such as GPTBot, ClaudeBot, or CCBot) or SEO Analytics bots (such as AhrefsBot or SemrushBot).

Sources

  1. Google Search Central — Verifying Googlebot and Other Google Crawlers
  2. Google Search Central — Overview of Google Crawlers and Fetchers (User-Agent Tokens)
  3. Meta for Developers — Sharing Debugger and Webmaster Crawler Guide
  4. Shopify Dev Docs — About Cart and Checkout Validation Functions

Part of our guide: Traffic & bot blocking →

Put your first blocking rule live in minutes

Install LabX Fraud Filter free on Shopify, turn on the Fraud Guard embed and add your rules.

Install free on Shopify

Free plan available