Start with three questions: can the crawler request the page, does the response contain the important product facts, and do your access rules match your intent?
A product page can load perfectly in your own browser while a search crawler receives a firewall challenge, a rate limit, a blocked path, or a thin response that depends on later browser activity. Each case needs a different fix.
Access evidence is necessary, but it is not a visibility promise. Allowing a crawler does not guarantee that a page will be indexed, selected, cited, ranked, or included in a shopping experience.
Robots rules are one part of access
A robots.txt file gives crawlers instructions about which paths they may request. The user-agent name matters. OpenAI, Google, Microsoft, and other providers can publish different crawler identities for search, user-requested visits, or other uses.
Read the rule that applies to the intended crawler and the exact product path. A general allow does not help if a later rule blocks the collection or product directory. A missing rule also does not rule out a network-level block.
- Confirm the robots.txt response can be fetched
- Find the rule group for the crawler being checked
- Test a real product path against the allow and disallow rules
- Review page-level robots directives and response headers
Firewalls and bot protection can tell a different story
A content-delivery network, firewall, proxy, or bot-management tool can challenge a crawler before the page is returned. In that case, robots.txt may say allowed while the network still blocks access.
Record the response status, returned content, redirect destination, and any clear challenge evidence. Treat a timeout, server error, or rate limit as indeterminate until it can be reproduced and diagnosed. Do not rewrite an unavailable check as an intentional merchant block.
If the request failed before useful page evidence was recovered, report the access limit and retry or review it manually. Do not infer a page-quality defect.
The response still needs usable product facts
A successful response is only the first gate. The returned page should expose the product title, price, currency, stock state, image, stable address, and important option context. If those facts appear only after complex client-side actions, a crawler may receive less than a shopper sees.
Compare the raw response with a rendered browser view. If rendering adds the missing facts, document the difference. The likely fix may be server-rendered product content or clearer structured product information, not a robots change.
What a responsible access report should say
A useful report names the user agent, URL, observation time, response, applied rule, visible or structured facts recovered, and any limit. It should distinguish allowed, blocked, and unknown evidence for each provider rather than applying one provider's behavior to all of them.
Once access is confirmed, the next check is product understanding. Price, stock, options, policy links, and built-in product information still need to agree.