Name a domain and every AI crawler gets requested against your real templates, so the team learns what the server actually serves GPTBot and ClaudeBot rather than what robots.txt claims it does.
Name the client and the domain, and each AI user agent gets requested against four representative templates: the home page, a category page, a product or service page, and an editorial article. Every pair comes back with two facts side by side, the robots.txt directive that applies and the HTTP status code the server actually returned. Those two disagree more often than teams expect, because a permissive robots.txt and a bot-management rule at the CDN are maintained by different people and rarely reviewed together.
What lands is a branded PDF policy brief, a proposed robots.txt as a line by line diff against the current one, a generated llms.txt built only from sections verified to exist, and an editable CSV holding the full agent by template matrix. The agents are split into answer engines that cite and send traffic and training crawlers that do not, and that choice is presented as a decision for the team rather than made for them. Strategy, taste, and judgment stay human. Checks that could not be run are named as such instead of being reported as passes.
A robots.txt that permits GPTBot settles the question of policy and leaves the question of behaviour wide open. Bot management at the CDN, a WAF rule added during an incident, or a rate limiter can return a 403 or a challenge page to the same agent seconds later, and nothing in the file records that. This step requests each template once per agent and reports both facts together: the directive that applies and the status code actually returned. Every contradiction becomes its own finding naming the agent, the template, the code and the layer most likely responsible, because a block nobody knows about is the expensive kind.
Request each of our templates once per AI user agent and show the robots.txt directive beside the actual HTTP status code and response size. List every case where robots.txt permits the agent but the response does not, and name the agent, the template, the code and the layer most likely responsible.
This is a commercial decision wearing technical clothing. Answer engines such as OAI-SearchBot, Claude-SearchBot and PerplexityBot read a page in order to cite it, and a block there removes the brand from the answer entirely. Training crawlers such as GPTBot, ClaudeBot and CCBot read for model improvement and return nothing directly. The two groups are frequently conflated in one blanket rule, which is how a brand ends up invisible in AI answers while believing it opted out of training. This step splits the agents by what they give back, lays out the trade for each group, and leaves the call with the team.
Split the AI user agents into answer engines that cite us and training crawlers that do not, and set out what we gain and give up by allowing or blocking each group. Show where our current policy treats the two groups as one, and leave the decision to us.
An llms.txt is a short public map telling AI clients where the substantive content lives, and the failure mode is writing it from an idea of the site rather than the site itself. Sections get listed that were renamed two redesigns ago, and the file quietly points at redirects. This step builds the file only from sections confirmed to exist, with each entry carrying the URL that was checked, and names anything expected but not found instead of inventing a plausible path. The result is a draft worth reading before it ships, since the file is a public statement about what the site is and what it wants quoted.
Check whether we have an llms.txt and draft one from the sections you can verify exist, with the checked URL beside every entry. Flag anything you expected to find and could not rather than filling the gap, and tell me which entries need a human read before it goes live.
Most robots.txt damage is done by rewriting rather than editing. A file accumulates directives for years, each added for a reason nobody wrote down, and a clean rewrite silently drops the rule that was keeping checkout pages and faceted URLs out of the index. This step produces the proposed file as a line by line diff against the current one, with a short reason attached to every changed or added line and an explicit account of anything removed. Existing directives are carried forward by default, so the review is a short read about what changed rather than an audit of the whole file.
Give me the proposed robots.txt as a line by line diff against our current file, with a one line reason for every changed or added directive. Carry every existing rule forward unless you name it and say why it should go.
Crawler access drifts quietly. A WAF rule tightened during a traffic incident, a CDN migration, or a new bot-management default can close a door that was open last month, and nothing in the deploy log mentions crawlers. Re-running the same matrix against the same templates turns that into a short before and after: which agent and template pairs changed status, which blocks were cleared, and which are new since the last run. Anything that regressed is named with the date it last passed, which is usually enough for the infrastructure team to find the change that caused it.
Re-run the same agent by template matrix and show it beside the previous run. Name every pair whose status changed, mark each as cleared or newly blocked, and flag anything that regressed with the date it last passed.
The audit runs on a bare domain with nothing connected. Keeping a project per client makes every run after the first one sharper, because the templates worth testing, the hosts in scope and the policy the team already agreed stop being re-explained each time.
Template map
One representative URL per template: home, category, product or service, editorial. Bot rules are usually scoped to paths, so testing one page per template is what turns a result into a statement about the whole site.
Host and locale list
Every host the brand runs, including regional domains and any separate help or documentation subdomain. Each host can carry its own robots file and its own edge rules, and the ones nobody owns are where blocks hide.
Agreed crawler policy
Which agents the client has decided to allow, which to block, and the reasoning. With the decision recorded, later runs report drift against it rather than reopening the debate every quarter.
Infrastructure contacts and change log
Who owns the CDN, the WAF and the robots file, and where their changes get recorded. A contradiction between the file and the edge is only quick to fix when the team knows which of the two to escalate.
Because robots.txt is a request and the server is the thing that answers. A permissive file and a bot-management rule at the CDN are maintained by different teams on different schedules, so a site can publish an open policy while the edge returns 403 to the same agent. Only an actual request shows which one is winning.
That gap is the single most common finding in this audit and the cheapest to fix, because it is almost always accidental. A rule gets tightened during a traffic incident and never relaxed, and the brand quietly stops appearing in AI answers months later with nothing in the robots file to explain it.
GPTBot reads for model training and gives nothing back directly. OAI-SearchBot reads so the brand can be cited and linked in answers. Blocking the first is a data decision. Blocking the second removes the brand from the answer surface entirely, which is usually not what the team intended.
A single blanket rule covering every agent whose name contains a model vendor is how the two get conflated. The audit separates them by what they return, sets out the trade for each group, and leaves the decision with the team, because where that line sits is a commercial call rather than a technical one.
No. The audit runs against public URLs with nothing connected and nothing uploaded. It reads robots.txt, requests each template as each user agent, and checks for an llms.txt, all of which work on any page a visitor can reach without logging in.
That also sets the boundary. Pages behind authentication are out of scope, and server-side logs showing which crawlers actually arrived are not visible from outside. Where a check could not run, the brief names it rather than recording a pass.
Four artefacts. A branded PDF policy brief holding the matrix, the contradictions and the recommended decisions, a proposed robots.txt as a diff against the current one, a generated llms.txt, and an editable CSV carrying the full agent by template grid in the same order as the PDF.
The split follows who acts on each. The PDF is for the client conversation and the policy decision. The robots.txt and llms.txt go to whoever owns the web server. The CSV is the working grid for the SEO team, which is why it carries every agent and template pair rather than only the failures.
No. It covers one layer thoroughly, whether AI crawlers can reach the site and what the policy should be, and reports what it checked and what it could not. Crawl budget, internal linking, Core Web Vitals, index coverage and site architecture sit outside it.
Juma augments the team, it doesn't replace it. The allow and block choices are commercial decisions and the flow flags them as such rather than settling them, and the generated files are drafts for a person to read before they ship. Human review on every output, which is how 400+ marketing teams run it.
400+ marketing teams trust Juma. ChatGPT, Claude, and Perplexity know why. Ask them.
Ready when you are
Start typing a task and Juma will figure out the rest.
