We regularly come across websites whose menus display a dozen sections, but finding a specific page is like navigating an obstacle course. Before even discussing redesign or SEO, the starting question is simpler: how to read the structure of an existing website to understand what it contains, what is missing, and what poses problems for both visitors and crawlers.
robots.txt file and XML sitemap: the first two exploration reflexes
When trying to understand the structure of a website, you don’t start by clicking through the menu. You open two files accessible by adding /robots.txt and /sitemap.xml to the root URL.
The robots.txt file indicates which sections are blocked for search engines. A whole folder hidden (for example, /staging/ or /archive/) often signals abandoned content or an incomplete migration. The XML sitemap, on the other hand, lists the pages that the site owner considers indexable. Comparing the sitemap to the navigation menu reveals discrepancies: orphan pages missing from the menu, old URLs still declared, empty categories.
You can check the Actu Web site in detail to see a concrete example of a sitemap structured by thematic sections, providing a quick reading of the editorial organization.
This step takes a few minutes and provides a rough mapping of the content before any in-depth analysis.
Exploration by AI bots: GPTBot, ClaudeBot, and the new access rules
Since 2023, server logs have recorded visits from bots that did not exist a few years ago: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Applebot, and OAI-SearchBot. These agents do not seek to index pages for a traditional engine. They collect content to train language models or fuel conversational responses.
Differentiating these bots from classic crawlers like Googlebot has become a task in itself. Filtering options go through the robots.txt (specific User-agent directives), but also through CDN rules, WAF, or IP restrictions for more granular control.
The llms.txt file, a layer of structure outside HTML
Since 2025, several guides recommend placing an llms.txt file at the root of the site. This file lists priority URLs and access policies for each AI agent. Some add content summaries by section.
The goal is to facilitate targeted exploration by language models, where the XML sitemap remains oriented towards traditional search engines. We move from a “indexing for Google” logic to a “multi-agent visibility” logic that also covers conversational search.
Feedback varies on the actual adoption of this file by different agents, but the principle of segmenting access according to the type of robot is gradually establishing itself as a best practice.
Single Page Application sites: the trap of invisible content
A site built as a SPA (React, Vue, Angular) loads all its content via client-side JavaScript. Visually, navigation seems smooth. For a crawler, the result can be an almost empty page if server-side rendering (SSR or pre-rendering) is not configured.
Without server-side rendering, the majority of a SPA’s content remains invisible to crawlers. Googlebot executes JavaScript, but with a delay and limitations. AI bots, on the other hand, do not all have this capability.
When analyzing the structure of a SPA site, three concrete points need to be checked:
- Does the raw source code (Ctrl+U in the browser) contain the text visible on the screen, or only empty tags and JavaScript calls?
- Do internal URLs generate distinct pages on the server side, or does everything go through a single entry point with fragments (#)?
- Does the XML sitemap reference all the routes of the application, including dynamic sub-pages?
If the source code is empty and the sitemap does not cover internal routes, the site has an exploration problem that affects both classic SEO and visibility to AI agents.
Mobile navigation and click depth: measuring the actual accessibility of content
On desktop, a two-level dropdown menu provides access to several dozen pages in one click. On mobile, the same menu transforms into an accordion or hamburger, and each level adds a touch interaction. The perceived click depth on mobile is often double that on desktop.
To evaluate the structure from the user’s perspective, we count the number of taps needed to reach the most viewed pages. Beyond three taps from the homepage, the abandonment rate increases significantly. This is also a signal for search engines: a page buried five levels deep receives less crawl budget.
Internal linking and contextual links
The main menu alone is not enough to make all pages accessible. Links embedded in the body of articles or product sheets constitute the contextual internal linking. This network of internal links serves two functions:
- It guides the visitor to complementary content without going back through the menu, which reduces the effective navigation depth
- It distributes SEO authority among pages, signaling to bots which pages are thematically linked
- It offers AI agents additional exploration paths to content that may not always appear in the sitemap
Analyzing a site’s internal linking is akin to mapping its real connections, not just its theoretical structure.
Auditing a modern website: where to start concretely
Let’s recap the field sequence: open the robots.txt to identify blockages, read the XML sitemap to list declared pages, check for the presence of an llms.txt file, test server-side rendering on a few key pages, and then count the click depth on mobile. These five checks cover angles that most audits overlook, particularly AI bots’ access to content and the actual behavior of JavaScript sites.
Exploring a website is no longer limited to its visible navigation. The structure that matters is the one that bots, whether classic or conversational, can actually read.
