The 2026 Technical SEO Checklist: Crawl, Index, Render, Speed and AI Access

A technical SEO checklist makes sure search engines, and now AI search crawlers, can crawl, render and index your site, and that its pages load quickly and securely. This 2026 checklist covers nine areas, from robots.txt and sitemaps to Core Web Vitals and AI crawler access, and tells you exactly how to check each item with free tools and Google Search Console. For page-level items like titles, headings and alt text, use our SEO audit checklist alongside this one.

The checklist at a glance

1. Crawling: can bots reach your pages?

• robots.txt loads and blocks only what you intend. How to check: open /robots.txt (it should return 200 as plain text), then test important URLs with the robots.txt tester. Don’t block the CSS and JavaScript Google needs to render pages.
• An XML sitemap lists only canonical, indexable URLs. How to check: validate it with the sitemap validator, which also status-checks a sample of listed URLs. Stay under 50,000 URLs and 50 MB per file, reference it in robots.txt and submit it in Search Console.
• The server answers reliably. How to check: Search Console → Settings → Crawl stats. Look at host status and the share of 5xx responses and timeouts. Frequent server errors make Google slow down crawling.
• No crawl traps. How to check: look for endless URL variations from faceted filters, sort orders, session IDs or calendars in your crawl and in Crawl stats. Limit them with consistent internal linking, canonicals and, where needed, robots.txt rules.
• Internal links are real links. How to check: links should be <a href="…"> elements. Buttons that navigate with JavaScript onclick handlers aren’t followed.
• Redirects are single hops. How to check: trace important URLs and old URLs from past migrations. Permanent moves should use 301 or 308 and land on the final URL in one step.
• No broken internal links or orphan pages. How to check: crawl the site, then compare the URLs found by crawling with the URLs in your sitemap. Sitemap URLs that no page links to are orphans.

2. Indexing: will Google keep them?

• The Page indexing report has no surprises. How to check: in Search Console, review “Not indexed” reasons. “Excluded by noindex” on pages you want, “Duplicate without user-selected canonical” and “Crawled – currently not indexed” on important URLs all need attention.
• noindex appears only where intended. How to check: look in both the robots meta tag and the X-Robots-Tag HTTP header. Staging settings and SEO plugin toggles are the usual culprits.
• Canonicals are self-referencing, absolute and consistent. How to check: each indexable page has one canonical pointing to its own preferred URL, which returns 200 and isn’t noindexed.
• One version of the site exists. How to check: http://, https://, www and non-www variants should all redirect to one host, and trailing-slash handling should be consistent.
• Duplicate and thin URLs are under control. How to check: tag archives, printer versions, parameter URLs and near-identical location pages. Merge, canonicalize or noindex them.
• Missing pages return a real 404 or 410. How to check: request a made-up URL. A “not found” page served with status 200 is a soft 404.

3. Rendering: is the content in the HTML?

• Main content and links are in the server response. How to check: compare view-source with what you see in the browser, or use URL Inspection → View crawled page in Search Console. Google can render JavaScript, but it’s slower and less reliable than plain HTML, and many AI crawlers fetch the raw HTML only.
• Titles, canonicals and robots tags don’t change after JavaScript runs. How to check: compare the raw HTML values with the rendered DOM. Google’s JavaScript SEO basics warns that conflicting values cause unpredictable results.
• URLs are real paths, not fragments. How to check: single-page apps should use the History API (/products/blue), not #/products/blue, which Google ignores.
• Lazy-loaded content appears without scrolling or clicking. How to check: content that loads only after a user action may never be seen by crawlers.

4. Speed and Core Web Vitals

• Core Web Vitals pass on mobile. How to check: the Core Web Vitals report in Search Console. Good means LCP ≤ 2.5 s, INP ≤ 200 ms and CLS ≤ 0.1 at the 75th percentile of real users. Our Core Web Vitals guide explains the fixes.
• Server response time stays under about 0.8 seconds. How to check: measure TTFB on uncached pages. Page caching and a CDN fix most slow responses.
• Text is compressed and served over HTTP/2 or HTTP/3. How to check: the Content-Encoding header should show br or gzip.
• Images are sized, compressed and modern. How to check: look for AVIF or WebP, explicit dimensions, and loading="lazy" below the fold but never on the main hero image.
• Render-blocking resources are minimal. How to check: scripts in the head should use defer or async; critical CSS should be small or inlined.

5. Mobile

• The mobile version has the same content. Google uses mobile-first indexing, so anything missing on mobile, including text, links, structured data and meta tags, effectively doesn’t exist. How to check: compare the mobile and desktop HTML of key templates.
• A viewport tag and responsive layout. How to check: look for <meta name="viewport" content="width=device-width, initial-scale=1"> and test at phone widths. Google retired its Mobile-Friendly Test in December 2023, so use Lighthouse or our mobile-friendly test.
• Readable text and usable tap targets, with no intrusive interstitials covering the main content on arrival.

6. Security

• HTTPS everywhere with a valid certificate. How to check: confirm the certificate isn’t close to expiring and renews automatically, and that HTTP redirects to HTTPS in one hop.
• No mixed content. How to check: on HTTPS pages, images, scripts and styles must not load over http://.
• HSTS once HTTPS is stable, plus sensible security headers such as X-Content-Type-Options and a Content Security Policy. They aren’t ranking factors, but they reduce the risk of the hacks that do destroy rankings.
• No security issues or manual actions. How to check: the Security issues and Manual actions reports in Search Console.

7. Structured data

• The right types for each template. Organization on the home page; Article, Product, LocalBusiness or BreadcrumbList where they fit. See the schema markup guide.
• Valid markup that matches visible content. How to check: the Rich Results Test for single pages and Search Console enhancement reports for the whole site.
• No reliance on retired features. FAQ rich results stopped showing in May 2026 and HowTo in 2023; the markup is harmless but won’t produce rich results.

8. International

• Hreflang is complete. Every version lists itself and all alternates, codes are valid (en-GB, not en-UK), and every alternate links back.
• Canonicals stay within each language. A translation should never canonicalize to the original.
• The lang attribute is set and visitors aren’t force-redirected by IP address. Details are in the hreflang guide.

9. AI crawlers and AI search

• Your robots.txt policy is deliberate. Decide separately for AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot), user-triggered fetchers and training crawlers (GPTBot, ClaudeBot, CCBot). Blocking search bots removes you from those assistants’ answers.
• Your CDN or firewall agrees with robots.txt. How to check: the AI crawler checker tests 25+ AI bots against your robots.txt and requests the page with real AI user agents to catch 403 blocks.
• Snippet directives don’t hide you. nosnippet or max-snippet:0 on a page also keeps it out of Google’s AI Overviews.
• Optional extras are in perspective. llms.txt is a low-cost proposal no major platform has confirmed using for ranking. IndexNow speeds up discovery on Bing, Yandex and others, but not Google.

How often to run this checklist

Run the full checklist before and after any migration, redesign, domain change or CMS upgrade; that’s when indexing accidents happen. Otherwise, review the Search Console reports monthly and repeat the full list every quarter. Keep a short log of what you changed and when, so a traffic drop can be matched to a cause.

Run a technical audit

Most of these checks can be run in one go. The website SEO audit gives you a free single-page audit of status codes, indexability, canonicals, robots rules, hreflang, schema, speed signals, security headers and AI crawler access, ranked by impact. On paid plans it crawls the whole site (up to 100 pages on Pro, 250 on Agency) to find broken internal links, duplicate titles and descriptions, thin pages and other site-wide issues. Use Search Console alongside it to confirm what Google actually indexed.