Technical SEO, Explained: What We Found When We Crawled Our Own Site

In the early hours of 10 October we ran a full crawl of our own website, the way we would for a client. Sixteen of our product pages, the ones our pricing pages told people to click, were showing visitors a single line: “Great things are on the horizon.” Twenty-one words. Our Terms & Conditions page said the same thing. Anyone who wasn’t logged in saw it, and so did Google, because the pages were open to indexing. We had built the pages, priced them and linked to them, and the only people who could see them were us.

Nothing was broken in the way most people picture a broken website. No error page, no crash. One setting in our shop plugin was doing exactly what it was told. That is technical SEO in a sentence: making sure what you built is what Google can actually find, read and show. This guide explains it in plain English, using what we found on our own site on 10 and 11 October 2026 as the worked example.

What technical SEO means

Google describes SEO as a whole like this: “SEO—short for search engine optimization—is about helping search engines understand your content, and helping users find your site and make a decision about whether they should visit your site through a search engine.”^[4] Our guide to what SEO is covers the whole picture.

Technical SEO is the part a visitor never sees. It isn’t your words or your links from other sites. It is the plumbing underneath: whether Google can reach each page, whether it is allowed to store it, whether there is one clear version of it, and whether it loads quickly and steadily on a phone. Good writing on a page Google can’t index ranks for nothing.

Crawled, indexed, served: the three stages

Google says its search “works in three stages, and not all pages make it through each stage”:^[1]

• Crawling: “Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.”
• Indexing: “Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database.”
• Serving: “When a user searches on Google, Google returns information that’s relevant to the user’s query.”

Before any of that, Google has to know the page exists. It calls this “URL discovery”: some pages it already knows, “Other pages are discovered when Google extracts a link from a known page to a new page”, and “Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl.”^[1] And Google is blunt about the limits: it “doesn’t guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials.”^[1]

Almost every technical SEO problem is a page stuck at one of those stages. The fix depends on which one.

What we found on our own site, 10–11 October 2026

We sell SEO, so we hold our own site to the standard we sell. Here is what the crawl and Google’s own reports turned up, and what we did about each.

1. Pages Google could see, saying nothing

The “Great things are on the horizon” pages came from the “coming soon” mode in WooCommerce, our shop plugin, set to hide store pages only. Our online store isn’t open yet, so hiding it was right. But the setting also covered our Terms page, because WooCommerce uses it as the shop’s terms, and every pricing button still linked to the hidden product pages. On 10 October we pointed those buttons to our enquiry form instead, told Google not to index the product pages (a noindex tag, explained below), took them out of the sitemap and gave the Terms page back its real content.

The lesson: a page can return “200 OK”, look fine to you when logged in, and still show Google something else entirely. Check pages logged out.

2. Guides Google had never heard of

On 11 October we checked eight of our SEO guides in Google Search Console’s URL Inspection. Four were indexed. Three came back as “URL is unknown to Google”, which Google’s help page explains as: “This means that Google hasn’t seen this URL before.”^[7] One more was “Discovered – currently not indexed”, which Google describes as “The page was found by Google, but not crawled yet.”^[8]

All four were live, open to indexing and listed in our sitemap. The oldest, our guide to what SEO is, had been published on 4 August, more than two months earlier. That is the point Google makes about sitemaps: one “helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexed.”^[3] A young site with few links pointing to it gets crawled less, so new pages wait. The fix is to request indexing for each page in Search Console and to link to it from pages Google already visits often. We will report how long it takes in a dated update.

3. Thirty-four “broken links” that weren’t

The crawl reported 34 broken links. Every one was an email address that our CDN rewrites to hide it from spam bots. The rewritten link works for people in a browser but looks broken to a crawler. We left them alone. Not every red line in an audit report is a problem, and an agency that “fixes” all of them is billing you for noise.

4. Smaller things worth fixing

• Two main headings on three pages. The theme printed the page name as a heading and the page content added its own. We kept one per page on 11 October. Headings are on-page SEO more than technical, and our on-page SEO checklist covers them.
• Clickjacking protection. Our pages didn’t tell browsers that only our own site may show them inside a frame. We added the two response headers that do this on 11 October. It’s a security fix rather than a ranking one, but it is part of the same plumbing.
• Speed on mobile. On 10 October our homepage scored 76 for mobile speed in Google PageSpeed Insights. After we stopped loading shop scripts on pages that don’t sell anything and resized two oversized images, it scored 83, with the largest image appearing in 3.7 seconds instead of 4.2. Scores move between runs, so treat them as a guide.

The technical SEO checklist, in plain English

Can Google find the page?

Links from your own pages are the main way. Google says “If your site’s pages are properly linked, Google can usually discover most of your site.”^[3] A sitemap (a file listing your pages for search engines) helps, especially for new pages. A page with no links to it, sometimes called an orphan page, is easy to miss.

Can Google crawl it?

The robots.txt file, a small text file at the root of your site, “tells search engine crawlers which URLs the crawler can access on your site.” Google adds a warning many site owners miss: it is “not a mechanism for keeping a web page out of Google.”^[5] Blocking a page in robots.txt stops the crawl, not necessarily the listing.

Is it allowed in the index?

That is the job of noindex, a line in the page’s code that asks search engines not to list it. When Google finds it, “Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.” But it only works if Google can read it: “the page or resource must not be blocked by a robots.txt file.”^[2] Use noindex for pages that shouldn’t appear, like our unopened shop. And check it hasn’t been left on pages that should, a common slip after a site launch.

Is there one version of each page?

Your homepage can often be reached four ways: with and without “www”, over http and https. Each should redirect to one version. Copies of the same page, such as filtered lists or old drafts left live, should point to the original with a canonical tag, the line of code that tells Google which copy is the main one.

Is it fast and steady on a phone?

Google measures this with Core Web Vitals, “a set of metrics that measure real-world user experience for loading performance, interactivity, and visual stability of the page.” It recommends an LCP (when the main content appears) “within the first 2.5 seconds of the page starting to load”, an INP (how quickly the page responds to a tap) “of less than 200 milliseconds”, and a CLS (how much things jump around while loading) “of less than 0.1”.^[6] Google says good scores align “with what our core ranking systems seek to reward”.^[6] In our experience speed rarely beats relevance, but a slow page loses visitors who did click. Our free website speed test runs the same Google check.

Is it secure?

HTTPS on every page, no “not secure” warnings, plugins kept up to date. A hacked WordPress site can be filled with spam pages in your name, which is a technical SEO problem as much as a security one. Our WordPress maintenance page explains what regular care covers.

What not to do

• Don’t use robots.txt to hide a page. Use noindex, and leave the page crawlable so Google can read it.
• Don’t judge a page only while logged in. Our placeholder pages looked perfect to us. Open your key pages in a private window.
• Don’t pay anyone to “get you indexed faster”. Google says it “doesn’t accept payment to crawl a site more frequently, or rank it higher. If anyone tells you otherwise, they’re wrong.”^[1]
• Don’t chase a perfect audit score. Fix what stops pages being found, indexed or used. Leave the false alarms.
• Don’t leave old copies live. “Home (old)” and “About copy” pages from a redesign compete with the real ones. Delete or redirect them.

Check it yourself

• Is a page indexed? In Google Search Console, paste the address into URL Inspection. If it says the URL is not on Google, use “Request indexing”, which asks Google to crawl it.^[7]
• Which pages aren’t? Open Indexing, then Pages, in Search Console. Read the reasons. “Excluded by noindex” on a page you want found is the one to fix first.
• Is noindex on a page by mistake? Open the page, right-click, choose “View page source” and search for robots. “index, follow” is what you want on pages you want found.
• What do visitors see? Open your pricing, service and contact pages in a private window, on your phone.
• How fast is it? Run your homepage through our website speed test on mobile and read the top two fixes it lists.

Frequently asked

What is technical SEO?

It is the work that makes sure search engines can find, crawl and index your pages and that they load quickly and safely. It covers links between pages, sitemaps, robots.txt, noindex and canonical tags, redirects, page speed, mobile use and security. It doesn’t cover your words or links from other sites, which are on-page and off-page SEO.

Does a small business website need technical SEO?

Yes, but usually not much of it. A ten-page site mostly needs the basics checked once and then after every big change: every page indexed, no stray noindex, one version of each page, and a decent mobile speed. Larger shops and catalogues need more ongoing work.

How long does Google take to index a new page?

There is no fixed time, and Google doesn’t guarantee it will index a page at all. On our own young site, a guide published on 4 August was still unknown to Google on 11 October, despite being in the sitemap. Linking to new pages from pages Google already visits, and requesting indexing in Search Console, helps.

Does page speed affect Google rankings?

Google recommends good Core Web Vitals and says they align with what its ranking systems reward, but relevant content still matters most. Treat speed as something that keeps the visitors you earn, not a shortcut to the top.

Want us to check your site’s technical SEO?

Send us your website. We will check whether your main pages are indexed, look for stray noindex tags, duplicate versions and slow pages, and reply in writing with what to fix first and what to ignore. Request a free written review, read how a full SEO audit works, or see our SEO services.

References

1. Google Search Central, “In-depth guide to how Google Search works”, last updated 18 December 2025. Accessed 12 October 2026.
2. Google Search Central, “Block Search indexing with noindex”, last updated 10 December 2025. Accessed 12 October 2026.
3. Google Search Central, “Learn about sitemaps”, last updated 10 December 2025. Accessed 12 October 2026.
4. Google Search Central, “SEO Starter Guide: The Basics”, last updated 10 December 2025. Accessed 12 October 2026.
5. Google Search Central, “Introduction to robots.txt”, last updated 10 December 2025. Accessed 12 October 2026.
6. Google Search Central, “Understanding Core Web Vitals and Google search results”, last updated 10 December 2025. Accessed 12 October 2026.
7. Search Console Help, “URL Inspection tool”. Accessed 12 October 2026.
8. Search Console Help, “Page indexing report”. Accessed 12 October 2026.
9. Total Web Solutions records: full crawl of totalwebsolutions.co finished 9 October 2026 (19:38 UTC, 10 October in Malaysia); store, Terms and pricing-button changes of 10 October 2026; URL Inspection of eight guides on 11 October 2026; heading and response-header fixes of 11 October 2026; Google PageSpeed Insights runs of the homepage on 10 October 2026.