Crawling comes first

A search engine can only rank what it can reach and read, so crawling comes before everything else. Googlebot follows links, fetches pages, may render their JavaScript, and then decides what to index. A failure at any stage makes everything after it irrelevant.

When a page isn't showing up, work in that order. Can Google fetch it? Is it blocked? Does it return a 200? Does it point its canonical somewhere else? The URL Inspection tool in Search Console answers most of these for a given address.

robots.txt manages crawling, not indexing

A robots.txt file tells crawlers which parts of a site to skip, mostly to keep them from overloading your server. Google is clear that it isn't a way to keep a page out of its results. A blocked page's address can still appear, just without a description.

To keep a page out of search, use noindex or password protection. The two can conflict: if robots.txt blocks a page, Google can't crawl it to see the noindex.

We handle this deliberately on our own site. Our admin area is disallowed, but our payment and checkout pages aren't. They carry noindex instead, so crawlers can actually read that instruction.

Sitemaps list what you want found

A sitemap is a file listing the pages you want search engines to know about. Google says a sitemap doesn't guarantee crawling or indexing, and that sites of about 500 pages or fewer with good internal links may not need one, though most sites still benefit.

Three rules cover most of it, per Google's sitemap instructions. Use full canonical URLs. Keep each file under 50,000 URLs or 50 MB uncompressed. And only trust lastmod if it's honest, because Google uses it only when it's consistently accurate. It ignores changefreq and priority.

Ours is generated at build time. Each page's date comes from the last git commit that touched its source file. If that date is unknown, or the file has uncommitted changes, we leave lastmod out rather than guess.

Canonicals give each page one address

A canonical tells search engines which address is the preferred one when several show the same content. The classic culprits are www and non-www, with and without a trailing slash, and tracking parameters.

Google treats canonicals as signals, not commands. Redirects and rel="canonical" are strong signals, and sitemap inclusion is a weak one. Combining them helps, so long as they agree. Avoid relative canonical paths, avoid using noindex to pick a canonical, and don't use robots.txt for it.

We use one host and one form: https://mediacitydesign.com/page/, without www, with a trailing slash. Internal links use that form too, so nobody takes an extra redirect hop.

Redirects and status codes tell the truth about a page

Status codes tell a crawler what happened, so each one should match reality. Here is how Google handles the common ones:

CodeMeaningGoogle's handling
200OKPassed on for possible indexing, which isn't guaranteed
301Moved permanentlyFollowed, a strong signal that the target is canonical
302Found (temporary)Followed, a weak signal, and the original usually stays
404, 410Not found, goneRemoved from the index if it was indexed, and crawled less
429, 5xxOverloaded or server errorCrawling slows, and persistent failures drop out of the index

Two mistakes come up often. The first is a soft 404: a "page not found" message served with a 200 status, which Search Console flags. The second is using a 302 for a move that's actually permanent.

Google recommends server-side redirects when you can use them, with JavaScript redirects as a last resort. A redesign that changes addresses without redirects leaves old links and search results pointing at dead pages. Our website redesign page covers how we plan for it.

Mobile is the version Google indexes

Google indexes the mobile version of your content, so anything missing on phones may as well not exist. It recommends responsive design, with the same HTML at the same address on every device.

Accordions and tabs are fine for keeping phone layouts compact. Just don't load primary content only after a click or a swipe, because Googlebot doesn't do either.

Core Web Vitals measure what visitors feel

Core Web Vitals are three measures of loading, responsiveness and visual stability. Web.dev says a page meets the target when all three are good at the 75th percentile of page loads, segmented by mobile and desktop.

MetricMeasuresGood
LCP (Largest Contentful Paint)Loading2.5 seconds or less
INP (Interaction to Next Paint)Responsiveness200 milliseconds or less
CLS (Cumulative Layout Shift)Visual stability0.1 or less

Those are field numbers, meaning real visits. A single run on your office wifi doesn't tell you the same thing. Use PageSpeed Insights and the Core Web Vitals report in Search Console.

JavaScript should not hide your content

Google can render JavaScript, but rendering is an extra step that can add delay, and not every crawler runs it. Google's JavaScript guide describes pages waiting in a rendering queue and still recommends server-side or pre-rendering.

We build this site with Astro, which prerenders most pages to plain HTML. A few pages that read live data, like hosting plans, render on the server. Either way, the crawler gets real content in the first response. If your site is a client-side app, use the crawled page view and live test in URL Inspection to see what Google received.

Structured data must match the page

Structured data is a standard way to label what a page is about, and it has to describe what visitors can actually see. Google recommends JSON-LD in most cases and warns against marking up information that isn't visible, even if it's accurate. Valid markup can make a page eligible for rich results, but doesn't guarantee them.

Test with the Rich Results Test. On our own pages, we don't publish review or rating markup without a source. Our build check flags any that appears, so a template change can't quietly add stars nobody can back up.

Internal links show what matters

Internal links are how crawlers and people find your pages, so every page you care about needs at least one. Use a real <a> element with an href, since Google generally can't crawl anything else.

Write anchor text that makes sense on its own. "Click here" tells nobody anything. "Website redesign" does.

Images need real markup too

Use standard image elements and describe them, because Google doesn't index CSS background images. Put images in <img> with useful alt text and a descriptive filename, and use srcset with a fallback src for different screens. Setting width and height also reserves the space, which reduces layout shift.

How we check our own site

We check the built site with a script, because people miss things and templates fail in bulk. seo-check reads the finished pages and reports problems including:

  • Any indexable page without exactly one correct canonical on the right host
  • Structured data that doesn't parse, or references that lead nowhere
  • Sitemap entries with no built page, and indexable pages missing from the sitemap
  • Internal links that are broken, lack a trailing slash, or point at a redirect
  • Any review or rating markup

None of this means every page passes every measure, and we don't claim that we pass Core Web Vitals. It means a whole class of mistakes gets caught by a machine instead of by a customer. For help on your own site, see our SEO services.

Written by Romik Hacobian, founder of Media City Design, a Burbank web design studio since 2021.

Sources