What Is Screaming Frog SEO Spider?

Screaming Frog is a company in England. The SEO Spider is the program they make. You install it on your own computer. Point it at a website. It then works through the pages it can reach.

That is the definition. The button-by-button walkthrough already exists elsewhere on this site. So this page will not repeat it. What gets explained far less often is the strange thing about the SEO Spider. Almost everything else in your toolbar hands you a report. The spider goes and looks.

The Definition Takes Three Sentences

Point it at a site. It follows the links. Then it hands you a table.

One row per page. The title. Then the address. Whether the page loaded at all. How long the description ran, and which other pages point at it.

The table is a record of what was there. Anything on it marked urgent got marked by a rule. And the rule never saw your business.

Every Other Machine That Visits Your Site Answers to Somebody Else

Machines come to your website all day. Google sends one. So does Bing, and so do scrapers you have never heard of.

Not one of them is yours.

They arrive on their own schedule. And they read whatever they feel like reading. Then they leave, and they tell you nothing about what happened while they were there.

You Have Never Once Asked Googlebot Anything

A machine belonging to Google has walked your website more times than you could count. Yet you have never been able to ask it about the site as a whole.

Search Console will fetch a single URL for you on demand. Ask about the whole website and what comes back is a summary. Written afterwards. In Google’s categories, on Google’s schedule.

Useful. Still second-hand.

Your link and ranking numbers work the same way. Those came off somebody else’s crawl of the web, run at some point, for their own reasons. Another machine went somewhere. Now you are reading the write-up.

So the Spider Is First-Hand, and Nothing Else in Your Toolbar Is

You press start. It goes. Right now, from your desk, down your own connection, to your own server.

None of that is a report. You sent it, and it went.

A crawler of your own is the closest you will get to standing behind Googlebot while it works. Send a machine down the same corridors. Watch where it stumbles. Trip up one spider and you have probably tripped up the others. Now you have seen it happen rather than guessed at it.

The Crawl Counts. It Cannot Know What Counts.

Then the trouble starts.

A first crawl of a real website is not a pleasant experience. Thousands of rows. Red cells everywhere. Warnings you have never heard of, sitting beside warnings you have.

That reads like thousands of problems. It is not. Instead it is thousands of facts.

Four Hundred Missing Descriptions Is Not Four Hundred Problems

Say the crawl reports four hundred pages with no meta description. Accurate, probably. Also close to meaningless on its own.

Three hundred and ninety could be tag archives no customer has ever landed on. Two could be your services pages.

The spider cannot tell those apart on its own. It has never met a customer. One of those pages exists only because a plugin made it years ago. Nothing in the crawl knows that.

Wire your analytics into it and traffic will appear beside every row. Even then the tool is only holding up a number you gave it. So the list arrives in no order that means anything. You have to give it one.

So a tool with a score is applying a rule. And the rule has to be the same for a plumber with twelve pages as for a retailer with eighty thousand.

Learning the Software Takes an Afternoon

So the skill was never the software.

Anybody can run a Screaming Frog crawl. The buttons are ordinary and the walkthroughs are free. What takes far longer is knowing which columns you are allowed to ignore. No tool arrives holding that knowledge. It is knowledge about your business.

A beginner opens a crawl and sees a wall of problems. Somebody who has been doing search engine optimization since 2007 opens the same crawl and sees a short list. No tool closes that gap, and neither does what machine learning does for SEO.

Read Your Own Crawl Like a Customer Would

Try this on your own site this week.

Run the crawl. Then go down whatever it flagged and put one question to every row. Would a customer of mine ever have arrived here?

The rows that survive are your work for the month. Perhaps plenty survive, and you have found something real. Or perhaps almost nothing does, and you have saved yourself a stretch of tidying pages nobody visits. Both answers are worth having.

No statistic about other people’s websites will tell you which one you are looking at. Yours sits one crawl away.

The One Question It Answers, and the Ones It Cannot

Be clear about the limit, though. The SEO Spider answers exactly one question. Can a machine get to this page and read what is on it?

That is a real question, and plenty of sites fail it quietly. A website audit exists partly to catch it. Yet it is not the question you actually care about, which is whether the page deserves finding at all.

Whether the argument on the page is any good. Or whether the keyword was worth chasing at all. Nothing in the table settles either.

No crawl has ever moved a ranking. It can tell you a page was unreachable. Yet it cannot tell you a page was unwanted. A spreadsheet full of green cells has never proved the content on a page was worth reading.

How To-The-TOP! Uses a Crawler in Calgary

SEO Company To-The-TOP! has been doing this work since 2007. A crawl is where the work starts rather than where it finishes. The list comes back. Then somebody who knows the business decides what on it matters, which is the part no tool has done yet.

That order matters for Calgary businesses in particular. Plenty of them run a few dozen pages, and a handful of those pages earn the money. A crawler treats every one of them the same. To-The-TOP! does not.

The ads side tells the same story in a different costume. Numbers come back from Google Ads management by the dozen. Not one of them says which to act on. The portfolio is where that judgment shows up.

Somebody probably sent a machine through your website this morning. It was not yours, and it left without saying a word. Auditing your own website starts with sending one you control.

Common Questions About Screaming Frog SEO Spider

Is Screaming Frog the same thing as Googlebot?

No. Both are crawlers and they behave similarly, so a page that blocks one will often block the other. Still, the spider has no say in rankings and no idea what Google concluded. It reports only what it could reach.

Do I still need Screaming Frog if I have Google Search Console?

They answer different questions. Search Console tells you what Google did with your site. A crawl tells you what is sitting on it right now. Technical faults often get found faster in the second and confirmed in the first.

Why does a crawl find hundreds of issues on a small site?

It flags patterns rather than consequences. Missing descriptions and long titles get counted wherever they occur, including on pages no visitor has opened. The count is accurate. Yet the importance is yours to supply.

Can a crawl tell me why a page is not ranking?

It can rule some things out. A page that cannot be reached or read is worth knowing about straight away. Past that it goes quiet. It has nothing to say about your competition. Whether your page answers the question better than the ones sitting above it is not something a crawl was built to settle.

Is the SEO Spider hard to learn?

The software is not. Reading the output is. Expect to be running it the same day you install it. Then expect to spend a good deal longer working out which findings deserve an hour of your time.