Screaming Frog: how to audit your website step by step

Short answer. Screaming Frog SEO Spider is a program that crawls your website the way a search engine would and gives you back, in a table, every URL with its response code, its links and its problems. The free version goes up to 500 URLs. For a basic audit: crawl, look at response codes and Issues, export and fix first what breaks pages.

What is Screaming Frog and what is it for in an audit?

Screaming Frog SEO Spider is a desktop application. You install it on your computer (there are versions for Windows, macOS and Ubuntu), give it a URL and it starts following the links of your website, page by page, just as a crawler would. According to its getting started guide, by default it crawls in a way similar to Google.

What it gives you is an X-ray. A table with all the URLs it has found and, for each one, data such as the response code, whether it is indexable or not, which pages link to it and which ones it links to. It is a map of what exists on your website as a crawler sees it.

Imagine a furniture shop as an example, made up. It has 800 URLs, a blog nobody has touched in two years and a catalog whose structure changed a year ago. Nobody knows how many pages return 404 or how many internal links point to them. With a half-hour crawl you know. Without it, you guess.

Is it free? What are you missing without a license?

The free version crawls up to 500 URLs at a time. For a small website or for practice, that's enough. The license, which is annual, removes that limit and opens up configuration, saving and reopening crawls, JavaScript rendering and the connection with Google Search Console, according to the tool's frequently asked questions page. If the license expires, you go back to the free version, with its 500 URL limit and without being able to open saved crawls.

My judgment: start with the free one. If your website goes beyond 500 URLs or you need to save crawls to compare before and after, then the license is worth it. Check the current price on its pricing page, which changes over time.

How do I do the first crawl?

There are few steps. The getting started guide explains them like this:

1. Download and install the program.
2. If you have a license, go to Licence > Enter Licence Key and restart the application. If you use the free one, skip this step.
3. Choose Spider mode, type your home page URL in the "Enter URL to spider" field and press Start.
4. Watch the table fill up. At the bottom you see the speed and how many URLs are left. You can pause and resume.

Spider mode crawls the URL you give and all those it discovers through HTML links on the same subdomain. There is another mode, List, to upload or paste a list of specific URLs. It works very well when you already know which pages you want to review, for example those in your sitemap.xml.

If you have an SSD, the guide recommends database storage, which is the default. Without an SSD, use memory mode. You set it in File > Settings > Storage Mode.

Which settings do I touch before starting?

Almost none, the first time. The tool comes configured to crawl in a way similar to Google, and the guide says you don't need to touch anything to crawl a website.

Three settings are worth knowing, and all three hang from the Configuration menu (in the paid version):

• Crawl a subfolder. If you only want to audit your blog, start the crawl from that folder.
• Exclude and Include. They serve to not crawl URLs that match a pattern, or to crawl only those that match. Useful if your website has filters that generate thousands of URLs.
• JavaScript rendering. If your content or your links load with JavaScript, enable it in Configuration > Spider > Rendering. The program will execute the JavaScript and crawl the already rendered HTML.

If your website is small and ordinary, leave everything as it comes.

How do I read the results without getting lost?

The screen is scary the first time, but it is organized in three zones.

The table at the top. The Internal tab gathers everything the program has discovered on your website. There are many columns, among them Indexability (whether the URL is indexable or not) and Indexability Status, which explains the reason when it isn't, for example if it is canonicalized to another URL. If you want to review what a canonical is, I explained it in what is a canonical.

The tabs and their filters. Each tab focuses on one element and has filters to keep what is problematic. The Response Codes tab with the Client Error (4xx) filter shows you the 404 pages it has found, for example.

The bottom panel. When you click a URL at the top, at the bottom you see detail: inlinks (the pages that link to it) and outlinks (the pages it links to). It is what tells you where the broken link you have to fix is.

On the right you have the Overview tab, with a summary of all the tabs and filters, and the Issues tab. In Issues, the program lists problems, warnings and opportunities, each with a type and an estimated priority. Each explanation inside the application says what it is, why it matters and what to do. According to the guide, there are more than 300 detectable issues. To start, Issues is the most comfortable starting point.

What do I check first in a half-hour audit?

This is my order, which is my own judgment and not a Google rule. It works for a small or medium website.

1. Response codes. In Response Codes, look first at 4xx and 5xx errors, and then at 3xx redirects. An important page that gives an error is the most urgent.
2. Who links to what is broken. For each URL with an error, open the bottom panel and look at its inlinks. Fixing a broken link is usually correcting one line on the page that contains it.
3. Indexability. In Internal, filter by the non-indexable URLs and check that they are so on purpose. A page you want to rank that appears as non-indexable is an alarm.
4. High-priority Issues. Sort by priority and read only the first ones. Don't try to solve all 300 types at once.
5. Compare with your sitemap. If you already have a sitemap.xml, crawl it in List mode and see whether all those URLs respond 200 and can be indexed. A sitemap with broken or non-indexable URLs is a sign that something is getting out of control.

With this you have a short task list, ordered by what does the most damage.

How do I export the data and what do I do with it?

At the top left you have the Export button, which sends to a spreadsheet the tab and filter you are viewing. For the data in the bottom panel, right-click on the URL.

Two exports are worth remembering, according to the getting started guide:

• Bulk Export, Client Error (4XX) Inlinks option. It gives you the list of pages that link to pages with errors. It is the pending work ready to be distributed.
• Bulk Export > Issues > All. It exports every detected problem, including the variants with inlinks, in one sheet per problem.

My advice: don't send anyone the full dump. A list of 3,000 rows isn't read and isn't executed. Filter what is important, assign an owner to each group and distribute short tasks. A useful report fits on one page.

If you prefer that I do that diagnosis and hand you the tasks ordered by priority, you can see what my SEO audit includes.

Which mistakes are made most when starting?

This comes from my experience and from what I see in people who are starting. It is my own judgment.

• Treating every warning as a serious error. Issues includes opportunities and warnings, not just failures. The program assigns a priority to each one for a reason.
• Crawling without knowing what you are looking for. Define the question first: are there broken links?, what can be indexed?, what happens with redirects? A crawl without a question is a mountain of data.
• Forgetting JavaScript rendering. If your website loads content with JavaScript and you crawl without rendering, you may see half the website.
• Creating a huge list and closing nothing. It is better to fix ten important things now than to document three hundred.
• Not repeating the crawl. An audit that isn't checked again proves nothing. After fixing, crawl again and compare.

What doesn't Screaming Frog tell you?

Screaming Frog tells you what a crawler sees when it enters your website. It doesn't tell you what Google has decided to index, nor how many times it has shown you, nor with which queries. That lives in Search Console. The two tools complement each other: the crawl shows you how your website is built, and Search Console shows you how Google treats it. If you want to see what to look at there and in what order, you have it in Search Console from scratch.

Nor does it decide for you. That something appears flagged doesn't mean it has to be changed. A page with noindex can be fine like that. A robots.txt that blocks a folder can be what you wanted. The tool detects; the judgment is yours. If you want to review what each file does, you have robots.txt and sitemap.xml.

Frequently asked questions

Is Screaming Frog free?

It has a free version that crawls up to 500 URLs. The paid license is annual and removes that limit, in addition to opening up configuration and other features.

Do I need to know how to code to use it?

No. It is an application with a graphical interface. For your first crawl you only need your website's URL. Advanced settings, like JavaScript rendering, you can leave for later.

How long does a crawl take?

It depends on the size of your website and your computer. You can pause and resume the crawl at any time. For a small website, we're talking minutes.

Does it work to see the 404 pages on my website?

Yes. In the Response Codes tab, the Client Error (4xx) filter lists them, and in the bottom panel you see which pages link to them.

Can it replace Search Console?

No. Crawling your website tells you how it is built. Search Console tells you what Google does with it. It is best to use both.

Sources

• Screaming Frog, Getting Started Guide.
• Screaming Frog, SEO Spider FAQ.
• Screaming Frog, User Guide: Tabs.