Many factors play key roles in a website’s search engine ranking, including keywords and backlinks. But a site structure that makes crawls simpler for search engine bots is also...
Many factors play key roles in a website’s search engine ranking, including keywords and backlinks. But a site structure that makes crawls simpler for search engine bots is also essential.
Indexing allows a website to show up in search results, and to make that happen, bots should be able to effectively crawl your site. Looking for an unindexed site is next to impossible (unless a recent indexing glitch has happened), and it would make the site practically non-existent since it can’t be found anywhere on the web.
And that’s why a site structure that enables efficient crawls is a vital part of a successful on-page SEO strategy.
SEO experts use various tools to ensure that search engine bots can crawl and index websites. Screaming Frog is one of these tools — it’s a web crawler that helps you audit your site and gives you insights on its performance.
The software might be tricky to use if you’re starting out, but don’t fret. Our in-depth Screaming Frog guide will help you explore and use the tool to your advantage.
Screenshot from Screaming Frog on YouTube
Screaming Frog works like Google’s crawlers: it lets you crawl any website, including e-commerce sites. But this SEO spider tool takes crawling up by a notch by giving you relevant on-site data and creating digestible statistics and reports. With simpler site data from Screaming Frog, you can easily see which areas your website needs to work on.
The Screaming Frog software can help you perform the following tasks for your SEO efforts:
And of course, you can do a lot more with Screaming Frog. We’ll cover these extra features in another section.
Before performing any site crawls through the Screaming Frog SEO spider tool, you’ll need to install the software on your computer. You can download the tool from Screaming Frog’s website.
The software works on Windows, Mac, and Linux. Here’s how to install Screaming Frog on any of these systems:
You can use the Ubuntu user interface or command line function to install Screaming Frog on your computer.
1. Type the following command in an open terminal window:
Screenshot from Screaming Frog
2. Enter your password.
3. Type Y to continue installing the software and accept the ttf-mscorefonts-install End User License Agreement (EULA).
Do keep in mind that you’re installing a free version of Screaming Frog. The free version lets you crawl a maximum of 500 URLs, and you’ll need a license to crawl more websites. You can buy a software license on the Screaming Frog website.
To start using Screaming Frog’s premium version, go to the License menu. Click on the “Enter License” option and type in your license key. You should see a dialog box that shows the license’s validity and its expiry date.
After installing the Screaming Frog tool, it’s best to study the software’s menus, options, and settings. Let’s take a closer look at the different menus and settings that you can use.
This menu lets you save a crawl as a file and see your last six (6) crawls you performed if you haven’t saved them yet. Plus, the File menu will let you set default settings for the software.
If you’re after custom crawl settings, Screaming Frog’s Configuration menu allows you to set and adjust those options. You can use the options below to customize your crawl settings.
As its name suggests, the Bulk Export menu lets you export multiple URLs. Screaming Frog will accept links with various site elements, including:
The Reports menu creates downloadable crawl overviews and on-site data reports. You can use these reports in your SEO audits.
With this menu, you can create your website’s sitemap. You’ll learn more about constructing sitemaps in the latter part of this Screaming Frog SEO tutorial.
Site crawls apply to big and small websites. But crawling larger websites consumes more memory and processing power. With Screaming Frog, you can allot a certain amount of your device’s memory for crawling websites. The software has database and RAM storage modes for this purpose.
Screaming Frog’s database storage mode is ideal for users with solid-state drives (SSDs). To use the mode, follow these steps:
Meanwhile, users without SSDs can settle for the RAM storage mode, with a default setting of 1GB RAM for 32-bit computers and 2GB RAM for 64-bit devices. A lower RAM prevents freezing and crashes while crawling with Screaming Frog.
But if you want to allocate more RAM for crawls, you can change the default RAM setting to a higher number. The changes will apply after you restart Screaming Frog.
Screaming Frog already follows default settings for site crawls. But you can tweak these settings and collect specific data using the software’s numerous tools. By using these tools, your crawls will take up less time and processing power. You can customize your settings from the Configuration menu.
Along with customizing settings, Screaming Frog lets you tweak windows and columns for easier access. When you run the software, you’ll see three (3) windows, and the right-side column allows you to access SEO elements and filters. On the other hand, you’ll find a window directly below the main that displays specific web page data.
Adjusting window sizes is a great way to customize your view in Screaming Frog. You can resize windows by dragging them to your desired size. Plus, the software will let you customize columns any way you want.
Clicking and dragging a column will move it to your desired position, while clicking a column will sort your data. For instance, if you want to sort your columns by number, you can simply click on the column and the software will arrange the numbers from highest to lowest.
Screaming Frog stands out from other SEO tools because it lets you search, filter, and customize your crawls. It also offers a wealth of benefits, from analyzing site enhancements and errors to filtering the way your webpages show up.
From website crawls to sitemap creation, here’s how to perform some SEO tasks in Screaming Frog.
Most of the time, you can just crawl websites with Screaming Frog’s basic tools. But as you explore the software, you’ll find more helpful features for site crawls. Here are some of its other features you might have missed:
SEO audits help you identify possible changes to your website’s various elements. This way, you can climb to the top of the search ladder and surpass your competitors. Plus, you can use insights from your audits to improve UX and other technical elements of your site.
With a reliable partner on your side, you can get a closer look at your site performance and find out how you can improve it further. Get in touch with Growth Rocket today and we’ll help you slowly make your way to the top of the search rankings.
HTTP/2 is a significant upgrade from HTTP/1.1, offering faster page loading times, reduced latency, and improved security. It's essential for modern websites, especially those with resource-heavy content or a focus on mobile performance.
AI is transforming SEO. It’s starting to automate time-consuming tasks and helping marketers analyze massive amounts of data. Find out how you can maximize the benefits of AI for SEO by achieving harmonious human-AI collaboration.
Competitor keyword analysis can put you ahead of your competitors on Google’s SERPs. Here’s why you should conduct one for your business.
Video media has evolved over the years, going beyond the TV screen and making its way into the Internet. Visit any website, and you’re bound to see video ads, interactive clips, and promotional videos from new and established brands.
Dig deep into video’s rise in marketing and ads. Subscribe to the Rocket Fuel blog and get our free guide to video marketing.