New Site Crawl: Rebuilt to Find More Issues on More Pages, Faster Than Ever!. First, the good news — as of today, all Moz Pro customer have access to the new version of Site Crawl, our entirely rebuilt deep site crawler and technical SEO auditing platform. View/search *all* URLs One major limitation of our old crawler is that you could only see pages with known issues. You can export your entire crawl to CSV, with additional data including per-page fetch times and redirect targets. Recrawl your site immediately Sometimes, you just can't wait a week for a new crawl. No problem, just click "Recrawl my site" from the top of any page in the Site Crawl section, and you'll be on your way... Prioritize quickly & tactically Prioritizing technical SEO problems requires deep knowledge of a site. With the new Site Crawl, we decided to categorize crawl issues tactically, using five buckets: Critical Crawler Issues Crawler Warnings Redirect Issues Metadata Issues Content Issues Hopefully, you can already guess what some of these contain. All else being equal, one 500 error may be more important than one duplicate page, but 10,000 duplicate pages is a different matter. Even better, we've been crawling existing campaigns with the Aardwolf engine for a couple of weeks, so you'll have history available from day one!
First, the good news — as of today, all Moz Pro customer have access to the new version of Site Crawl, our entirely rebuilt deep site crawler and technical SEO auditing platform. The bad news? There isn’t any. It’s bigger, better, faster, and you won’t pay an extra dime for it.
A moment of humility, though — if you’ve used our existing site crawl, you know it hasn’t always lived up to your expectations. Truth is, it hasn’t lived up to ours, either. Over a year ago, we set out to rebuild the back end crawler, but we realized quickly that what we wanted was an entirely re-imagined crawler, front and back, with the best features we could offer. Today, we launch the first version of that new crawler.
Code name: Aardwolf
The back end is entirely new. Our completely rebuilt “Aardwolf” engine crawls twice as fast, while digging much deeper. For larger accounts, it can support up to ten parallel crawlers, for actual speeds of up to 20X the old crawler. Aardwolf also fully supports SNI sites (including Cloudflare), correcting a major shortcoming of our old crawler.
View/search *all* URLs
One major limitation of our old crawler is that you could only see pages with known issues. Click on “All Crawled Pages” in the new crawler, and you’ll be brought to a list of every URL we crawled on your site during the last crawl cycle:
You can sort this list by status code, total issues, Page Authority (PA), or crawl depth. You can also filter by URL, status codes, or whether or not the page has known issues. For example, let’s say I just wanted to see all of the pages crawled for Moz.com in the “/blog” directory…
I just click the [+], select “URL,” enter “/blog,” and I’m on my way.
Do you prefer to slice and dice the data on your own? You can export your entire crawl to CSV, with additional data including per-page fetch times and redirect targets.
Recrawl your site immediately
Sometimes, you just can’t wait a week for a new crawl. Maybe you relaunched your site or made major changes, and you have to know quickly if those changes are working. No problem, just click “Recrawl my site” from the top of any page in the Site Crawl section, and you’ll be on your way…
…
COMMENTS