MozCon 2020 Resouces

Published 14th July 2020, by Rob Ousbey

Examples of how to use a novel technique to build a Google Scraper, SERP Analysis tool, and more.

Thanks for watching my Mozcon 2020 session: A Novel Approach to Scraping Websites.

As promised, here are some additional resources as well as the bookmarklets that I built during the session.

Google Site Structure Scraper

Step 1: Drag this bookmarklet to your browser's bookmark bar

Google Structure Scraper

Step 2: Go to Google, and click the bookmark to inject the code

Step 3: Enter a domain name (or subdomain) in the box, and hit the 'crawl site' button.

Step 4: After crawling is finished, hit the same button again to discover more folders from the same site.

Download the code for this example here. Try reading the code and extending it, to create your own tools that can extract useful data from Google.

SERP Analysis: Lighthouse Data in Moz Pro

Step 1: Drag this bookmarklet to your browser's bookmark bar

Moz Pro / Lighthouse

Step 2: Log into Moz Pro (or sign up for a free trial)

Step 3: Go into a campaign, then to an 'Analyze a Keyword' page, and wait for the SERP analysis table to load

Step 4: Click the bookmarklet to inject the code

Step 5: Click one or more 'Run Lighthouse' links, and wait for data to be collected.

Download the code for this example here. Try reading the code and extending it, to create your own tools that combine Moz Pro or Lighthouse data with other sources.

You can run your own independent Lighthouse tests at web.dev/measure/, or read more about Lighthouse.

Crawl a Publishing Site

Step 1: Drag this bookmarklet to your browser's bookmark bar

G2 Crawler

Step 2: Visit any page on G2.com, and hit the bookmarklet to inject code into the page

Step 3: Enter a list of product names, and hit the 'Go' button

Step 4: Wait while the site crawls itself, and places information about each product into the table

Download the code for this example here. Try reading the code and extending it, to create your own tools that can crawl & extract data from sites.

Other Resources

Let's keep in touch!