Make your website crawl-friendly

If your website contains data that you want to make widely available, e.g. if you're a College wanting to widely publicise your courses, then it might be of benefit to make your site scrape-friendly.

Making your site scrape-friendly means that other organisations and individuals can more easily build web-crawlers to harness your data and republish/share on other parts of the internet.

Benefits include:

  • Your data is easily scraped by third-party web services, expanding the reach of your data.
  • Data scraped from your site is more accurate, as complicated selectors are not needed to gather the data.
Make your website crawl-friendly