{"id":68180,"date":"2019-07-17T04:20:25","date_gmt":"2019-07-17T11:20:25","guid":{"rendered":"http:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/"},"modified":"2019-07-17T04:20:25","modified_gmt":"2019-07-17T11:20:25","slug":"the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy","status":"publish","type":"post","link":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/","title":{"rendered":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy"},"content":{"rendered":"<link rel=\"canonical\" href=\"https:\/\/www.smashingmagazine.com\/2019\/07\/ultimate-guide-scalable-web-scrapers-scrapy\/\"><title>The Ultimate Guide To Building Scalable Web Scrapers With Scrapy<\/title><\/p>\n<article>\n<header>\n<h1>The Ultimate Guide To Building Scalable Web Scrapers With Scrapy<\/h1>\n<address>Daniel Ni<\/address>\n<p>                  <time datetime=\"2019-07-16T14:30:59+02:00\">2019-07-16T14:30:59+02:00<\/time><time datetime=\"2019-07-17T11:06:54+00:00\">2019-07-17T11:06:54+00:00<\/time><\/header>\n<p>Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your browser can access the data, you will be able to scrape it.<\/p>\n<p>Realistically, most of the time you could just go through a website manually and grab the data \u2018by hand\u2019 using copy and paste, but in a lot of cases that would take you many hours of manual work, which could end up costing you a lot more than the data is worth, especially if you\u2019ve hired someone to do the task for you. Why hire someone to work at 1\u20132 minutes per query when you can get a program to perform a query automatically every few seconds?<\/p>\n<p>For example, let\u2019s say that you wish to compile a list of the Oscar winners for best picture, along with their director, starring actors, release date, and run time. Using Google, you can see there are several sites that will list these movies by name, and maybe some additional information, but generally you\u2019ll have to follow through with links to capture all the information you want.<\/p>\n<p>Obviously, it would be impractical and time-consuming to go through every link from 1927 through to today and manually try to find the information through each page. With web scraping, we just need to find a website with pages that have all this information, and then point our program in the right direction with the right instructions.<\/p>\n<p>In this tutorial, we will use Wikipedia as our website as it contains all the information we need and then use Scrapy on Python as a tool to scrape our information.<\/p>\n<div data-component=\"FeaturePanel\" data-audience=\"non-subscriber\" data-remove=\"true\"><\/div>\n<p>A few caveats before we begin:<\/p>\n<p>Data scraping involves increasing the server load for the site that you\u2019re scraping, which means a higher cost for the companies hosting the site and a lower quality experience for other users of that site. The quality of the server that is running the website, the amount of data you\u2019re trying to obtain, and the rate at which you\u2019re sending requests to the server will moderate the effect you have on the server. Keeping this in mind, we need to make sure that we stick to a few rules.<\/p>\n<p>Most sites also have a file called <em>robots.txt<\/em> in their main directory. This file sets out rules for what directories sites do not want scrapers to access. A website\u2019s Terms &#038; Conditions page will usually let you know what their policy on data scraping is. For example, IMDB\u2019s conditions page has the following clause:<\/p>\n<blockquote><p>Robots and Screen Scraping: You may not use data mining, robots, screen scraping, or similar data gathering and extraction tools on this site, except with our express-written consent as noted below.<\/p><\/blockquote>\n<p>Before we try to obtain a website\u2019s data we should always check out the website\u2019s terms and <code>robots.txt<\/code> to make sure we are obtaining legal data. When building our scrapers, we also need to make sure that we do not overwhelm a server with requests that it can\u2019t handle.<\/p>\n<p>Luckily, many websites recognize the need for users to obtain data, and they make the data available through APIs. If these are available, it\u2019s usually a much easier experience to obtain data through the API than through scraping.<\/p>\n<p>Wikipedia allows data scraping, as long as the bots aren\u2019t going \u2018way too fast\u2019, as specified in their <code><a href=\"https:\/\/en.wikipedia.org\/robots.txt\">robots.txt<\/a><\/code>. They also provide downloadable datasets so people can process the data on their own machines. If we go too fast, the servers will automatically block our IP, so we\u2019ll implement timers in order to keep within their rules.<\/p>\n<h3>Getting Started, Installing Relevant Libraries Using Pip<\/h3>\n<p>First of all, to start off, let\u2019s install Scrapy.<\/p>\n<h4>Windows<\/h4>\n<p>Install the latest version of Python from <a href=\"https:\/\/www.python.org\/downloads\/windows\/\">https:\/\/www.python.org\/downloads\/windows\/<\/a><\/p>\n<p><strong>Note:<\/strong> <em>Windows users will also need Microsoft Visual C++ 14.0, which you can grab from \u201cMicrosoft Visual C++ Build Tools\u201d over <a href=\"https:\/\/visualstudio.microsoft.com\/downloads\/\">here<\/a>.<\/em><\/p>\n<p>You\u2019ll also want to make sure you have the latest version of pip.<\/p>\n<p>In <em>cmd.exe<\/em>, type in:<\/p>\n<pre><code>python -m pip install --upgrade pip\n\npip install pypiwin32\n\npip install scrapy\n<\/code><\/pre>\n<p>This will install Scrapy and all the dependencies automatically.<\/p>\n<h4>Linux<\/h4>\n<p>First you\u2019ll want to install all the dependencies:<\/p>\n<p>In Terminal, enter:<\/p>\n<div>\n<pre><code>sudo apt-get install python3 python3-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev\n<\/code><\/pre>\n<\/div>\n<p>Once that\u2019s all installed, just type in:<\/p>\n<pre><code>pip install --upgrade pip\n<\/code><\/pre>\n<p>To make sure pip is updated, and then:<\/p>\n<pre><code>pip install scrapy\n<\/code><\/pre>\n<p>And it\u2019s all done.<\/p>\n<h4>Mac<\/h4>\n<p>First you\u2019ll need to make sure you have a c-compiler on your system. In Terminal, enter:<\/p>\n<pre><code>xcode-select --install\n<\/code><\/pre>\n<p>After that, install homebrew from <a href=\"https:\/\/brew.sh\/\">https:\/\/brew.sh\/<\/a>.<\/p>\n<p>Update your PATH variable so that homebrew packages are used before system packages:<\/p>\n<div>\n<pre><code>echo \"export PATH=\/usr\/local\/bin:\/usr\/local\/sbin:$PATH\" >> ~\/.bashrc\n\nsource ~\/.bashrc\n<\/code><\/pre>\n<\/div>\n<p>Install Python:<\/p>\n<pre><code>brew install python\n<\/code><\/pre>\n<p>And then make sure everything is updated:<\/p>\n<pre><code>brew update; brew upgrade python\n<\/code><\/pre>\n<p>After that\u2019s done, just install Scrapy using pip:<\/p>\n<pre><code>pip install Scrapy\n<\/code><\/pre>\n<p>><\/p>\n<h3>Overview Of Scrapy, How The Pieces Fit Together, Parsers, Spiders, Etc<\/h3>\n<p>You will be writing a script called a \u2018Spider\u2019 for Scrapy to run, but don\u2019t worry, Scrapy spiders aren\u2019t scary at all despite their name. The only similarity Scrapy spiders and real spiders have are that they like to crawl on the web.<\/p>\n<p>Inside the spider is a <code>class<\/code> that you define that tells Scrapy what to do. For example, where to start crawling, the types of requests it makes, how to follow links on pages, and how it parses data. You can even add custom functions to process data as well, before outputting back into a file.<\/p>\n<div><\/div>\n<h3>Writing Your First Spider, Write A Simple Spider To Allow For Hands-on Learning<\/h3>\n<p>To start our first spider, we need to first create a Scrapy project. To do this, enter this into your command line:<\/p>\n<pre><code>scrapy startproject oscars\n<\/code><\/pre>\n<p>This will create a folder with your project.<\/p>\n<p>We\u2019ll start with a basic spider. The following code is to be entered into a python script. Open a new python script in <code>\/oscars\/spiders<\/code> and name it <code>oscars_spider.py<\/code><\/p>\n<p>We\u2019ll import Scrapy.<\/p>\n<pre><code>import scrapy\n<\/code><\/pre>\n<p>We then start defining our Spider class. First, we set the name and then the domains that the spider is allowed to scrape. Finally, we tell the spider where to start scraping from.<\/p>\n<div>\n<pre><code>class OscarsSpider(scrapy.Spider):\n   name = \"oscars\"\n   allowed_domains = [\"en.wikipedia.org\"]\n   start_urls = ['https:\/\/en.wikipedia.org\/wiki\/Academy_Award_for_Best_Picture']\n<\/code><\/pre>\n<\/div>\n<div><\/div>\n<p>Next, we need a function which will capture the information that we want. For now, we\u2019ll just grab the page title. We use CSS to find the tag which carries the title text, and then we extract it. Finally, we return the information back to Scrapy to be logged or written to a file.<\/p>\n<pre><code>def parse(self, response):\n   data = {}\n   data['title'] = response.css('title::text').extract()\n   yield data\n<\/code><\/pre>\n<p>Now save the code in <code>\/oscars\/spiders\/oscars_spider.py<\/code><\/p>\n<p>To run this spider, simply go to your command line and type:<\/p>\n<pre><code>scrapy crawl oscars\n<\/code><\/pre>\n<p>You should see an output like this:<\/p>\n<div>\n<pre><code>2019-05-02 14:39:31 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: oscars)\n...\n2019-05-02 14:39:32 [scrapy.core.engine] DEBUG: Crawled (200) <get https:> (referer: None)\n2019-05-02 14:39:34 [scrapy.core.engine] DEBUG: Crawled (200) <get https:> (referer: None)\n2019-05-02 14:39:34 [scrapy.core.scraper] DEBUG: Scraped from \n{'title': ['Academy Award for Best Picture - Wikipedia']}\n2019-05-02 14:39:34 [scrapy.core.engine] INFO: Closing spider (finished)\n2019-05-02 14:39:34 [scrapy.statscollectors] INFO: Dumping Scrapy stats:\n{'downloader\/request_bytes': 589,\n 'downloader\/request_count': 2,\n 'downloader\/request_method_count\/GET': 2,\n 'downloader\/response_bytes': 74517,\n 'downloader\/response_count': 2,\n 'downloader\/response_status_count\/200': 2,\n 'finish_reason': 'finished',\n 'finish_time': datetime.datetime(2019, 5, 2, 7, 39, 34, 264319),\n 'item_scraped_count': 1,\n 'log_count\/DEBUG': 3,\n 'log_count\/INFO': 9,\n 'response_received_count': 2,\n 'robotstxt\/request_count': 1,\n 'robotstxt\/response_count': 1,\n 'robotstxt\/response_status_count\/200': 1,\n 'scheduler\/dequeued': 1,\n 'scheduler\/dequeued\/memory': 1,\n 'scheduler\/enqueued': 1,\n 'scheduler\/enqueued\/memory': 1,\n 'start_time': datetime.datetime(2019, 5, 2, 7, 39, 31, 431535)}\n2019-05-02 14:39:34 [scrapy.core.engine] INFO: Spider closed (finished)\n<\/get><\/get><\/code><\/pre>\n<\/div>\n<p>Congratulations, you\u2019ve built your first basic Scrapy scraper!<\/p>\n<p>Full code:<\/p>\n<div>\n<pre><code>import scrapy\n\nclass OscarsSpider(scrapy.Spider):\n   name = \"oscars\"\n   allowed_domains = [\"en.wikipedia.org\"]\n   start_urls = [\"https:\/\/en.wikipedia.org\/wiki\/Academy_Award_for_Best_Picture\"]\n\n   def parse(self, response):\n       data = {}\n       data['title'] = response.css('title::text').extract()\n       yield data\n<\/code><\/pre>\n<\/div>\n<p>Obviously, we want it to do a little bit more, so let\u2019s look into how to use Scrapy to parse data.<\/p>\n<p>First, let\u2019s get familiar with the Scrapy shell. The Scrapy shell can help you test your code to make sure that Scrapy is grabbing the data you want.<\/p>\n<p>To access the shell, enter this into your command line:<\/p>\n<div>\n<pre><code>scrapy shell \u201chttps:\/\/en.wikipedia.org\/wiki\/Academy_Award_for_Best_Picture\u201d\n<\/code><\/pre>\n<\/div>\n<p>This will basically open the page that you\u2019ve directed it to and it will let you run single lines of code. For example, you can view the raw HTML of the page by typing in:<\/p>\n<pre><code>print(response.text)\n<\/code><\/pre>\n<p>Or open the page in your default browser by typing in:<\/p>\n<pre><code>view(response)\n<\/code><\/pre>\n<p>Our goal here is to find the code that contains the information that we want. For now, let\u2019s try to grab the movie title names only.<\/p>\n<div><\/div>\n<p>The easiest way to find the code we need is by opening the page in our browser and inspecting the code. In this example, I am using Chrome DevTools. Just right-click on any movie title and select \u2018inspect\u2019:<\/p>\n<figure><a href=\"https:\/\/i0.wp.com\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/863f5db5-6a48-48e5-a997-3846a7801224\/chrome-devtools.png?ssl=1\"><\/p>\n<p>    <img data-recalc-dims=\"1\" decoding=\"async\" srcset=\"http:\/\/www.taterboy.com\/blog\/wp-content\/uploads\/2019\/07\/chrome-devtools.png 400w,\n\t\t\t        https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_auto\/w_800\/https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/863f5db5-6a48-48e5-a997-3846a7801224\/chrome-devtools.png 800w,\n\t\t\t        https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_auto\/w_1200\/https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/863f5db5-6a48-48e5-a997-3846a7801224\/chrome-devtools.png 1200w,\n\t\t\t        https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_auto\/w_1600\/https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/863f5db5-6a48-48e5-a997-3846a7801224\/chrome-devtools.png 1600w,\n\t\t\t        https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_auto\/w_2000\/https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/863f5db5-6a48-48e5-a997-3846a7801224\/chrome-devtools.png 2000w\" src=\"https:\/\/i0.wp.com\/www.taterboy.com\/blog\/wp-content\/uploads\/2019\/07\/chrome-devtools.png?w=900\" sizes=\"100vw\" alt=\"Using Chrome DevTools to inspect HTML and CSS\"><\/a><figcaption>\n      Chrome DevTools window. (<a href=\"https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/863f5db5-6a48-48e5-a997-3846a7801224\/chrome-devtools.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>As you can see, the Oscar winners have a yellow background while the nominees have a plain background. There\u2019s also a link to the article about the movie title, and the links for movies end in <code>film)<\/code>. Now that we know this, we can use a CSS selector to grab the data. In the Scrapy shell, type in:<\/p>\n<div>\n<pre><code>response.css(r\"tr[style='background:#FAEB86'] a[href*='film)']\").extract()\n<\/code><\/pre>\n<\/div>\n<p>As you can see, you now have a list of all the Oscar Best Picture Winners!<\/p>\n<div>\n<pre><code>> response.css(r\"tr[style='background:#FAEB86'] a[href*='film']\").extract()\n['<a href=\"\/wiki\/Wings_(1927_film)\" title=\"Wings (1927 film)\">Wings<\/a>', \n...\n '<a href=\"\/wiki\/Green_Book_(film)\" title=\"Green Book (film)\">Green Book<\/a>', '<a href=\"\/wiki\/Jim_Burke_(film_producer)\" title=\"Jim Burke (film producer)\">Jim Burke<\/a>']\n<\/code><\/pre>\n<\/div>\n<p>Going back to our main goal, we want a list of the Oscar winners for best picture, along with their director, starring actors, release date, and run time. To do this, we need Scrapy to grab data from each of those movie pages.<\/p>\n<div><\/div>\n<p>We\u2019ll have to rewrite a few things and add a new function, but don\u2019t worry, it\u2019s pretty straightforward.<\/p>\n<p>We\u2019ll start by initiating the scraper the same way as before.<\/p>\n<div>\n<pre><code>import scrapy, time\n\nclass OscarsSpider(scrapy.Spider):\n   name = \"oscars\"\n   allowed_domains = [\"en.wikipedia.org\"]\n   start_urls = [\"https:\/\/en.wikipedia.org\/wiki\/Academy_Award_for_Best_Picture\"]\n<\/code><\/pre>\n<\/div>\n<p>But this time, two things will change. First, we\u2019ll import <code>time<\/code> along with <code>scrapy<\/code> because we want to create a timer to restrict how fast the bot scrapes. Also, when we parse the pages the first time, we want to only get a list of the links to each title, so we can grab information off those pages instead.<\/p>\n<div>\n<pre><code>def parse(self, response):\n   for href in response.css(r\"tr[style='background:#FAEB86'] a[href*='film)']::attr(href)\").extract():\n       url = response.urljoin(href)\n       print(url)\n       req = scrapy.Request(url, callback=self.parse_titles)\n       time.sleep(5)\n       yield req\n<\/code><\/pre>\n<\/div>\n<p>Here we make a loop to look for every link on the page that ends in <code>film)<\/code> with the yellow background in it and then we join those links together into a list of URLs, which we will send to the function <code>parse_titles<\/code> to pass further. We also slip in a timer for it to only request pages every 5 seconds. Remember, we can use the Scrapy shell to test our <em>response.css<\/em> fields to make sure we\u2019re getting the correct data!<\/p>\n<div>\n<pre><code>def parse_titles(self, response):\n   for sel in response.css('html').extract():\n       data = {}\n       data['title'] = response.css(r\"h1[id='firstHeading'] i::text\").extract()\n       data['director'] = response.css(r\"tr:contains('Directed by') a[href*='\/wiki\/']::text\").extract()\n       data['starring'] = response.css(r\"tr:contains('Starring') a[href*='\/wiki\/']::text\").extract()\n       data['releasedate'] = response.css(r\"tr:contains('Release date') li::text\").extract()\n       data['runtime'] = response.css(r\"tr:contains('Running time') td::text\").extract()\n   yield data\n<\/code><\/pre>\n<\/div>\n<p>The real work gets done in our <code>parse_data<\/code> function, where we create a dictionary called <code>data<\/code> and then fill each key with the information we want. Again, all these selectors were found using Chrome DevTools as demonstrated before and then tested with the Scrapy shell.<\/p>\n<p>The final line returns the data dictionary back to Scrapy to store.<\/p>\n<p>Complete code:<\/p>\n<div>\n<pre><code>import scrapy, time\n\nclass OscarsSpider(scrapy.Spider):\n   name = \"oscars\"\n   allowed_domains = [\"en.wikipedia.org\"]\n   start_urls = [\"https:\/\/en.wikipedia.org\/wiki\/Academy_Award_for_Best_Picture\"]\n\n   def parse(self, response):\n       for href in response.css(r\"tr[style='background:#FAEB86'] a[href*='film)']::attr(href)\").extract():\n           url = response.urljoin(href)\n           print(url)\n           req = scrapy.Request(url, callback=self.parse_titles)\n           time.sleep(5)\n           yield req\n\n   def parse_titles(self, response):\n       for sel in response.css('html').extract():\n           data = {}\n           data['title'] = response.css(r\"h1[id='firstHeading'] i::text\").extract()\n           data['director'] = response.css(r\"tr:contains('Directed by') a[href*='\/wiki\/']::text\").extract()\n           data['starring'] = response.css(r\"tr:contains('Starring') a[href*='\/wiki\/']::text\").extract()\n           data['releasedate'] = response.css(r\"tr:contains('Release date') li::text\").extract()\n           data['runtime'] = response.css(r\"tr:contains('Running time') td::text\").extract()\n       yield data\n<\/code><\/pre>\n<\/div>\n<p>Sometimes we will want to <a href=\"https:\/\/www.scraperapi.com\/blog\/the-10-best-rotating-proxy-services-for-web-scraping\">use proxies<\/a> as websites will try to block our attempts at scraping.<\/p>\n<p>To do this, we only need to change a few things. Using our example, in our <code>def parse()<\/code>, we need to change it to the following:<\/p>\n<div>\n<pre><code>def parse(self, response):\n   for href in (r\"tr[style='background:#FAEB86'] a[href*='film)']::attr(href)\").extract()\n:\n       url = response.urljoin(href)\n       print(url)\n       req = scrapy.Request(url, callback=self.parse_titles)\n       req.meta['proxy'] = \"http:\/\/yourproxy.com:80\"\n       yield req\n<\/code><\/pre>\n<\/div>\n<p>This will route the requests through your proxy server.<\/p>\n<h3>Deployment And Logging, Show How To Actually Manage A Spider In Production<\/h3>\n<p>Now it is time to run our spider. To make Scrapy start scraping and then output to a CSV file, enter the following into your command prompt:<\/p>\n<pre><code>scrapy crawl oscars -o oscars.csv\n<\/code><\/pre>\n<p>You will see a large output, and after a couple of minutes, it will complete and you will have a CSV file sitting in your project folder.<\/p>\n<h3>Compiling Results, Show How To Use The Results Compiled In The Previous Steps<\/h3>\n<p>When you open the CSV file, you will see all the information we wanted (sorted out by columns with headings). It\u2019s really that simple.<\/p>\n<figure><a href=\"https:\/\/i0.wp.com\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/b212d5cc-26c6-45d1-b960-a4979c315065\/resultsscreen.png?ssl=1\"><\/p>\n<p>    <img data-recalc-dims=\"1\" decoding=\"async\" srcset=\"http:\/\/www.taterboy.com\/blog\/wp-content\/uploads\/2019\/07\/resultsscreen.png 400w,\n\t\t\t        https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_auto\/w_800\/https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/b212d5cc-26c6-45d1-b960-a4979c315065\/resultsscreen.png 800w,\n\t\t\t        https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_auto\/w_1200\/https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/b212d5cc-26c6-45d1-b960-a4979c315065\/resultsscreen.png 1200w,\n\t\t\t        https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_auto\/w_1600\/https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/b212d5cc-26c6-45d1-b960-a4979c315065\/resultsscreen.png 1600w,\n\t\t\t        https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_auto\/w_2000\/https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/b212d5cc-26c6-45d1-b960-a4979c315065\/resultsscreen.png 2000w\" src=\"https:\/\/i0.wp.com\/www.taterboy.com\/blog\/wp-content\/uploads\/2019\/07\/resultsscreen.png?w=900\" sizes=\"100vw\" alt=\"A CSV of Oscar winning movies and associated information\"><\/a><figcaption>\n      Oscar winning movies list and information. (<a href=\"https:\/\/cloud.netlifyusercontent.com\/assets\/344dbf88-fdf9-42bb-adb4-46f01eedd629\/b212d5cc-26c6-45d1-b960-a4979c315065\/resultsscreen.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>With data scraping, we can obtain almost any custom dataset that we want, as long as the information is publicly available. What you want to do with this data is up to you. This skill is extremely useful for doing market research, keeping information on a website updated, and many other things.<\/p>\n<p>It\u2019s fairly easy to set up your own web scraper to obtain custom datasets on your own, however, always remember that there might be other ways to obtain the data that you need. Businesses invest a lot into providing the data that you want, so it\u2019s only fair that we respect their terms and conditions.<\/p>\n<h4>Additional Resources For Learning More About Scrapy And Web Scraping In General<\/h4>\n<ul>\n<li><a href=\"https:\/\/scrapy.org\/\">The Official Scrapy Website<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/scrapy\/scrapy\">Scrapy\u2019s GitHub Page<\/a><\/li>\n<li>\u201c<a href=\"https:\/\/www.scraperapi.com\/blog\/the-10-best-web-scraping-tools\">The 10 Best Data Scraping Tools and Web Scraping Tools<\/a>,\u201d Scraper API<\/li>\n<li>\u201c<a href=\"https:\/\/www.scraperapi.com\/blog\/5-tips-for-web-scraping\">5 Tips For Web Scraping Without Getting Blocked or Blacklisted<\/a>,\u201d Scraper API<\/li>\n<li><a href=\"https:\/\/github.com\/scrapy\/parsel\">Parsel<\/a>, a Python library to use regular expressions to extract data from HTML.<\/li>\n<\/ul>\n<div>\n  <img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/www.taterboy.com\/blog\/wp-content\/uploads\/2019\/07\/logo-red-12.png?w=900\" alt=\"Smashing Editorial\"><span>(dm, yk, il)<\/span>\n<\/div>\n<\/article>\n<p class=\"wpematico_credit\"><small>Powered by <a href=\"http:\/\/www.wpematico.com\" target=\"_blank\">WPeMatico<\/a><\/small><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Ultimate Guide To Building Scalable Web Scrapers With Scrapy The Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00 Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your&#8230;<a class=\"moretag\" href=\"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/\"> Read the full article&#8230;<\/a><\/p>\n","protected":false},"author":4,"featured_media":68181,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[371],"tags":[],"class_list":["post-68180","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-user-experience"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.0.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"The Ultimate Guide To Building Scalable Web Scrapers With ScrapyThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your browser can\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Guest Contribution\"\/>\n\t<meta name=\"keywords\" content=\"user experience\" \/>\n\t<link rel=\"canonical\" href=\"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.0.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"Design for Immersive Technologies | User Experience for Games, Virtual Reality (VR) and Augmented\/Mixed Reality (AR\/MR)\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies\" \/>\n\t\t<meta property=\"og:description\" content=\"The Ultimate Guide To Building Scalable Web Scrapers With ScrapyThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your browser can\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-07-17T11:20:25+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-07-17T11:20:25+00:00\" \/>\n\t\t<meta name=\"twitter:card\" content=\"summary\" \/>\n\t\t<meta name=\"twitter:title\" content=\"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies\" \/>\n\t\t<meta name=\"twitter:description\" content=\"The Ultimate Guide To Building Scalable Web Scrapers With ScrapyThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your browser can\" \/>\n\t\t<script type=\"application\/ld+json\" class=\"aioseo-schema\">\n\t\t\t{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#article\",\"name\":\"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies\",\"headline\":\"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy\",\"author\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/author\\\/guestcontribution\\\/#author\"},\"publisher\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/#organization\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/www.taterboy.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/07\\\/chrome-devtools.png?fit=400%2C177&ssl=1\",\"width\":400,\"height\":177},\"datePublished\":\"2019-07-17T04:20:25-07:00\",\"dateModified\":\"2019-07-17T04:20:25-07:00\",\"inLanguage\":\"en-US\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#webpage\"},\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#webpage\"},\"articleSection\":\"User Experience\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#breadcrumblist\",\"itemListElement\":[{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog#listItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.taterboy.com\\\/blog\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/category\\\/user-experience\\\/#listItem\",\"name\":\"User Experience\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/category\\\/user-experience\\\/#listItem\",\"position\":2,\"name\":\"User Experience\",\"item\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/category\\\/user-experience\\\/\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#listItem\",\"name\":\"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog#listItem\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#listItem\",\"position\":3,\"name\":\"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy\",\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/category\\\/user-experience\\\/#listItem\",\"name\":\"User Experience\"}}]},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/#organization\",\"name\":\"Design for Immersive Technologies\",\"description\":\"User Experience for Games, Virtual Reality (VR) and Augmented\\\/Mixed Reality (AR\\\/MR)\",\"url\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/author\\\/guestcontribution\\\/#author\",\"url\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/author\\\/guestcontribution\\\/\",\"name\":\"Guest Contribution\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#authorImage\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d2aad73eb0f48c67b141e9fc978a0e498969791ed623346e361d02d19775b0bb?s=96&d=mm&r=g\",\"width\":96,\"height\":96,\"caption\":\"Guest Contribution\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#webpage\",\"url\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/\",\"name\":\"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies\",\"description\":\"The Ultimate Guide To Building Scalable Web Scrapers With ScrapyThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00Web scraping is a way to grab data from websites without needing access to APIs or the website\\u2019s database. You only need access to the site\\u2019s data \\u2014 as long as your browser can\",\"inLanguage\":\"en-US\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/#website\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#breadcrumblist\"},\"author\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/author\\\/guestcontribution\\\/#author\"},\"creator\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/author\\\/guestcontribution\\\/#author\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/www.taterboy.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/07\\\/chrome-devtools.png?fit=400%2C177&ssl=1\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#mainImage\",\"width\":400,\"height\":177},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/2019\\\/07\\\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\\\/#mainImage\"},\"datePublished\":\"2019-07-17T04:20:25-07:00\",\"dateModified\":\"2019-07-17T04:20:25-07:00\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/\",\"name\":\"Design for Immersive Technologies\",\"description\":\"User Experience for Games, Virtual Reality (VR) and Augmented\\\/Mixed Reality (AR\\\/MR)\",\"inLanguage\":\"en-US\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.taterboy.com\\\/blog\\\/#organization\"}}]}\n\t\t<\/script>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies","description":"The Ultimate Guide To Building Scalable Web Scrapers With ScrapyThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your browser can","canonical_url":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/","robots":"max-image-preview:large","keywords":"user experience","webmasterTools":{"miscellaneous":""},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#article","name":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies","headline":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy","author":{"@id":"https:\/\/www.taterboy.com\/blog\/author\/guestcontribution\/#author"},"publisher":{"@id":"https:\/\/www.taterboy.com\/blog\/#organization"},"image":{"@type":"ImageObject","url":"https:\/\/i0.wp.com\/www.taterboy.com\/blog\/wp-content\/uploads\/2019\/07\/chrome-devtools.png?fit=400%2C177&ssl=1","width":400,"height":177},"datePublished":"2019-07-17T04:20:25-07:00","dateModified":"2019-07-17T04:20:25-07:00","inLanguage":"en-US","mainEntityOfPage":{"@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#webpage"},"isPartOf":{"@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#webpage"},"articleSection":"User Experience"},{"@type":"BreadcrumbList","@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#breadcrumblist","itemListElement":[{"@type":"ListItem","@id":"https:\/\/www.taterboy.com\/blog#listItem","position":1,"name":"Home","item":"https:\/\/www.taterboy.com\/blog","nextItem":{"@type":"ListItem","@id":"https:\/\/www.taterboy.com\/blog\/category\/user-experience\/#listItem","name":"User Experience"}},{"@type":"ListItem","@id":"https:\/\/www.taterboy.com\/blog\/category\/user-experience\/#listItem","position":2,"name":"User Experience","item":"https:\/\/www.taterboy.com\/blog\/category\/user-experience\/","nextItem":{"@type":"ListItem","@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#listItem","name":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy"},"previousItem":{"@type":"ListItem","@id":"https:\/\/www.taterboy.com\/blog#listItem","name":"Home"}},{"@type":"ListItem","@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#listItem","position":3,"name":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy","previousItem":{"@type":"ListItem","@id":"https:\/\/www.taterboy.com\/blog\/category\/user-experience\/#listItem","name":"User Experience"}}]},{"@type":"Organization","@id":"https:\/\/www.taterboy.com\/blog\/#organization","name":"Design for Immersive Technologies","description":"User Experience for Games, Virtual Reality (VR) and Augmented\/Mixed Reality (AR\/MR)","url":"https:\/\/www.taterboy.com\/blog\/"},{"@type":"Person","@id":"https:\/\/www.taterboy.com\/blog\/author\/guestcontribution\/#author","url":"https:\/\/www.taterboy.com\/blog\/author\/guestcontribution\/","name":"Guest Contribution","image":{"@type":"ImageObject","@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#authorImage","url":"https:\/\/secure.gravatar.com\/avatar\/d2aad73eb0f48c67b141e9fc978a0e498969791ed623346e361d02d19775b0bb?s=96&d=mm&r=g","width":96,"height":96,"caption":"Guest Contribution"}},{"@type":"WebPage","@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#webpage","url":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/","name":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies","description":"The Ultimate Guide To Building Scalable Web Scrapers With ScrapyThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your browser can","inLanguage":"en-US","isPartOf":{"@id":"https:\/\/www.taterboy.com\/blog\/#website"},"breadcrumb":{"@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#breadcrumblist"},"author":{"@id":"https:\/\/www.taterboy.com\/blog\/author\/guestcontribution\/#author"},"creator":{"@id":"https:\/\/www.taterboy.com\/blog\/author\/guestcontribution\/#author"},"image":{"@type":"ImageObject","url":"https:\/\/i0.wp.com\/www.taterboy.com\/blog\/wp-content\/uploads\/2019\/07\/chrome-devtools.png?fit=400%2C177&ssl=1","@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#mainImage","width":400,"height":177},"primaryImageOfPage":{"@id":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/#mainImage"},"datePublished":"2019-07-17T04:20:25-07:00","dateModified":"2019-07-17T04:20:25-07:00"},{"@type":"WebSite","@id":"https:\/\/www.taterboy.com\/blog\/#website","url":"https:\/\/www.taterboy.com\/blog\/","name":"Design for Immersive Technologies","description":"User Experience for Games, Virtual Reality (VR) and Augmented\/Mixed Reality (AR\/MR)","inLanguage":"en-US","publisher":{"@id":"https:\/\/www.taterboy.com\/blog\/#organization"}}]},"og:locale":"en_US","og:site_name":"Design for Immersive Technologies | User Experience for Games, Virtual Reality (VR) and Augmented\/Mixed Reality (AR\/MR)","og:type":"article","og:title":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies","og:description":"The Ultimate Guide To Building Scalable Web Scrapers With ScrapyThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your browser can","og:url":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/","article:published_time":"2019-07-17T11:20:25+00:00","article:modified_time":"2019-07-17T11:20:25+00:00","twitter:card":"summary","twitter:title":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy | Design for Immersive Technologies","twitter:description":"The Ultimate Guide To Building Scalable Web Scrapers With ScrapyThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy Daniel Ni 2019-07-16T14:30:59+02:002019-07-17T11:06:54+00:00Web scraping is a way to grab data from websites without needing access to APIs or the website\u2019s database. You only need access to the site\u2019s data \u2014 as long as your browser can"},"aioseo_meta_data":{"post_id":"68180","title":null,"description":null,"keywords":null,"keyphrases":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_custom_url":null,"og_image_custom_fields":null,"og_custom_image_width":null,"og_custom_image_height":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"location":null,"local_seo":null,"created":"2021-02-07 18:55:45","updated":"2026-09-01 16:16:24","focus_keyword":null,"additional_keywords":null,"truseo_locale":null,"primary_term":null,"og_image_url":null,"og_image_width":null,"og_image_height":null,"twitter_image_url":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"Article","isEnabled":true},"graphs":[]},"limit_modified_date":false,"ai":null,"breadcrumb_settings":null,"seo_analyzer_scan_date":null},"aioseo_breadcrumb":"<div class=\"aioseo-breadcrumbs\"><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.taterboy.com\/blog\" title=\"Home\">Home<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.taterboy.com\/blog\/category\/user-experience\/\" title=\"User Experience\">User Experience<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\tThe Ultimate Guide To Building Scalable Web Scrapers With Scrapy\n\t\t<\/span><\/div>","aioseo_breadcrumb_json":[{"label":"Home","link":"https:\/\/www.taterboy.com\/blog"},{"label":"User Experience","link":"https:\/\/www.taterboy.com\/blog\/category\/user-experience\/"},{"label":"The Ultimate Guide To Building Scalable Web Scrapers With Scrapy","link":"https:\/\/www.taterboy.com\/blog\/2019\/07\/the-ultimate-guide-to-building-scalable-web-scrapers-with-scrapy\/"}],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8wzr5-hJG","jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.taterboy.com\/blog\/wp-content\/uploads\/2019\/07\/chrome-devtools.png?fit=400%2C177&ssl=1","_links":{"self":[{"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/posts\/68180","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/comments?post=68180"}],"version-history":[{"count":0,"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/posts\/68180\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/media\/68181"}],"wp:attachment":[{"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/media?parent=68180"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/categories?post=68180"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.taterboy.com\/blog\/wp-json\/wp\/v2\/tags?post=68180"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}