Scrapy Tutorial: Web Scraping with Python & Scrapy 2.11

Introduction

Scrapy is a powerful and fast open-source web crawling framework written in Python. It is designed for extracting data from websites and can be used for a wide range of purposes, from data mining to monitoring and automated testing. This tutorial walks you through the core concepts of Scrapy 2.11.2, including creating a project, writing a spider, extracting data with selectors, and storing the scraped data.

Installing Scrapy

Before you begin, ensure you have Python 3.6+ installed. Install Scrapy via pip:

pip install Scrapy

For more detailed installation instructions, refer to the official Scrapy installation guide.

Creating a Scrapy Project

Start by creating a new Scrapy project. Open your terminal and run:

scrapy startproject tutorial

This command creates a tutorial directory with the following structure:

tutorial/
    scrapy.cfg
    tutorial/
        __init__.py
        items.py
        middlewares.py
        pipelines.py
        settings.py
        spiders/
            __init__.py

Writing Your First Spider

Spiders are classes that define how to scrape a website. Create a file named quotes_spider.py inside the spiders directory:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

This spider starts at the first page of quotes, extracts each quote’s text, author, and tags, then follows the pagination link to the next page.

Understanding the Spider

• name: Identifies the spider. It must be unique within a project.
• start_urls: A list of URLs where the spider begins to crawl.
• parse: The method called to handle the response downloaded for each URL. It parses the page, extracts data, and returns either dicts with extracted data or new requests.

Running the Spider

Navigate to the project’s top-level directory and run:

scrapy crawl quotes

You will see output with scraped data printed to the console. To save the output to a file, use the -o flag:

scrapy crawl quotes -o quotes.json

Scrapy supports multiple output formats, including JSON, CSV, XML, and more.

Extracting Data with Selectors

Scrapy provides two built-in selector mechanisms: CSS and XPath. The example above uses CSS selectors. You can also use XPath for more complex queries:

response.xpath('//div[@class="quote"]/span[@class="text"]/text()').get()

Selector Methods

• .get(): Returns the first matching result as a string.
• .getall(): Returns a list of all matching results.
• .css() and .xpath(): Return selector lists for further chaining.

Storing Scraped Data

Scrapy can export data to various formats using Feed exports. Common formats include:

• JSON: scrapy crawl quotes -o quotes.json
• CSV: scrapy crawl quotes -o quotes.csv
• XML: scrapy crawl quotes -o quotes.xml

You can also store data in databases or cloud storage by writing custom item pipelines.

Following Links

To crawl multiple pages, you can extract links and create new requests. In the example, we used response.follow() to follow the next page link. This method automatically handles relative URLs and passes the response to the callback.

Conclusion

This tutorial covered the basics of Scrapy 2.11.2: installing, creating a project, writing a spider, extracting data with CSS selectors, and storing results. Scrapy’s true power lies in its extensibility—with middlewares, pipelines, and custom settings, you can handle authentication, caching, and complex crawling logic. For further reading, explore the official Scrapy documentation.