Scrapy from Scratch: Step-by-Step Guide to

Scrapy from Scratch: Step-by-Step Guide to Building Your First Web Scraper

Web scraping is one of the most powerful skills a developer can master. Whether you're gathering data for machine learning, monitoring competitor prices, or building a research dataset, the ability to programmatically extract information from websites opens up endless possibilities. Among the many tools available, Scrapy stands out as the most robust, flexible, and production-ready web crawling framework for Python. In this tutorial, we'll walk through everything you need to know to build your first Scrapy project from scratch.

What Is Scrapy?

Scrapy is an open-source Python framework designed specifically for large-scale web scraping and web crawling. Unlike simple libraries such as requests + BeautifulSoup, Scrapy is a full-featured framework that handles asynchronous requests, automatic throttling, retries, data pipelines, and much more out of the box. It is built on top of Twisted, an asynchronous networking engine, which allows it to make hundreds of concurrent requests efficiently.

Scrapy follows a clear architectural pattern with several core components working together:

Spiders — Classes that define how a site is scraped, including which URLs to follow and how to parse responses.
Engine — The central component that orchestrates data flow between spiders, schedulers, and pipelines.
Scheduler — Manages the queue of requests to be processed.
Downloader — Fetches web pages and returns responses to the engine.
Item Pipelines — Process and clean extracted data, then store it in databases, files, or external systems.
Middlewares — Hooks that modify requests and responses globally, useful for adding proxies, user agents, or retry logic.

Why Scrapy Matters

There are many reasons developers choose Scrapy over alternatives. First, its asynchronous architecture makes it dramatically faster than synchronous scraping libraries. A Scrapy spider can crawl thousands of pages in minutes. Second, Scrapy's modular design means you can plug in custom middlewares, pipelines, and extensions without rewriting your spiders. Third, it has built-in support for exporting data in JSON, CSV, XML, and other formats. Finally, Scrapy scales — the same spider you test locally can be deployed to production using tools like Scrapy Cloud or Scrapyd.

Scrapy is used by companies in e-commerce, finance, news aggregation, and search engine indexing. If you're serious about web scraping, learning Scrapy is an investment that pays off quickly.

Prerequisites and Installation

Before installing Scrapy, make sure you have Python 3.8 or newer installed. It's strongly recommended to use a virtual environment to avoid conflicts with system packages. On most systems, you'll also need build tools for some of Scrapy's dependencies.

# Create and activate a virtual environment
python -m venv scrapyenv
source scrapyenv/bin/activate  # On Windows: scrapyenv\Scripts\activate

# Install Scrapy
pip install scrapy

# Verify installation
scrapy version

If you encounter issues on Linux, you may need to install system dependencies first:

# Ubuntu / Debian
sudo apt-get install python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev

Creating Your First Scrapy Project

Scrapy projects follow a standard directory structure. The scrapy startproject command scaffolds everything for you. Let's create a project that scrapes a book store demo site.

# Create a new Scrapy project
scrapy startproject bookscraper

# Navigate into the project
cd bookscraper

This command generates the following structure:

bookscraper/
├── scrapy.cfg            # Deployment configuration
└── bookscraper/
    ├── __init__.py
    ├── items.py          # Data models
    ├── middlewares.py    # Request/response middlewares
    ├── pipelines.py      # Data processing pipelines
    ├── settings.py       # Project settings
    └── spiders/          # Directory for your spiders
        └── __init__.py

Defining Data Items

Items are containers for the data you scrape. They work like dictionaries but provide a structured schema and validation. Open items.py and define the fields you want to extract.

import scrapy

class BookItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    availability = scrapy.Field()
    rating = scrapy.Field()
    url = scrapy.Field()

Writing Your First Spider

A spider is a Python class that subclasses scrapy.Spider. It must define a name, and either a start_urls list or a start_requests method. The parse method handles the response for each URL. Let's create a spider that scrapes books.toscrape.com, a sandbox site built for scraping practice.

import scrapy
from bookscraper.items import BookItem

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

    def parse(self, response):
        books = response.css("article.product_pod")

        for book in books:
            item = BookItem()
            item["title"] = book.css("h3 a::attr(title)").get()
            item["price"] = book.css("p.price_color::text").get()
            item["availability"] = book.css("p.instock.availability::text").get().strip()
            item["rating"] = book.css("p.star-rating::attr(class)").get().split()[-1]
            item["url"] = response.urljoin(book.css("h3 a::attr(href)").get())
            yield item

        # Follow pagination links
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Save this file as bookscraper/spiders/books_spider.py. Let's break down what's happening here. The start_urls list tells Scrapy where to begin crawling. The parse method receives the downloaded response and uses CSS selectors to extract data. Each yield item sends a populated BookItem through the pipeline. The pagination logic at the bottom uses response.follow to crawl the next page recursively until there are no more pages.

Running the Spider

To run your spider and save the output to a JSON file, use the following command from the project root:

scrapy crawl books -o books.json

The -o flag specifies the output file. Scrapy infers the format from the file extension, so .csv, .xml, and .jsonl all work automatically. You should see Scrapy's logging output showing each request being made, followed by the scraped data saved to your file.

Configuring Settings

The settings.py file controls global behavior. Some of the most important settings to configure are:

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure a realistic user agent
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

# Limit concurrent requests to be polite
CONCURRENT_REQUESTS = 16

# Add a delay between requests
DOWNLOAD_DELAY = 1

# Auto-throttle extension for adaptive delays
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 8.0

# Enable and configure pipelines
ITEM_PIPELINES = {
    "bookscraper.pipelines.BookscraperPipeline": 300,
}

# Retry settings
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]

Building a Data Pipeline

Pipelines process every item yielded by your spiders. They're perfect for cleaning data, removing duplicates, validating fields, and saving to databases. Open pipelines.py and add cleaning logic:

import re

class BookscraperPipeline:
    def process_item(self, item, spider):
        # Clean price: remove currency symbol and convert to float
        if item.get("price"):
            price_str = re.sub(r"[^\d.]", "", item["price"])
            item["price"] = float(price_str)

        # Normalize rating to numeric value
        rating_map = {
            "One": 1, "Two": 2, "Three": 3,
            "Four": 4, "Five": 5
        }
        if item.get("rating"):
            item["rating"] = rating_map.get(item["rating"], 0)

        return item

class DuplicatesPipeline:
    def __init__(self):
        self.seen = set()

    def process_item(self, item, spider):
        if item["title"] in self.seen:
            raise DropItem(f"Duplicate item: {item['title']}")
        self.seen.add(item["title"])
        return item

To enable both pipelines, update your settings:

ITEM_PIPELINES = {
    "bookscraper.pipelines.BookscraperPipeline": 300,
    "bookscraper.pipelines.DuplicatesPipeline": 400,
}

The integer values represent execution order — lower numbers run first.

Handling Dynamic Content with Playwright

Many modern websites render content with JavaScript, which Scrapy's default downloader cannot execute. For these cases, integrate the scrapy-playwright package, which uses a headless browser to render pages before scraping.

pip install scrapy-playwright
playwright install

Update your settings to enable Playwright:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Then use Playwright in your spider by passing playwright=True in the request meta:

import scrapy

class JSSpider(scrapy.Spider):
    name = "js_spider"

    def start_requests(self):
        yield scrapy.Request(
            url="https://quotes.toscrape.com/js/",
            meta={"playwright": True},
            callback=self.parse,
        )

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
            }

Using Middlewares for Proxies and Headers

Down middlewares let you modify every request before it's sent. This is useful for rotating user agents, adding proxies, or attaching authentication headers. Here's a simple middleware that rotates user agents:

import random

class RotateUserAgentMiddleware:
    USER_AGENTS = [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)",
        "Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/605.1.15",
    ]

    def process_request(self, request, spider):
        request.headers["User-Agent"] = random.choice(self.USER_AGENTS)
        return None

Enable it in settings:

DOWNLOADER_MIDDLEWARES = {
    "bookscraper.middlewares.RotateUserAgentMiddleware": 400,
}

Best Practices for Production Scraping

Respect robots.txt — Keep ROBOTSTXT_OBEY = True unless you have explicit permission to ignore it.
Be polite — Use DOWNLOAD_DELAY and CONCURRENT_REQUESTS settings to avoid overwhelming target servers.
Use autothrottle — The AUTOTHROTTLE_ENABLED extension adapts request speed based on server response times.
Cache responses during development — Set HTTPCACHE_ENABLED = True to avoid re-downloading pages while testing selectors.
Handle errors gracefully — Always check if selectors return None before processing, and use try/except blocks for fragile parsing logic.
Log strategically — Use self.logger.info() for important events and adjust LOG_LEVEL to control verbosity.
Store data properly — Use pipelines to write to databases like PostgreSQL, MongoDB, or Elasticsearch rather than relying solely on file exports.
Rotate proxies for large crawls — Use services like ScraperAPI or rotating proxy pools to avoid IP bans.
Test selectors in the Scrapy shell — Run scrapy shell "https://example.com" to interactively test CSS and XPath selectors before writing spider code.
Version your spiders — Keep spiders in version control and document the structure of target sites, since layout changes will break selectors over time.

Debugging with the Scrapy Shell

The Scrapy shell is an interactive environment for testing selectors and inspecting responses. It's invaluable for development:

# Launch the shell with a URL
scrapy shell "https://books.toscrape.com/catalogue/page-1.html"

# Inside the shell, test selectors
response.css("article.product_pod h3 a::attr(title)").getall()
response.xpath("//article[@class='product_pod']/div[@class='product_price']/p/text()").getall()

# Exit the shell
exit()

Exporting Data to a Database

For production use, you'll want to persist data in a database. Here's a pipeline that stores items in SQLite:

import sqlite3

class SQLitePipeline:
    def open_spider(self, spider):
        self.conn = sqlite3.connect("books.db")
        self.cursor = self.conn.cursor()
        self.cursor.execute("""
            CREATE TABLE IF NOT EXISTS books (
                title TEXT,
                price REAL,
                availability TEXT,
                rating INTEGER,
                url TEXT UNIQUE
            )
        """)
        self.conn.commit()

    def process_item(self, item, spider):
        self.cursor.execute(
            "INSERT OR REPLACE INTO books VALUES (?, ?, ?, ?, ?)",
            (item["title"], item["price"], item["availability"],
             item["rating"], item["url"])
        )
        self.conn.commit()
        return item

    def close_spider(self, spider):
        self.conn.close()

Conclusion

Scrapy is a mature, powerful framework that takes web scraping from a fragile scripting exercise to a robust engineering discipline. By following the step-by-step approach in this tutorial, you now have a working project with structured items, a spider with pagination, data cleaning pipelines, configurable settings, and strategies for handling both static and dynamic content. As you build more complex scrapers, remember that the keys to long-term success are respecting the sites you scrape, writing defensive parsing code, and leveraging Scrapy's extensibility through middlewares and pipelines. Start small, test often in the Scrapy shell, and scale up gradually — your future data-hungry applications will thank you for it.