Scrapy-Cookies Tutorial¶

In this tutorial, we’ll assume that Scrapy-Cookies is already installed on your system. If that’s not the case, see Installation guide.

This tutorial will walk you through these tasks:

Use various storage classes in this middleware
Save cookies on disk

Use various storage classes in this middleware¶

Before you start scraping, just put the following code into your settings.py:

DOWNLOADER_MIDDLEWARES.update({
    'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None,
    'scrapy_cookies.downloadermiddlewares.cookies.CookiesMiddleware': 700,
})

With the default settings of this middleware, a in-memory storage will be used.

There is a storage named SQLiteStorage. If you want to use it instead of the in-memory one, simple put the following code below the previous one:

COOKIES_STORAGE = 'scrapy_cookies.storage.sqlite.SQLiteStorage'
COOKIES_SQLITE_DATABASE = ':memory:'

There are other storage classes provided with this middleware, please refer to Storage.

When you implement your own storage, you can set COOKIES_STORAGE to your own one.

Save cookies and restore in your next run¶

By default this middleware would not save the cookies. When you need to keep the cookies for further usage, for example a login cookie, you wish to save the cookies on disk for next run.

This middleware provides this ability with one setting:

COOKIES_PERSISTENCE = True

Most of time the file saved cookies is named cookies under the folder .scrapy. If you want to change it, use this setting:

COOKIES_PERSISTENCE_DIR = 'your-cookies-path'

After these settings, this middleware would load the previous saved cookies in the next run.

Note

Please keep the storage is the same class when you want save the cookies and restore them. The cookies persistence file is not compatible between different storage classes.

Note

This feature depends on the storage class used.

Next steps¶

This tutorial covered only the basics of Scrapy-Cookies, but there’s a lot of other features not mentioned here. Check the What else? section in Scrapy-Cookies at a glance chapter for a quick overview of the most important ones.

You can continue from the section Basic concepts to know more about this middleware, storage and other things this tutorial hasn’t covered. If you prefer to play with an example project, check the Examples section.