asyncio — Scrapy documentation
asyncio
New in version 2.0.
Scrapy has partial support for asyncio
. After you install the asyncio reactor, you may use asyncio
and asyncio
-powered libraries in any coroutine.
Installing the asyncio reactor
To enable asyncio
support, set the :setting:`TWISTED_REACTOR` setting to 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'
.
If you are using CrawlerRunner
, you also need to install the AsyncioSelectorReactor
reactor manually. You can do that using install_reactor()
:
install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')
Using custom asyncio loops
You can also use custom asyncio event loops with the asyncio reactor. Set the :setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event loop class to use it instead of the default asyncio event loop.
Awaiting on Deferreds
When the asyncio reactor isn’t installed, you can await on Deferreds in the coroutines directly. When it is installed, this is not possible anymore, due to specifics of the Scrapy coroutine integration (the coroutines are wrapped into asyncio.Future
objects, not into Deferred
directly), and you need to wrap them into Futures. Scrapy provides two helpers for this:
Tip
If you need to use these functions in code that aims to be compatible with lower versions of Scrapy that do not provide these functions, down to Scrapy 2.0 (earlier versions do not support asyncio
), you can copy the implementation of these functions into your own code.