Exceptions — Scrapy documentation
Exceptions
Built-in Exceptions reference
Here’s a list of all exceptions included in Scrapy and their usage.
CloseSpider
- exception scrapy.exceptions.CloseSpider(reason='cancelled')
- This exception can be raised from a spider callback to request the spider to be closed/stopped. Supported arguments:
- Parameters
- reason (str) – the reason for closing
For example:
def parse_page(self, response):
if 'Bandwidth exceeded' in response.body:
raise CloseSpider('bandwidth_exceeded')
DontCloseSpider
- exception scrapy.exceptions.DontCloseSpider
This exception can be raised in a :signal:`spider_idle` signal handler to prevent the spider from being closed.
DropItem
- exception scrapy.exceptions.DropItem
The exception that must be raised by item pipeline stages to stop processing an Item. For more information see Item Pipeline.
IgnoreRequest
- exception scrapy.exceptions.IgnoreRequest
This exception can be raised by the Scheduler or any downloader middleware to indicate that the request should be ignored.
NotConfigured
- exception scrapy.exceptions.NotConfigured
This exception can be raised by some components to indicate that they will remain disabled. Those components include:
- Extensions
- Item pipelines
- Downloader middlewares
- Spider middlewares
The exception must be raised in the component’s __init__
method.
NotSupported
- exception scrapy.exceptions.NotSupported
This exception is raised to indicate an unsupported feature.
StopDownload
New in version 2.2.
- exception scrapy.exceptions.StopDownload(fail=True)
Raised from a bytes_received or headers_received signal handler to indicate that no further bytes should be downloaded for a response.
The fail
boolean parameter controls which method will handle the resulting response:
- If
fail=True
(default), the request errback is called. The response object is available as theresponse
attribute of theStopDownload
exception, which is in turn stored as thevalue
attribute of the receivedFailure
object. This means that in an errback defined asdef errback(self, failure)
, the response can be accessed thoughfailure.value.response
. - If
fail=False
, the request callback is called instead.
In both cases, the response could have its body truncated: the body contains all bytes received up until the exception is raised, including the bytes received in the signal handler that raises the exception. Also, the response object is marked with "download_stopped"
in its Response.flags
attribute.
Note
fail
is a keyword-only parameter, i.e. raising StopDownload(False)
or StopDownload(True)
will raise a TypeError
.
See the documentation for the bytes_received and headers_received signals and the Stopping the download of a Response topic for additional information and examples.