Scrapy SQS Pipeline

Each time this pipeline receives an item, it sends it to SQS.

Requirements

• Python 3+
• Scrapy 1.1+
• boto3

Installation

pip3 install scrapy-sqspipeline

Configuration

1. Add the following lines to your Scrapy project settings.

   ITEM_PIPELINES = {
      'sqspipeline.SQSPipeline': 100,
   }
   
   # Either `SQSPIPELINE_QUEUE_URL` or `SQSPIPELINE_QUEUE_NAME` is required.
   SQSPIPELINE_QUEUE_URL='https://sqs.ap-northeast-1.amazonaws.com/xxxxxxxxxx/scrapy-sqspipeline'
   # SQSPIPELINE_QUEUE_NAME=''

2. Use AWS CLI's aws configure command to set up credentials. Alternatively, you can use Scrapy's settings AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION_NAME.