Cannot create SQS subscription to an SNS topic through Cloudformation in LocalStack

cloudformation sns::subscription lambda
sns delivery status logging cloudformation
cloudformation create subscription
cloudwatch sns subscription
localstack init script
aws sns subscription cfn
serverless sns subscription
cross region sns subscription

Using localstack I am trying to create a template that does the following:

  1. Create an SNS topic
  2. Create an SQS queue
  3. Create a subscription that subscribes the SQS queue to the SNS topic.

My docker-compose file looks like this:

version: '3'

services:

  localstack:
    image: localstack/localstack
    container_name: localstack
    environment:
      - SERVICES=sns,sqs,cloudformation
      - DEBUG=1
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - HOSTNAME=localstack
      - AWS_DEFAULT_REGION=eu-west-2
      - AWS_ACCESS_KEY_ID=XX
      - AWS_SECRET_ACCESS_KEY=XX
    ports:
      - "4575:4575"
      - "4576:4576"
      - "4581:4581"
      - "8080:8080"
    volumes:
      - ./config/formation.yml:/usr/stuff/formation.yml
      - ./config/init.sh:/docker-entrypoint-initaws.d/init.sh

My init.sh file looks like this:

#!/bin/bash
aws cloudformation create-stack --stack-name fincorestack --template-body file:///usr/stuff/formation.yml --endpoint-url=http://localstack:4581

And finally my Cloudformation file looks like this:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Test'
Resources:
  MySnsTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: MySnsTopic
  MySnsTopicSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Protocol: sqs
      TopicArn: !Ref MySnsTopic
      Endpoint: !GetAtt
        - MySqsQueue
        - QueueArn
  MySqsQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: MySqsQueue

This produces a very vague 500 Internal Server Error. Since this is localstack and I know that the arns are quite static I tried replacing the contents of the yml file for the subscriptions TopcArn and Endpoint with the following:

TopicArn: arn:aws:sns:eu-west-2:123456789012:MySnsTopic
Endpoint: arn:aws:sqs:elasticmq:000000000000:MySqsQueue

This time I am not getting an error but the subscription does not get created. From the debug output of localstack I can see this:

And the subscription is not created. Is this feature not supported in the Cloudformation implementation of localstack or am I doing something wrong?

I think QueueArn isn't an attribute of AWS::SQS::Queue, It should just be Arn.

AWS::SNS::Subscription Cloudformation support · Issue #1191 , Hi I have issues creating sns subscriptions through Cloudformation. Endpoint: arn:aws:sqs:elasticmq:000000000000:MySqsQueue by creating the subscription when creating the topic but the issue there is that I cannot add any I am also having this issue, but with the serverless localstack plugin. I can get around this issue by creating the subscription when creating the topic but the issue there is that I cannot add any attributes and in my case I need RawMessageDelivery set to true. So my questions are: Is there no support to create AWS::SNS::Subscription resources through Cloudformation or am I doing something wrong?

Found a workaround so posting in case anyone else needs this. I have a feeling the creating subscriptions as a resource is simply not supported by localstack so instead I create them when defining the SNS Topic resource. Changing the yml file to the below did the trick:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Test'
Resources:
  MySqsQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: MySqsQueue

  MySqsQueue:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: MySqsQueue
      Subscription:
        - Protocol: sqs
          Endpoint:
            "Fn::GetAtt": ["MySqsQueue", "Arn"]
          RawMessageDelivery: 'true'

EDIT:

Unfortunately creating subscriptions this way does not allow for attributes to be set. In my case I need RawMessageDelivery=true which is not supported (see here). Quite annoying...

Unable to publish message through SNS->SQS · Issue #83 , Then I've created: queue: aws --endpoint-url=http://localhost:4576 sqs create-​queue --queue-name queue1 topic: aws ERROR:localstack.mock.​generic_proxy:Error forwarding request: occurred (NotFound) when calling the Subscribe operation: Topic with arn You can't perform that action at this time. You can subscribe one or more Amazon SQS queues to an Amazon SNS topic from a list of topics available for the selected queue. Amazon SQS manages the subscription and any necessary permissions. When you publish a message to a topic, Amazon SNS sends the message to every subscribed queue.

This has now been fixed: https://github.com/localstack/localstack/issues/1191

Although TopcArn and Endpoint still need to be hard-coded.

Create a Subscription Between an Amazon SQS Queue and an , How do I create a subscription between my Amazon Simple Queue Service Simple Notification Service (Amazon SNS) topic in AWS CloudFormation? Resolution. Choose one of the following solutions based on your use case: If the SNS topic is in one stack and the SQS queue that will subscribe to that  Using an AWS CloudFormation Template to Create a Topic that Sends Messages to Amazon SQS Queues in the Amazon Simple Notification Service Developer Guide Javascript is disabled or is unavailable in your browser.

Using localstack with AWS CLI, Localstack is a really useful project by Atlassian, which allows for of using AWS command line to work with localstack for S3, SNS, SQS, and DynamoDB. Assuming you're using an OSX/Linux base, check if you have make bash-3.2$ aws --endpoint-url=http://localhost:4575 sns subscribe --topic-arn  To grant Amazon S3 permissions to publish messages to the SNS topic or SQS queue, you attach an AWS Identity and Access Management (IAM) policy to the destination SNS topic or SQS queue. For an example of how to attach a policy to an SNS topic or an SQS queue, see Walkthrough: Configure a bucket for notifications (SNS topic and SQS queue) .

localstack/localstack By, LocalStack spins up the following core Cloud APIs on your local machine: start --docker if $TMPDIR contains a symbolic link that cannot be mounted by Docker. to start Kinesis on the default port, Lambda on port 4569, and SQS on port 4570. used for Serverless apps ( iam , lambda , dynamodb , apigateway , s3 , sns ). In moto, components are often hard-wired in memory (e.g., when forwarding a message on an SNS topic to an SQS queue, the queue endpoint is looked up in a local hash map). In contrast, LocalStack services live in isolation (separate processes communicating via HTTP), which fosters true decoupling and more closely resembles the real cloud

localstack/localstack, LocalStack spins up the following core Cloud APIs on your local machine: in RAM (e.g., when forwarding a message on an SNS topic to an SQS queue, start --docker if $TMPDIR contains a symbolic link that cannot be mounted by Docker.) sh module); add CloudFormation validate-template; fix Lambda execution in  AWS CloudFormation begins to create the MyQueue stack and displays the CREATE_IN_PROGRESS status. When the process is complete, AWS CloudFormation displays the CREATE_COMPLETE status. (Optional) To display the name, URL, and ARN of the queue, choose the name of the stack and then on the next page expand the Outputs section.

Comments
  • I think you are correct so +1 although I still had issues which are now resolved (see answer).