AWS SNS - .Net Core Central

Hello everyone and welcome to .Net Cour Central. In this blog, I am going to walk through another AWS serverless feature which is called AWS SNS.

AWS SNS or Simple Notification Service is another fully managed serverless messaging service provided by AWS. We can use SNS both for application to application as well as application to person communication.

The application to application publish-subscribe provides high throughput messaging between distributed systems like microservices.

AWS SNS Details

SNS topics are fanout topics using which we can fan out messages to multiple subscribers. Once we get into the example of how to create an SNS topic and different subscribers it will be much clearer what it means.

Features of AWS SNS

Following are few key features of SNS:

Firstly, reliability: SNS out of box provides retry mechanism of messages. But you can use SQS along with SNS either through dead letter queue design or standard Exchange-Queue pattern to achieve reliability.
Secondly, Ordering: SNS provides FIFO topics to ensure message order. FIFO topics can handle 300 messages or 10 MB per second per FIFO topics.
Thirdly, Filtering: SNS provides powerful message filtering mechanism to reduce filtering logic from consumers.
Fourthly, Subscribers: SNS provides various subscribers, including Lambda, SQS, Kinesis Firehose and HTTP/HTTPS endpoint.
Finally, using SNS we can directly send SMS Text messages or Emails.

In terms of subscribers for application to application communication, SNS provides various subscribers including AWS Lambda, SQS, Kinesis firehose, and HTTP or HTTPS endpoints.

So if you are building any of the serverless features using Lambda, SNS can be added out of the box.

If you are using SQS, SNS can be added out of the box.

But if you are using an EC2 or ECS-based application, then you can integrate with HTTP or HTTPS endpoint. Where when a message is delivered to SNS, your HTTP endpoint or HTTPS will get a callback with the message.

Creating SNS using AWS Console

So now that we have a basic understanding of what is SNS, let us get into the AWS Console and create a new SNS topic.

So in AWS Console just like any other feature, we can search SNS on the top search bar. Which will result in showing SNS in the dropdown. And we can select Simple Notification Service once it shows up. It will take us to the SNS dashboard.

From the dashboard, we can go into the Topics section by clicking the Topics link from the left navigation. And in the Topics page, we can create a new topic using the Create topic button.

On the create the new topic page, we will get multiple options to choose from.

Options

Firstly, we will need to choose the Type of SNS. And we have two options here FIFO (first-in, first-out) or Standard.
- As I noted above, FIFO allows 300 messages published per second.
- And for Standard type, it is the best effort ordering.
- And for Standard type, the subscription protocol can use SQS, Lambda, HTTP, SMS, email, mobile application endpoints.
- For the FIFO, the only subscription protocol supported is available only for SQS.
Secondly, we can pass the name of the topic.
Thirdly, we can pass the optional display name for the topic.
Fourthly, we have a checkbox option to set Content-based message deduplication. [Only for Standard]
Fifthly, SNS provides in-transit message encryption.
Sixthly, SNS provides granular access policy for the topic. In most cases we will use the default selection, which is Basic.
Seventhly, a delivery retry policy for HTTP/HTTPS endpoints. [Only for Standard]
Eighthly, we can enable Delivery status logging into cloudwatch.
Finally, we can provides tags associated to the topic. Which is nothing but a key value pair for group services during reports.

Option for retry policy

The default values for the different options should be good enough for most cases.
The default is three, but if we don’t want to use default retry we can go as much as 100 retries for a particular SNS message.
Retry without delay is the number of retry that will happen without delays, which is set to 0.
And than there is an The minimum delay and maximum delay.
It also have options for minimum and maximum delay retries.
A maximum receive rate
Finaly, it has the retry backup function. It is the type of mechanism that will be used for retry. And it has four options, Linear, Arithmetic, Geometric and Exponential.

Create SNS Subscription

After we create the SNS topic, we can create subscriptions for the topic. For creating subscriptions we can go to the detail page for the topic, and click Create subscription.

When we go to the create subscription page, we are presented with multiple options.

Firstly, the topic ARN will be selected by default
Secondly, we can select the protocol for the subscription. The options available are:
- Amazon Kinesis Data Firehose
- Amazon SQS
- AWS Lambda
- Email
- Email-JSON
- HTTP
- HTTPS
- SMS
Thridly, Subscription filter policy. This is where you can add filter. Which is nothing but a key/value pair of the attribute that you want to use. If we provide this filtering, then the only message which fits this filter criteria will be sent to this particular subscription. The filtering will use message attributes and not the message body.
Finally, there is an option for dead-letter queue, in case we want to send undelivered messages to a queue for post processing.

For that purpose, I will use the AWS Lambda and SQS queue I created as a part of the previous blog post here.

Using SNS through SQS

For testing the SNS we can add the SNS topic to an SQS queue. Through the queue, we can see how the message is being delivered.

For that in the subscriptions, I will create a new subscription, and I will select SQS as the protocol. Once I select SQS, I will select the test-queue from the next Endpoint selection box.

Now I am going to send a message on the SNS topic. For that, I am going to select the topic and click on Publish message.

I am not going to give any subject or time to live, I am just going to send a text “Hello World” and publish this message. And once we publish this message, that message will go to the SQS test-queue. Which will trigger the lambda function to write the message in AWS Cloudwatch.

Lambda with SNS

As I mentioned above, we can use AWS Lambda as well as a subscriber for the SNS. For doing that I will create a new .NET 5 project using the AWS template for SNS.

dotnet new lambda.SNS --name SNSTest

Once the code is created I am just going to go into the folder SNSTest and open the code in Visual Studio 2019.

The code has the main class as Function. This class has the FunctionHandler method, which is the main method of the lambda, and it takes the SNSEvent object as a parameter along with ILambdaContext. This method is just calling ProcessMessageAsync which is a private function.

In the ProcessMessageAsync method, it is just logging the incoming message using the ILambdaContext logger. Which will log the message into AWS Cloudwatch.

using System.Threading.Tasks;

using Amazon.Lambda.Core;
using Amazon.Lambda.SNSEvents;


// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace SNSTest
{
    public class Function
    {
        /// <summary>
        /// Default constructor. This constructor is used by Lambda to construct the instance. When invoked in a Lambda environment
        /// the AWS credentials will come from the IAM role associated with the function and the AWS region will be set to the
        /// region the Lambda function is executed in.
        /// </summary>
        public Function()
        {

        }


        /// <summary>
        /// This method is called for every Lambda invocation. This method takes in an SNS event object and can be used 
        /// to respond to SNS messages.
        /// </summary>
        /// <param name="evnt"></param>
        /// <param name="context"></param>
        /// <returns></returns>
        public async Task FunctionHandler(SNSEvent evnt, ILambdaContext context)
        {
            foreach(var record in evnt.Records)
            {
                await ProcessRecordAsync(record, context);
            }
        }

        private async Task ProcessRecordAsync(SNSEvent.SNSRecord record, ILambdaContext context)
        {
            context.Logger.LogLine($"Processed record {record.Sns.Message}");

            // TODO: Do interesting work based on the new message
            await Task.CompletedTask;
        }
    }
}

Nuget packages

Now the NuGet package that is installed automatically are as follows:

Amazon.Lambda.Core
Amazon.Lambda.Serialization.SystemTextJson
Amazon.Lambda.SNSEvents

Deploying the function

Next, I will build this project, and then I will deploy this lambda function in AWS.

For that I am going to go into the SNSTest folder path and then here I am going to run the following command:

dotnet lambda deploy function SNSTest

One important thing to remember, in the aws-lambda-tools-defaults.json file I will specify my AWS profile and the region for deployment.

{
  "Information": [
    "This file provides default values for the deployment wizard inside Visual Studio and the AWS Lambda commands added to the .NET Core CLI.",
    "To learn more about the Lambda commands with the .NET Core CLI execute the following command at the command line in the project root directory.",
    "dotnet lambda help",
    "All the command line options for the Lambda command can be specified in this file."
  ],
  "profile": "personal",
  "region": "us-east-2",
  "configuration": "Release",
  "framework": "netcoreapp3.1",
  "function-runtime": "dotnetcore3.1",
  "function-memory-size": 256,
  "function-timeout": 30,
  "function-handler": "SNSTest::SNSTest.Function::FunctionHandler"
}

When I execute this command, the CLI will prompt me to select the role and I am going to go ahead and select the lambda-demo role.

I can test this same way as before and verify the log in the AWS Clodwatch for the newly created lambda.

Conclusion

It is extremely easy to configure SNS and connect it to SQS or lambda for handling the message. Now the question is what scenario we are going to use it.

SNS fits very well for broadcast message scenarios. We can create multiple subscriptions to the same SNS topic and broadcast the same messages to multiple subscribers. ANd let them handle the message based on their needs.

The entire process is captured in my YouTube channel here.