8

AWS SNS 订阅到 HTTP 的过程及消息报文

 1 year ago
source link: https://yanbin.blog/aws-sns-subscribe-http-endpoint-messages/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

AWS SNS 订阅到 HTTP 的过程及消息报文

2023-02-23 | 阅读(10)

AWS SNS(Simple Notification Service) 以消息订阅,推送的方式对组件进行解藕。当有新消息发送到 SNS 主题中,SNS 会向当前所有的订阅者发送一个消息(广播),它本身不像 SQS 那样会存储消息,而只是一个纯粹的消息路由。SNS 消息可以订阅到 Amazon Kinesis Data Firehose, SQS, Lambda, Email, Email-JSON, HTTP, HTTPs, Platform application endpoint, 和 SMS。同邮件列表一样,订阅 SNS 消息也是需要确认的,不然 SNS 消息就可能恶意满天飞。

本文试验如何用 HTTP 端点订阅 SNS 消息,订阅确认,以及发送消息到 SNS 主题后消息推送到 HTTP 端点的细节,重点是了解订阅及被推送过来消息时的 HTTP 报文内容。SNS 的 HTTP 端点订阅需要一个公网上的 HTTP URL, 对 SNS 可见,所以我在本地测试时在家中路由器上加一个端口映射,对 Modem 获得的公有 IP 的 8080 端口访问转发到写此文用所用机器的 8080 端口上。

在本机需要在 8080 端口上启动一个 HTTP 服务以迎接 AWS 消息的到来,比如用 python 3 的话,简单运行命令 python -m http.server 8080。如果不想在 API 代码中分析 HTTP 报文数据,只需打开 Wireshark(过滤条件 tcp.port=8080 && http) 抓取 8080 上的 HTTP 数据通信即可。在 API 代码中如何处理 HTTP 请求数据不是本文的重点。

为方便起见,暂时设定了一个 DNS A 记录  sns.yanbin.blog 指向到 Modem 的 8080,拟定 HTTP 端点为 http://sns.yanbin.blog:8080/。

SNS 到 HTTP 端点的订阅与确认

首先在 Amazon 中创建一个 SNS topic, 命名为 test-topic,然后创建订阅时选择协议为 HTTP,在 Endpoint 中填入 "http://sns.yanbin.blog:8080/", 这里有个选项 "Enable raw message delivery", 先不勾选, 后面讲对比选与不选该项的不同; 在点击 "Create subscription" 按钮之前打开 Wireshark 准备捕获从 AWS 过来的报文。

现在点击 "Create subscription" 按钮,这时候在 "http://sns.yanbin.blog:8080/" 端上收到一个来自 AWS 的 HTTP POST 请求,它的 HTTP 协议文本是

POST / HTTP/1.1
x-amz-sns-message-type: SubscriptionConfirmation
x-amz-sns-message-id: 11ec7b76-af78-467b-97c9-cf9f4ec6508f
x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic
Content-Length: 1517
Content-Type: text/plain; charset=UTF-8
Host: sns.yanbin.blog:8080
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate
  "Type" : "SubscriptionConfirmation",
  "MessageId" : "11ec7b76-af78-467b-97c9-cf9f4ec6508f",
  "Token" : "2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297d731d9a9831a535c1af210e23359570340825d2a5cbd2c7d7e256d2c8709477ceab93973ce3d79617cffa703afe8a533ca5efacac1ffe29ba697e6c7724b18cd100adc4060cc60b6afd3a72a5a94f637815bb417d30f337",
  "TopicArn" : "arn:aws:sns:us-east-1:0123456789012:test-topic",
  "Message" : "You have chosen to subscribe to the topic arn:aws:sns:us-east-1:0123456789012:test-topic.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
  "SubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-east-1:0123456789012:test-topic&Token=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297d731d9a9831a535c1af210e23359570340825d2a5cbd2c7d7e256d2c8709477ceab93973ce3d79617cffa703afe8a533ca5efacac1ffe29ba697e6c7724b18cd100adc4060cc60b6afd3a72a5a94f637815bb417d30f337",
  "Timestamp" : "2023-02-23T05:06:56.065Z",
  "SignatureVersion" : "1",
  "Signature" : "DiEDpqoGs41NfZkNpvBSeY1OOUQGRvAwqQax1Oo51po+h6NLO7F3j2pdTR5jwWvCOIQ4vH5r48+4Ha9VsvGDf81/ATRuMDO5rvfT5p1ptMDRR+leJ9W4YDvzOg96XIggbUl8DYWbKWDiEdmFZQhkgWasILbMPsGt9WJCVRLFekaH18fdT38oTf7w8dulOBVLpWVVHj//gpR6DvGWBRHM9zc6GS8XmhcddGdgfRe50tNfwf8NNpYK8unIvxDZfsWiK9KmBsj7V0bEpzAa8GVnTNyKvfflxcI/XD7wiDcqM7Pivcb9DZAvOiydUgBs2TiRnGcRUmzCMrotnq5XKz4tyQ==",
  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem"

现在 SNS 主题 test-topic 上有一个待确认的订阅

sns-http-1-800x163.png
 

在这个界面上可以选择这个 Pending 的订阅,进行 "Confirm subscription", 确认时需要填入 "Subscription confirmation url", 就是在点击 "Create subscription" 按钮时收到 HTTP 请求中的 "SubscribeURL", 其中包含 Token。

或者直接访问 "SubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-east-1:0123456789012:test-topic&Token=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297fde704abfc2c9ef4c5ba9ebca38627878248a51e55f6ad10793b1a7945993c8556591ca55772e787de1a6f2d06b27a4ff2a9e66db529fa4b317306d6587efa7341b550a811ee95d21091ab4ac99dddc2de3496837e2ebde" 立即进行订阅的确认,GET 或 POST 随意。

以下 curl 命令后的 URL 中 ?= , & 需要用 \ 进行转义

$ curl https://sns.us-east-1.amazonaws.com/\?Action\=ConfirmSubscription\&TopicArn\=arn:aws:sns:us-east-1:0123456789012:test-topic\&Token\=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297fde704abfc2c9ef4c5ba9ebca38627878248a51e55f6ad10793b1a7945993c8556591ca55772e787de1a6f2d06b27a4ff2a9e66db529fa4b317306d6587efa7341b550a811ee95d21091ab4ac99dddc2de3496837e2ebde
<ConfirmSubscriptionResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
  <ConfirmSubscriptionResult>
    <SubscriptionArn>arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277</SubscriptionArn>
  </ConfirmSubscriptionResult>
  <ResponseMetadata>
    <RequestId>058113b3-8c26-5a9a-aed7-4fc3763f869b</RequestId>
  </ResponseMetadata>
</ConfirmSubscriptionResponse>

"Request confirmation" 按钮可重新发送 Type: SubscriptionConfirmation 的消息到 HTTP 端点,以防我们丢失了之前确认订阅用的 "SubscribeURL"

订阅确认后就变得可用

sns-http-2-800x160.png

理解了这个过程之后,我们在实现 http://sns.yanbin.blog:8080/ API 时,当收到 AWS 过来的特定条件的 SNS "SubscriptionConfirmation" 消息后可以立马访问 "SubscribeURL" 来完成订阅的确认。

懂得了发生成后面的对话过程,无论是用 AWS CLI 还是 Terraform 来完成订阅与确认过程就好理解了,比如用 AWS CLI 的订阅过程是

$ aws sns subscribe --topic-arn arn:aws:sns:us-east-1:0123456789012:test-topic --protocol http --notification-endpoint http://sns.yanbin.blog:8080/
{
    "SubscriptionArn": "pending confirmation"
}
$ aws sns confirm-subscription --topic-arn arn:aws:sns:us-east-1:0123456789012:test-topic --token 2336412f37fb687f5d51e6e2425c464de7ccff1bc......
{
    "SubscriptionArn": "arn:aws:sns:us-east-1:0123456789012:test-topic:0cb76e08-16fa-422b-a5da-02d59501a720"
}

 Token 就是订阅时 POST 到 notification-endpoint 去 SubscribeURL 中的 token

SNS 向 HTTP 端点推着消息

HTTP 订阅确认后,我们现在往主题 test-topic 中发送一条消息来观察 http://sns.yanbin.blog:8080/ 上会收到什么 HTTP 报文。直接在 AWS SNS Web 控制台,选择 test-topic 主题,点击按钮 "Publish message"

  1. Subject: "test subject"
  2. Message body: "test message body"
  3. Message attributes: 加一条 "Type: String, Name: key1, Value: value1" 的属性条目

最后点击 "Publish message" 按钮,来到 Wireshark, 收到一个 POST http://sns.yanbin.blog:8080/ 请求, 报文为

POST / HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: 3118e726-bb2f-5816-a923-bb5646a17faa
x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic
x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277
X-Amzn-Trace-Id: Root=1-63f6fb4e-a0d0c0197f004f52ebbb79b7;Sampled=1
Content-Length: 1031
Content-Type: text/plain; charset=UTF-8
Host: sns.yanbin.blog:8080
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate
  "Type" : "Notification",
  "MessageId" : "3118e726-bb2f-5816-a923-bb5646a17faa",
  "TopicArn" : "arn:aws:sns:us-east-1:0123456789012:test-topic",
  "Subject" : "test subject",
  "Message" : "test message body",
  "Timestamp" : "2023-02-23T05:36:14.681Z",
  "SignatureVersion" : "1",
  "Signature" : "E6TrWsmMWts81MYnT54WYJH1bFTubU1bOvPbriIbekwbb4ZYP3l/wOqEbc4Zj9UH9dRkqPwO8NjHKoZWJEL08gOvrcPE50tqLSc9beyT/BVA+1TnqrtGjuFuK+Dnlq0kG3mJYEpTLDDv0IAAj0Mzjdg9Kcho2as9D84F9yeReitJuaBGVTVInKv4WQMyFtIHebhFEqFn5jrcDtWg/KifHe61pifllr/3DyaaZIfE1TZLyaIc4aEzcY/DwAcA/x/0syNlIlFor8prJXZh4ds32kiDYMiQm85qBinVRq+SI1qPXL6f4zWEuGmxxCUCvkUoa2mRjRW/P2DsZ14TeeY/ng==",
  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem",
  "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277",
  "MessageAttributes" : {
    "key1" : {"Type":"String","Value":"value1"}

标准的消息推着,有消息就调用一下订阅时设定的 HTTP 端点, post body 是一个 JSON, Type 为 "Notification", 包含有完整的消息内容,并且带上了友好的退订链接。

上面是订阅是未勾选 "Enable raw message delivery" 选项,我们可以修改现有的订阅,把该选项勾上,然后重新发送一条相同的消息内容,对应的 HTTP 报文是

POST / HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: d9f3a6bd-c69c-55d7-9a80-facd39b5d520
x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic
x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277
x-amz-sns-rawdelivery: true
X-Amzn-Trace-Id: Root=1-63f6fce9-63d62f6f16a38463db9deb42;Sampled=1
Content-Length: 17
Content-Type: text/plain; charset=UTF-8
Host: sns.yanbin.blog:8080
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate
test message body

收到一条只有消息体的 SNS 消息,没有 subject, 还有其他的 Message Attribute 也都被忽略掉了。

为了单一 HTTP 端点能够根据消息 Type 的类型进行复用,还是别勾选上 "Enable raw message delivery" 为好,当然某些时候该选项还是有意义的。

进行了上面的分析后,订阅 SNS 的 HTTP endpoint 中怎么无需多讲了,本文也就此告结了。

最后附一个 S3 Event 到 SNS topic, 然后订阅到 HTTP 的消息样例

POST / HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: c1640ed7-6ae6-5bd8-a492-52a6a132b1ce
x-amz-sns-topic-arn: arn:aws:sns:us-east-1:069762108088:test-topic
x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:069762108088:test-topic:2c6c6284-7faa-44c4-92f3-c7aacae249e8
Content-Length: 1790
Content-Type: text/plain; charset=UTF-8
Host: 71.228.15.105:8080
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate
  "Type" : "Notification",
  "MessageId" : "c1640ed7-6ae6-5bd8-a492-52a6a132b1ce",
  "TopicArn" : "arn:aws:sns:us-east-1:069762108088:test-topic",
  "Subject" : "Amazon S3 Notification",
  "Message" : "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2023-02-23T20:16:19.516Z\",\"eventName\":\"ObjectCreated:Copy\",\"userIdentity\":{\"principalId\":\"AWS:AROARAPRFJK4H4D3QEBDP:[email protected]\"},\"requestParameters\":{\"sourceIPAddress\":\"71.228.15.105\"},\"responseElements\":{\"x-amz-request-id\":\"4NG9RE9GMHMYDEHJ\",\"x-amz-id-2\":\"1w77gaj1JHP4LwOn9Evk71xbh/pYkHyYlo2N959Azg2kQjNmGlFQ/4v57/GHEyIYtZkfmZ73P1xbd7L/Pa3bZTg1z+tgx6cW\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"xxx\",\"bucket\":{\"name\":\"yanbin-bucket\",\"ownerIdentity\":{\"principalId\":\"A377XRXTMT5NTD\"},\"arn\":\"arn:aws:s3:::yanbin-bucket\"},\"object\":{\"key\":\"tfstate.jpg\",\"size\":7850,\"eTag\":\"c1d01a509fd92f78574d312064767284\",\"sequencer\":\"0063F7C9937962C5EA\"}}}]}",
  "Timestamp" : "2023-02-23T20:16:20.924Z",
  "SignatureVersion" : "1",
  "Signature" : "KaBUZTOhvtPksBDoGEYcWX/hS5Z13EPeSI6Z0Fu0vwl0QN0+DNflf/Kj9ws5Va1VwLDpgCitQM7o10Y/uiLD9syl3sru8Al97ZBHblN7mcWrAoKl+5cESYRBznTY3lXMbrfhb28WzHUWCPVGADv2K/R/c7SNRe78MMx1BIW0t96WH1Ttet0Nl2hsdSNnYNN0Os2DRY3MrNalBBWKi9QmW95ZlBvvO9Nq7GIerqgrqA8ShsH0xymyl3MXdM3ryHCpfBo0qSLXcohKB5/E6M3xfJLk8EH+nOlqKgxjRJXgMW9Bz6jh7fRxT45fKAweSstEfGOD/1MrM1tzRhwV/NjMcg==",
  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem",
  "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:069762108088:test-topic:2c6c6284-7faa-44c4-92f3-c7aacae249e8"

S3 的消息以字符串形式包裹在 SNS 消息的 "Message" 体中,如果选择了 "Enable raw message delivery",将只有 S3 的消息部分,即 "{Records:[{...."

Categories: AWS

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK