5

GitHub - udhos/sqs-to-sns: sqs-to-sns is an utility written in Go to forward mes...

 1 year ago
source link: https://github.com/udhos/sqs-to-sns
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

sqs-to-sns

sqs-to-sns is an utility written in Go to forward messages from AWS SQS Queues to AWS SNS Topics.

Created by gh-md-toc

Build and run

git clone https://github.com/udhos/sqs-to-sns

cd sqs-to-sns

./build.sh ;# compile

sts-to-sns ;# run the executable

Configuration

Env vars

export QUEUES=queues.yaml ;# queue list configuration file

#
# Health check
#
# HINT: Recommended for livenessProbe only, not for readinessProbe
#

export HEALTH_ADDR=:8888
export HEALTH_PATH=/health

#
# Prometheus metrics
#

export METRICS_ADDR=:3000
export METRICS_PATH=/metrics
export METRICS_NAMESPACE=sqstosns

#
# These env vars define global defaults for per-queue config
#

export QUEUE_ROLE_ARN=arn:aws:iam::111111111111:role/sqs_consumer
export TOPIC_ROLE_ARN=arn:aws:iam::222222222222:role/sns_producer

export READERS=1                  ;# number of goroutines reading from SQS queue
export WRITERS=1                  ;# number of goroutines writing to SNS topic
export BUFFER=10                  ;# buffer size between readers and writers
export READ_ERROR_COOLDOWN=10s    ;# cooldown holdtime after read errors
export WRITE_ERROR_COOLDOWN=10s   ;# cooldown holdtime after write errors
export DELETE_ERROR_COOLDOWN=10s  ;# cooldown holdtime after delete errors
export EMPTY_RECEIVE_COOLDOWN=10s ;# cooldown holdtime after empty receive
export COPY_ATTRIBUTES=true       ;# enable copying of message attributes from SQS message to SNS message
export DEBUG=true                 ;# enable debug logs

Queue list configuration file

$ cat queues.yaml
- id: q1
  #
  # required
  #
  queue_url: https://sqs.us-east-1.amazonaws.com/111111111111/queue_name1
  topic_arn: arn:aws:sns:us-east-1:222222222222:topic_name1
  #
  # optional
  #
  #queue_role_arn: arn:aws:iam::111111111111:role/sqs_consumer1
  #topic_role_arn: arn:aws:iam::222222222222:role/sns_producer1
  #readers: 1
  #writers: 1
  #buffer: 10
  #error_cooldown_read: 10s
  #error_cooldown_write: 10s
  #error_cooldown_delete: 10s
  #empty_receive_cooldown: 10s
  #copy_attributes: true
  #debug: true
- id: q2
  #
  # required
  #
  queue_url: https://sqs.us-east-1.amazonaws.com/111111111111/queue_name2
  topic_arn: arn:aws:sns:us-east-1:222222222222:topic_name2
  #
  # optional
  #
  #queue_role_arn: arn:aws:iam::111111111111:role/sqs_consumer2
  #topic_role_arn: arn:aws:iam::222222222222:role/sns_producer2
  #readers: 1
  #writers: 1
  #buffer: 10
  #error_cooldown_read: 10s
  #error_cooldown_write: 10s
  #error_cooldown_delete: 10s
  #empty_receive_cooldown: 10s
  #copy_attributes: true
  #debug: true

Roles

You can use the "queue role ARN" to specify a role to access the source queue, and "topic role ARN" to specify a role to access the destination topic.

The role in "queue role ARN" must allow actions sqs:ReceiveMessage and sqs:DeleteMessage to source queue.

The role in "topic role ARN" must allow action sns:Publish to destination topic.

Prometheus Metrics

# HELP sqstosns_buffer How many SQS messages are buffered with us, partitioned by queue.
# TYPE sqstosns_buffer gauge

# HELP sqstosns_receive_total How many SQS receives called, partitioned by queue.
# TYPE sqstosns_receive_total counter

# HELP sqstosns_receive_error_total How many SQS receives errored, partitioned by queue.
# TYPE sqstosns_receive_error_total counter

# HELP sqstosns_receive_empty_total How many SQS empty receives, partitioned by queue.
# TYPE sqstosns_receive_empty_total counter

# HELP sqstosns_receive_messages_total How many SQS messages received, partitioned by queue.
# TYPE sqstosns_receive_messages_total counter

# HELP sqstosns_publish_error_total How many SNS publishes errored, partitioned by queue.
# TYPE sqstosns_publish_error_total counter

# HELP sqstosns_delete_error_total How many SQS deletes errored, partitioned by queue.
# TYPE sqstosns_delete_error_total counter

# HELP sqstosns_delivery_total How many SQS deliveries fully processed, partitioned by queue.
# TYPE sqstosns_delivery_total counter

# HELP sqstosns_delivery_duration_seconds How long it took to fully process the delivery, partitioned by queue.
# TYPE sqstosns_delivery_duration_seconds histogram

Utility to populate SQS queue

Use batch-sqs to send messages to an SQS queue.

CAUTION: With default parameters you would inject your SQS with 10000 messages at a rate of ~1000 messages/sec.

batch-sqs -queueURL https://sqs.us-east-1.amazonaws.com/111111111111/queue_name

2023/04/17 00:27:04 batch-sqs: sent=10000 interval=9.275251836s rate=1111.111111111111 messages/sec
  • Forward messages.
  • Docker image.
  • Message attributes.
  • Helm chart.
  • Multiple queues.
  • Metrics.
  • Health check.
  • Tests.

Docker

Docker hub:

https://hub.docker.com/r/udhos/sqs-to-sns

Run from docker hub:

docker run -p 8080:8080 --rm udhos/sqs-to-sns:0.0.0

Build recipe:

./docker/build.sh

docker push udhos/sqs-to-sns:0.0.0

Helm chart

Using the repository

See https://udhos.github.io/sqs-to-sns/.

Create

mkdir charts
cd charts
helm create sqs-to-sns

Then edit files.

helm lint ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml

Test rendering chart templates locally

helm template sqs-to-sns ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml

Render templates at server

helm install sqs-to-sns ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml --dry-run

Generate files for a chart repository

A chart repository is an HTTP server that houses one or more packaged charts. A chart repository is an HTTP server that houses an index.yaml file and optionally (*) some packaged charts.

(*) Optionally since the package charts could be hosted elsewhere and referenced by the index.yaml file.

docs
├── index.yaml
└── sqs-to-sns-0.1.0.tgz

See script update-charts.sh:

# generate chart package from source
helm package ./charts/sqs-to-sns -d ./docs

# regenerate the index from existing chart packages
helm repo index ./docs --url https://udhos.github.io/sqs-to-sns/

Install

helm install sqs-to-sns ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml

Upgrade

helm upgrade sqs-to-sns ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml --install

Uninstall

helm uninstall sqs-to-sns

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK