5

How to deal with custom recorder of AWS Config?

 3 years ago
source link: https://zoph.me/posts/2021-08-15-scrapping-aws-config-resources/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client
Aug 15, 2021

How to deal with custom recorder of AWS Config?

Disclaimer: I’m not a REGEX expert 😄

Lately, I was working for one of my customers on a custom configuration of AWS Config recorder.

My customer wanted to record using AWS Config All resources except a few of them:

  • 'AWS::EC2::Subnet'
  • 'AWS::EC2::VPC'
  • 'AWS::EC2::SecurityGroup'

Unfortunately, the AWS API and Console do not allow you to do this, you should cherry-pick manually which resource you want to record.

The trade-off of this method is that if a new AWS Config resource type came out, it won’t be recorded until you manually select it in your AWS Config recorder setting.

To deal with this, I’ve created a very simple python script that is using AWS documentation web-scrapping to extract supported resource types, and then apply all resources except those which are blacklisted by my customer.

The idea is to schedule this script once a week to be up-to-date.

import boto3
import re
from urllib.request import urlopen
import logging

# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/config.html#ConfigService.Client.put_configuration_recorder

# Purpose:
# Activate Custom AWS Record for AWS Config
# Supported resource type: https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html#supported-resources
# Scraping AWS Docs using: https://realpython.com/python-web-scraping-practical-introduction/

# Get information about the current regional config recorder: aws configservice describe-configuration-recorders --region eu-west-1

# Logging
root = logging.getLogger()
if root.handlers:
    for handler in root.handlers:
        root.removeHandler(handler)
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s',level=logging.INFO)

# Configuration
recorder_name = "<name of your AWS Config Recorder>"
role_arn = "<role arn of AWS Config Recorder>"

# Put here the AWS Config resources type to blacklist
blacklisted_types = [
    'AWS::EC2::Subnet',
    'AWS::EC2::VPC',
    'AWS::EC2::SecurityGroup'
]

def get_config_resources():
    url = "https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html#supported-resources"
    page = urlopen(url)
    html = page.read().decode("utf-8")

    # Target format: AWS::ApiGateway::Stage
    pattern = "AWS::.*"
    match_results = re.findall(pattern, html)
    cleaned_list = []
    count = 0

    for result in match_results:
        # remove HTML tags
        results = re.sub("<.*?>", "", result) 
        # remove ending *
        results = results.replace("*", "")
        # remove space
        results = results.replace(" ", "")
        # remove long items (sentences)
        if len(results) >= 60:
            continue
        # distinct list while preserving order
        list(dict.fromkeys(results))
        # Count items
        count += 1
        # Create the target cleaned list
        cleaned_list.append(results)
    logging.info("Scraped Config supported resources: %s", count)

    return cleaned_list

def apply_custom_recorder(config_resources):
    # Remove Blacklisted resources from the globql list
    result_list = list(set(config_resources) - set(blacklisted_types))

    # counter
    count_result = 0

    # Count resulted number of resource types (minus blacklisted types)
    for type in result_list:
        count_result += 1
    logging.info("result_types: %s", count_result)

    client = boto3.client('config')

    try:
        r = client.put_configuration_recorder(
                ConfigurationRecorder={
                    'name': recorder_name,
                    'roleARN': role_arn,
                    'recordingGroup': {
                        'allSupported': False,
                        'includeGlobalResourceTypes': False,
                        'resourceTypes': result_list
                    }
                }
            )
    except Exception as e:
        logging.error(e)
    logging.info("Response: %s", r)

if __name__ == "__main__":
    config_resources = get_config_resources()
    apply_custom_recorder(config_resources)

Gist version if you want to contribute.

I think this approach could help someone else, so I’m sharing it with you. Don’t hesitate to comment, and enhance it as you like.

That’s all folks!

zoph.


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK