Amazon Managed Workflows for Apache Airflow (MWAA)
source link: https://aws.amazon.com/managed-workflows-for-apache-airflow/?nc2=h_ql_prod_ap_af
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Amazon Managed Workflows for Apache Airflow (MWAA) – Amazon Web Services
Learn About AWS Pricing
AWS Free Tier
AWS Pricing Calculator
Optimize Your Costs
Documentation
AWS Customer Enablement
AWS Support
AWS Professional Services
AWS IQ
AWS Training and Certification
AWS Managed Services
AWS re:Post
AWS Events and Webinars
AWS Summit Online
AWS Innovate Online Conference
Online Tech Talks
Public Sector Events
AWS Training and Certification Events and Webinars
Amazon Managed Workflows for Apache Airflow (MWAA) is a managed orchestration service for Apache Airflow1 that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale. Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as “workflows.” With Managed Workflows, you can use Airflow and Python to create workflows without having to manage the underlying infrastructure for scalability, availability, and security. Managed Workflows automatically scales its workflow execution capacity to meet your needs, and is integrated with AWS security services to help provide you with fast and secure access to data.
Benefits
Run Airflow with built-in security
With Managed Workflows, your data is secure by default as workloads run in your own isolated and secure cloud environment using Amazon’s Virtual Private Cloud (VPC), and data is automatically encrypted using AWS Key Management Service (KMS). You can control role-based authentication and authorization for Apache Airflow's user interface via AWS Identity and Access Management (IAM), providing users Single Sign-ON (SSO) access for scheduling and viewing workflow executions.
How it works
Amazon Managed Workflows for Apache Airflow (MWAA) orchestrates and schedules your workflows by using Directed Acyclic Graphs (DAGs) written in Python. You provide Managed Workflows an S3 bucket where your DAGs, plugins, and Python dependencies list reside and upload to it, manually or using a code pipeline, to describe and automate the Extract, Transform, Load (ETL), and Learn process. Then, run and monitor your DAGs from the CLI, SDK or Airflow UI.
Use Cases
Coordinate Extract, Transform, and Load (ETL) Jobs
You can use Managed Workflows as an open source alternative to orchestrate multiple ETL jobs involving a diverse set of technologies in an arbitrarily complex ETL workflow. For example, you may want to explore the correlations between online user engagement and forecasted sales revenue and opportunities. You can use Managed Workflows to coordinate multiple AWS Glue, Batch, and EMR jobs to blend and prepare the data for analysis.
1Apache, Apache Airflow, and Airflow are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK