Amazon EMR Adds Native Data Quality in Spark
🔍 Amazon EMR now performs native data quality validation inside Apache Spark jobs, letting teams detect and act on bad data inline without separate validation infrastructure. Powered by AWS Glue, Amazon EMR Data Quality uses the same declarative Data Quality Definition Language and APIs as Glue, supports Apache Iceberg, Hudi, and Delta Lake, and runs across EC2, EKS, and Serverless modes. It offers 30+ rule types, dynamic historical comparisons, and ML-based anomaly detection, with actions like failing pipelines, filtering, or quarantining records. Results are produced at dataset, rule, and row level, and the feature is available on emr-spark-8.1.0 in all AWS Commercial and GovCloud regions where EMR is offered.
