Apache Spark Lineage Now in SageMaker Unified Studio
🔍 Amazon SageMaker now provides Data Lineage for Apache Spark jobs run on Amazon EMR and AWS Glue within IDC-based SageMaker Unified Studio domains. The feature captures schema and column-level transformations from EMR-EC2, EMR-Serverless, EMR-EKS, and Glue, and makes lineage explorable as a visual graph or queryable via APIs. Teams can compare transformation history across Spark jobs to investigate regressions, trace root causes, and assess impact. Spark lineage is available in all existing SageMaker Unified Studio regions.
