Google Cloud introduces Lightning Engine for Spark
⚡ Google Cloud has announced general availability of Lightning Engine for Managed Service for Apache Spark, available in both serverless and managed cluster modes. Built on Gluten and Velox with Google-engineered enhancements, the engine compiles Spark physical plans into native C++ to accelerate vectorized operations, window functions, and sorting. It also optimizes connectors for Cloud Storage and BigQuery and adds cost-based optimizations like hash table caching, aggregation pushdown, and adaptive shuffle partitioning.
