All news with #gcp cloud storage tag

BigLake Metastore Adds Iceberg REST Catalog Support

🔔 Google Cloud announced general availability of BigLake metastore support for the Iceberg REST Catalog, offering a serverless, standards-based runtime metastore that enables interoperability across Iceberg-compatible engines (Spark, Trino) and BigQuery. The service provides credential vending, integrated governance via Dataplex Universal Catalog for lineage and data quality, and a UX console for creating and managing Iceberg catalogs. By removing the need to run custom metastore deployments, BigLake metastore aims to reduce operational overhead while preserving enterprise scale and security.

Lightricks Scales Video Diffusion Training with JAX

🚀 Lightricks rewrote its training stack in JAX to scale high-performance video diffusion models on TPUs after hitting limits with PyTorch/XLA. The migration enabled reliable sharding, fixed FlashAttention and data-loading issues, and delivered linear scaling across small and large TPU pods. These improvements translated to ~40% more training steps per day, faster iteration, and doubled team productivity. Their stack leverages Flax, Optax, Orbax, and the MaxText blueprint for robust, testable, and efficient large-scale training.

Buildertrend Migrates to Memorystore for Valkey at Scale

🚀 Buildertrend describes migrating from Memorystore for Redis to Google Cloud’s managed Memorystore for Valkey to gain native cross‑regional replication, improved networking via Private Service Connect, and performance advantages. The team exported cache data to Google Cloud Storage and seeded Valkey instances to minimize downtime, eliminated a proxy layer, and now uses Valkey for caching, session state, job queues, pub/sub idempotency, and authentication tokens.

GKE Managed Lustre CSI Driver for AI and HPC Workloads

🚀 Managed Lustre on GKE is a managed parallel file system with a CSI driver that brings low-latency, high-throughput POSIX storage to Kubernetes for demanding AI and HPC workloads. It is recommended for training, checkpointing, and small-file patterns where GPUs/TPUs must stay utilized, while Cloud Storage is an alternative for large, higher-latency files. The article presents five operational best practices—data locality, tiering, networking, provisioning, and using Kubernetes Jobs with a shared PVC—to maximize performance and control costs.

Seattle Children’s Uses AI to Accelerate Pediatric Care

🤖 Seattle Children’s partnered with Google Cloud to build Pathway Assistant, a multimodal AI chatbot that turns thousands of pediatric clinical pathway PDFs into conversational, searchable guidance. Using Vertex AI and Gemini, the assistant extracts JSON metadata, parses diagrams and flowcharts, and returns cited answers in seconds. The tool logs clinician feedback to BigQuery and stores source documents in Cloud Storage, enabling continuous improvement of documentation and metadata.

Google releases XProf and Cloud Diagnostics XProf tools

🔧 Google has open-sourced XProf, an upgraded ML profiler, and published the Cloud Diagnostics XProf library to simplify profiling and optimizing models on xPUs. The release brings unified XLA-based profiling across JAX, PyTorch/XLA and TensorFlow/Keras, and supports programmatic and on-demand trace capture. The Cloud Diagnostics library packages dependencies, stores profiles in Google Cloud Storage for retention, provisions TensorBoard on VMs or GKE for faster loading, and produces shareable links for collaborative analysis with tunable machine types for performance.

Storage Insights datasets optimize Cloud Storage spend

📊 Storage Insights datasets put object and bucket metadata into a BigQuery-linked dataset that refreshes automatically, enabling detailed analysis of storage spend, distribution, lifecycle and Autoclass usage. Administrators can run SQL queries or use Gemini Cloud Assist for natural-language insights, then feed outputs into serverless batch operations to relocate, transition or delete data at scale. The feature supports organization-, folder-, project- or bucket-scoped datasets with daily updates and up to 90-day retention for operational and FinOps workflows.