All news with #databricks tag

🔍 Databricks has previewed Lakewatch, an agentic SIEM designed to extend the lakehouse into security analytics and offer a lower-cost alternative to traditional SIEMs. The vendor says it will charge for compute rather than data ingestion or storage, claiming up to an 80% reduction in total cost of ownership while retaining years of hot data. Analysts acknowledge the ingestion-cost problem and note potential savings for organizations that retain large volumes, but warn that costs can shift to compute and processing if usage is uncontrolled. Databricks bolstered its security credibility with acquisitions such as Antimatter and SiftD.ai, indicating a broader security roadmap.

🚀 The preview of the Spanner columnar engine enables low-latency serving of Apache Iceberg lakehouse data with Spanner’s horizontal scale and strong consistency. It adds a columnar storage layer and vectorized execution to accelerate analytical scans — Google cites up to 200× faster scans — while isolating heavy analytical queries from transactional workloads. The feature supports on-demand columnar conversion, automatic query routing, and reverse ETL integrations with BigQuery, Databricks, Snowflake and Oracle to make curated analytical data available for real-time applications.

🔁 Back Market migrated its data and core tech stack from AWS-based Snowflake and Databricks to Google Cloud, consolidating all historical and operational data in BigQuery. The team executed a two-week proof of concept and a live double-run migration that kept production on Databricks while writing to cloned BigQuery tables until outputs matched. They replaced AWS DMS with Datastream, implemented hourly batching to control small-file costs, and completed critical switchover in six months. The move halved data processing times, cut CDC costs by 90%, reduced technical debt, and improved observability, governance, and developer productivity.

🧠 Microsoft details a broad expansion of its database portfolio and deeper integration with Microsoft Fabric to simplify data architectures and accelerate AI. Key launches include general availability of SQL Server 2025, GA of Azure DocumentDB (MongoDB-compatible), the preview of Azure HorizonDB, and Fabric-hosted SaaS databases for SQL and Cosmos DB. OneLake mirroring, Fabric IQ semantic modeling, expanded agent capabilities, and partner integrations (SAP, Salesforce, Databricks, Snowflake, dbt) are positioned to deliver zero-ETL analytics and operational AI at scale.

🔒 Avnet confirmed unauthorized access to externally hosted cloud storage that supported an internal sales tool used in the EMEA region. The company says most stolen files are not easily readable without access to Avnet's proprietary sales tool, which it says was not impacted, while attackers claim they exfiltrated 1.3TB of compressed (7–12TB raw) data. Avnet detected the activity on September 26, rotated secrets across Azure/Databricks, notified authorities, and will contact affected customers and suppliers; the number of potentially impacted individuals remains unknown.

🔒 Databricks has introduced Data Intelligence for Cybersecurity, an AI-driven platform that unifies fragmented security telemetry on its Lakehouse architecture to provide real-time, context-rich threat detection. The offering includes Agent Bricks to build governed AI agents, conversational dashboards, and natural-language queries for nontechnical stakeholders. Early adopters such as Arctic Wolf, Palo Alto Networks, and SAP report sharper detection, lower costs, and faster operations, while Databricks expands integrations across a broad partner ecosystem to challenge established SIEM and analytics vendors.

⚡ Azure's in-house Arm-based Cobalt 100 VMs are now live in 29 regions and designed to deliver higher performance, energy efficiency, and lower costs for a broad set of cloud workloads. Customers and platform vendors — including Databricks, Snowflake, Siemens, and Temenos — report measurable throughput and price-performance gains, while Microsoft services like Teams and Defender for Endpoint show meaningful internal improvements. The post presents these results as validation of an end-to-end systems strategy that couples custom silicon with infrastructure-level optimization.

🔁 Google Cloud and an ecosystem of partners — including Confluent, Databricks, dbt, Fivetran, Informatica, and Snowflake — reaffirm support for the open table format Apache Iceberg to power modern lakehouse architectures. The post highlights Google innovations such as BigLake and a REST Catalog API that unify metadata and enable interoperability across engines like BigQuery, Databricks, and Snowflake. The collaboration aims to reduce data silos, enable time travel and pruning, and accelerate AI-ready analytics.