All news with #bigquery tag

BigQuery Agent Analytics: Stream and Analyze Agent Data

📊 Google introduces BigQuery Agent Analytics, an ADK plugin that streams agent interaction events into BigQuery to capture, analyze, and visualize performance, usage, and cost. The plugin provides a predefined schema and uses the BigQuery Storage Write API for low-latency, high-throughput streaming of requests, responses, and tool calls. Developers can filter and preprocess events (for example, redaction) and build dashboards in Looker Studio or Grafana while leveraging vector search and generative AI functions for deeper analysis.

BigQuery Data Transfer Service Enhancements and Compliance

🔔 The BigQuery Data Transfer Service expands its connector ecosystem with new GA integrations (Oracle, Salesforce, ServiceNow, SFMC, Facebook Ads, and GA4) and preview connectors like Stripe, PayPal, Snowflake, and Hive. Platform improvements include event-driven transfers, incremental ingestion, GAQL-based custom Google Ads reports, and enhanced Oracle scale. Security and compliance gains—EU Data Boundary GA, FedRAMP High, CJIS, access transparency, regional endpoints, and key usage tracking—support regulated workloads. A new consumption-based pricing model applies to third-party connectors once they reach GA.

BigLake Metastore Adds Iceberg REST Catalog Support

🔔 Google Cloud announced general availability of BigLake metastore support for the Iceberg REST Catalog, offering a serverless, standards-based runtime metastore that enables interoperability across Iceberg-compatible engines (Spark, Trino) and BigQuery. The service provides credential vending, integrated governance via Dataplex Universal Catalog for lineage and data quality, and a UX console for creating and managing Iceberg catalogs. By removing the need to run custom metastore deployments, BigLake metastore aims to reduce operational overhead while preserving enterprise scale and security.

TimesFM Integration Brings Forecasting to BigQuery

🕒 Google is integrating the TimesFM time-series foundation model into BigQuery and AlloyDB, enabling zero-shot forecasting on customer data without retraining. AI.FORECAST and AI.EVALUATE are now Generally Available in BigQuery, while AI.DETECT_ANOMALIES is in public preview. TimesFM 2.5 offers improved accuracy and lower latency, supports dynamic context windows up to 15K, and can return historical data with forecasts. AlloyDB preview lets users call TimesFM endpoints hosted on Vertex AI so operational data can be forecasted in-place, preserving data residency and reducing export overhead.

Advancing Text-to-SQL: Gemini's BIRD Benchmark Breakthrough

🚀 Google Cloud reports a new state-of-the-art Single Trained Model Track score on the BIRD benchmark, achieving 76.13 with a fine-tuned Gemini 2.5-pro. The team credits rigorous data filtering, multitask supervised fine-tuning, and test-time self-consistency selection for the gains. These improvements bolster NL2SQL features in AlloyDB AI and BigQuery, and enhance developer tooling such as Gemini Code Assist for reliable SQL generation.

Using BigQuery ML to Solve Lookalike Audiences at Zeotap

🔍 Zeotap and Google Cloud describe a SQL-first approach to building scalable lookalike audiences entirely within BigQuery. They convert low-cardinality categorical features into one-hot and multi-hot vectors, use Jaccard similarity reframed via dot-product and Manhattan norms, and index vectors with BigQuery’s VECTOR_SEARCH. By combining pre-filtering on discriminative features and batching queries, the workflow reduces compute, latency, and cost while avoiding a separate vector database.

BigQuery adds MATCH_RECOGNIZE for row-sequence SQL

🔍 BigQuery now supports MATCH_RECOGNIZE, a SQL clause for identifying ordered patterns across rows and time-series data. It lets analysts express complex sequence logic—using PARTITION BY, ORDER BY, PATTERN, DEFINE and MEASURES—inside a single query without heavy joins or external processing. The feature targets use cases like funnels, fraud detection, log sequencing, and financial pattern detection, and is immediately available to all BigQuery users.

BigQuery AI Functions: Reimagining SQL for the AI Era

🤖 BigQuery is introducing managed AI functions in public preview — AI.IF, AI.CLASSIFY, and AI.SCORE — that let analysts apply generative AI directly inside SQL queries. These functions enable semantic filtering and joins, label-based classification of text and images, and natural-language ranking, while BigQuery applies prompt, query-plan, and endpoint optimizations to reduce LLM calls and control cost. They complement existing Gemini inference functions and remove much of the need for complex prompt tuning or separate model selection, making AI-driven analytics more accessible within familiar SQL workflows.

How BigQuery Brought Vector Search to Analytics at Scale

🔍 In early 2024 Google introduced native vector search in BigQuery, embedding semantic search directly into the data warehouse to remove the need for separate vector databases. Users can create indexes with a simple CREATE VECTOR INDEX statement and run semantic queries via the VECTOR_SEARCH function or through Python integrations like LangChain. BigQuery provides serverless scaling, asynchronous index refreshes, model rebuilds with no downtime, partitioned indexes, and ScaNN-based TreeAH for improved price/performance, while retaining row- and column-level security and a pay-as-you-go pricing model.

Zeotap cuts costs 46% migrating to Bigtable from ScyllaDB

🚀 Zeotap migrated its Customer Data Platform from ScyllaDB to Bigtable to address scaling challenges, operational overhead, and highly spiky workloads. The cloud-native stack—using Dataflow, a home-grown streaming engine, Memorystore as a cache, Bigtable as the hot store, and BigQuery for analytics—delivers predictable low-latency reads and writes at scale. The transition yielded a 46% reduction in TCO and a ~20% drop in operational tasks while enabling sub-second SLAs and faster ML deployment.

Ericsson Secures Data Integrity with Dataplex Governance

🔒 Ericsson has implemented a global data governance framework using Dataplex Universal Catalog on Google Cloud to ensure data integrity, discoverability, and compliance across its Managed Services operation. The program standardized a business glossary, automated quality checks with incident-driven alerts, and visualized column-level lineage to support analytics, AI, and automation at scale. It balances defensive compliance with offensive innovation and embeds stewardship through Ericsson’s Data Operating Model.

Automating FinOps Governance with Workload Manager

🔧 Workload Manager automates FinOps governance by codifying cost-control policies and enforcing them across Google Cloud environments. It supports both predefined checks (for example, bigquery-missing-labels) and custom rules written in Open Policy Agent (OPA) Rego, allowing organization-, folder-, or project-level scans. Scheduled evaluations can export results to BigQuery, trigger notifications (email, Slack, PagerDuty), and feed Looker Studio dashboards for reporting and trend analysis. New pricing reduces scan costs by up to 95% and includes a small free tier to accelerate adoption.

BigQuery's Data Engineering Agent: Automating Pipelines

🔧 The preview of the Data Engineering Agent in BigQuery introduces a Gemini-powered assistant that automates pipeline development, maintenance, and migrations. The agent converts natural-language requirements into SQL, enforces engineering best practices, and supports custom instructions and UDFs to reflect organizational logic. Integrated with Dataplex, it uses governance metadata to improve table descriptions, data quality assertions, and PII-aware handling, and it also generates documentation and troubleshooting guidance. The feature is available in preview via BigQuery Pipelines and the Dataform UI.

Mercado Libre's Spanner-Based Platform for Scale and AI

🚀 Mercado Libre leverages Spanner as the core of a developer-facing platform, exposing consistent, globally-scalable transactions through its internal gateway, Fury. Fury abstracts distributed database complexity and serves both relational and key-value workloads. Integration with BigQuery via Data Boost and Change Streams enables near-real-time analytics and reverse ETL to operational systems.

Log Analytics Query Builder Makes Log SQL Easier for Teams

🔍 The Log Analytics query builder in Google Cloud Console provides a UI-driven way to build and preview SQL-based log queries without hand-coding. It helps DevOps engineers, SREs, and application developers search across fields, infer JSON schemas, select nested values, and apply aggregations via an intuitive interface. Real-time SQL preview and one-click visualizations let users switch to the editor to fine-tune queries and save dashboards.

Integrating Oracle with Google Cloud for AI Automation

🔁 This Google Cloud post explains how enterprises can integrate Oracle Database with cloud-native analytics and AI by moving transactional data into BigQuery. It recommends ingestion patterns such as low-latency Change Data Capture via Datastream, batch staging to Cloud Storage, and notes ODBC/JDBC for interactive queries but not continuous replication. Once data resides in BigQuery, organizations can leverage Gemini-powered features, BigQuery ML, and AI agents (via the Agent Developer Kit) for natural-language exploration, assisted coding, multimodal analysis, and automated workflows across retail and education use cases.

Agent Factory Recap: AI Agents for Data Engineering

🔍 The episode of The Agent Factory reviewed practical AI agents for data engineering and data science, highlighting demos that combine Gemini, BigQuery, Colab Enterprise, and Spanner-based graph queries. It showcased a BigQuery Data Engineering Agent that generates pipelines, time dimensions, and data-quality assertions from SQL, and a Data Science Agent that runs end-to-end anomaly detection in Colab. The post also covered CodeMender for autonomous code security fixes and a creative Spanner+ADK comic demo illustrating multi-region concepts.

SmarterX Builds Custom LLMs with Google Cloud Tools

🔍 SmarterX uses Google Cloud to build custom LLMs that help retailers, manufacturers, and logistics companies manage regulatory compliance across product lifecycles. Using BigQuery, Cloud Storage, Gemini, and Vertex AI, the company ingests, normalizes, and indexes unstructured regulatory and product data, applies RAG and grounding, and trains customer-specific models. The integrated platform empowers subject matter experts to evaluate, correct, and deploy model updates without heavy engineering overhead.

Dataplex Supports Column-Level Lineage for BigQuery

🔍 Dataplex Universal Catalog now captures column-level lineage for BigQuery, extending object-level tracing to granular column transformations at no extra cost. The update provides interactive visual lineage graphs so users can inspect upstream and downstream flows for individual columns, trace origins, and assess downstream impact of modifications. This granularity helps validate authoritative sources for AI/ML features, enforce column-level governance, and improve compliance. It also surfaces freshness and usage metadata to support context-aware agents.

Design Patterns for Scalable AI Agents on Google Cloud

🤖 This post explains how System Integrator partners can build, scale, and manage enterprise-grade AI agents using Google Cloud technologies like Agent Engine, the Agent Development Kit (ADK), and Gemini Enterprise. It summarizes architecture patterns including runtime, memory, the Model Context Protocol (MCP), and the Agent-to-Agent (A2A) protocol, and contrasts managed Agent Engine with self-hosted options such as Cloud Run or GKE. Customer examples from Deloitte and Quantiphi illustrate supply chain and sales automation benefits. The guidance highlights security, observability, persistent memory, and model tuning for enterprise readiness.