All news with #bigquery tag

🧭 This post describes Google's Governance Agent project that automates metadata propagation using column-level lineage, Knowledge Catalog, and BigQuery. It explains how the agent propagates descriptions, glossary terms, policy tags, and trust scores from upstream sources while applying confidence thresholds and conservative grounding. The project provides both a Gradio dashboard and a CLI to support steward review and automated pipelines, and emphasizes that automation is meant to reduce repetitive work, not remove steward oversight.

🧭 BigQuery Graph introduces measures to unify governed metrics with relationship mapping, enabling agents to reason across complex, multi-hop dependencies without ETL. By mapping tables to an in-place property graph and defining MEASURE in the Property Graph DDL, BigQuery resolves graph paths before computing aggregations using GRAPH_EXPAND and AGG. The release includes a visual graph modeler in BigQuery Studio and native Looker integration to keep business metrics at the data layer.

🔍 Malachyte applies attention-based neural networks and LLM-inspired sequence modeling to address the retail "cold start" problem, updating user vectors in real time to personalize search and product pages. By streaming every interaction through Managed Service for Apache Kafka into Bigtable and combining multimodal embeddings, the platform refines predictions and privacy-friendly personalization within 100 milliseconds. Built on Google Cloud's AI stack, the solution leverages GKE, GCE, and Cloud Pub/Sub to enable continuous learning across retailers.

🚀 BigQuery Data Transfer Service (DTS) reduces engineering overhead by automating zero-code data ingestion into BigQuery, enabling teams to shift focus from pipeline maintenance to analytics. Recent additions include Open Lakehouse ingestion to Apache Iceberg, a managed Model Context Protocol (MCP) Server, expanded database connectors (PostgreSQL, MySQL, SQL Server), SaaS connectors (Shopify, Klaviyo, HubSpot, Mailchimp), and a Snowflake migration path. DTS emphasizes free ingestion for many first-party sources, low consumption-based pricing for third-party SaaS, integrated Cloud IAM security, and a 99.99% SLA for resilient data pipelines.

🧭 BigQuery introduces autonomous, history-based query optimizations and an upgraded advanced runtime to improve performance and reduce compute costs without user intervention. These capabilities include enhanced vectorization, short query optimizations, and support for open formats like Apache Iceberg, delivering up to 35% faster queries and 40% lower slot usage in 2025. The platform’s fluid scaling autoscaler enables per-second billing and average cost reductions up to 34%, with built-in safety guardrails to prevent regressions.

🔍 Google Cloud announces general availability of Cortex Framework v7, designed to convert SAP transactional data into AI-ready, semantically rich data products deployed in BigQuery and registered in Knowledge Catalog. The release introduces purpose-built accelerators for SAP ERP and SAP Business Data Cloud, modular Dataform-powered deployments, and incremental, cost‑efficient processing to scale without extra infrastructure. New agent skills include an agentic data product builder to automate custom data product creation and preserve separation between vendor-delivered content and customer customizations.

🗂️ Google Cloud has advanced Conversational Analytics from experiments into enterprise-ready offerings across BigQuery, Looker, and preview support for AlloyDB, Cloud SQL, and Spanner. The platform supports querying data across clouds, Lakehouse and Iceberg catalogs, and integrates into tools like BigQuery Studio, Looker, and Gemini Enterprise. Enterprises gain governance features such as CMEK, VPC, DRZ, and row- and column-level access controls, plus cost and observability tools via OpenTelemetry. Agentic Workflows, anomaly detection, and APIs/SDKs enable embedding conversational agents across applications and workflows.

🚀 SAP and Google Cloud announced general availability of SAP Business Data Cloud Connect for BigQuery, enabling zero-copy, bi-directional access between SAP Business Data Cloud and BigQuery. The integration exposes SAP tables, metadata, and business semantics directly in BigQuery and Knowledge Catalog to accelerate analytics and agentic AI while reducing data replication and costs. Early adopters report faster data pipelines and improved operational insights.

🚀 In the agentic era, organizations must move from passive data stores to proactive systems of action by providing AI agents with trusted business context. Google introduces the Agentic Data Cloud to unify data, models, and operational databases on an AI-native stack, leveraging BigQuery, Spanner, and open standards like Apache Iceberg. The approach reduces latency, operational overhead, and integration gaps that hinder production-grade agentic AI.

🔍 BigQuery announced GA for Autonomous Embedding Generation and AI.SEARCH(), plus a public preview of Hybrid Search to simplify retrieval and analytics over unstructured data. It automates embedding creation (including images), offers large single-query performance gains, and combines semantic and lexical techniques for more precise results. These features integrate into a broader end-to-end document analytics workflow in BigQuery.

🔒 This post introduces the preview of IAM data governance tags for BigQuery column-level security. Built on Google Cloud Resource Manager tags with purpose=DATA_GOVERNANCE, these tags are global, support hierarchical classification up to five levels, and are replicated for disaster recovery. The article explains creating tag keys/values, attaching tags to columns via JSON or SQL, and defining regional BigQuery data policies for masking or raw access. It highlights decoupled governance, regional policy enforcement, and layered security requirements.

🔗 This post introduces the %%bqsql IPython cell magic and the BigFrames library to bridge Python (pandas) and BigQuery SQL within notebooks. It explains setup steps for local or Colab environments, how to enable the BigQuery sandbox, and how to load the bigframes extension. The article walks through a USDA wheat data example showing chained workflows that alternate between Python and BigQuery SQL, plus tips for schema handling and visualization.

🔎 Google Cloud outlines how to integrate Gemini Enterprise telemetry into BigQuery to enable scalable analytics and governance. The article explains pre-computed dashboards, streaming log sinks, and five partitioned telemetry tables for prompts, model responses, user activity, and audit logs. It highlights BigQuery Conversational Analytics, auto-generated schema documentation, and techniques to build executive dashboards and compliance workflows.

🧭 Conversational Analytics in BigQuery is now generally available, enabling business and technical users to query data, run multi-step analyses, and produce visual reports using natural language directly where data resides. Built on Google’s Gemini models and BigQuery’s governed foundation, it offers inspectable answers, context citations, proactive disambiguation, and long-term memory. The feature integrates with Lakehouse sources, supports enterprise security and governance controls, and provides agentic workflows for scheduled monitoring and automated reports.

🔍 Curve partnered with Google Cloud to adopt BigQuery Graph, moving beyond relational joins to perform multi-hop network analysis across billions of connections. By modeling users and shared identifiers as a property graph, they can traverse massive datasets with GQL, combine graph traversals with standard SQL and ML workflows, and avoid costly data migrations. This integration has improved detection, operational efficiency, and enabled plans for real-time signals and visualization.

🧭 This post introduces the preview of BigQuery's new AI.AGG() function, which enables natural-language aggregation over millions of rows of unstructured and multimodal data directly inside SQL. It explains practical uses—analyzing logs, discovering product categories, and summarizing image collections—while showing how AI.AGG() batches inputs, handles NULLs, reports errors, and integrates with other BigQuery AI functions like AI.CLASSIFY(). The write-up outlines best practices for token usage, model endpoint selection, and struct handling to help users deploy AI.AGG() effectively.

📣 Google Cloud now lets you create alerts in Observability Analytics using SQL to query logs and traces. This preview feature runs scheduled SQL queries via BigQuery on telemetry linked datasets and supports row count and boolean conditions. When conditions are met, Cloud Monitoring opens incidents and notifies configured channels. Note that BigQuery execution costs apply under your billing model.

🛠️ Google Cloud announced enhancements to its Observability suite, rebranding Log Analytics as Observability Analytics and bringing trace data and the Observability API to general availability. The update unifies logs and traces, enables SQL queries across telemetry, and allows in-place analysis without duplicating data. Use cases include diagnosing AI agent tool failures and correlating latency with customer impact. Users can link observability buckets to BigQuery and run cross-dataset analytics directly in the Cloud console.

🐍 BigQuery now supports fully managed Python User-Defined Functions (UDFs) in GA, enabling data teams to run custom Python code securely inside BigQuery using SQL or BigQuery DataFrames. The service runs on BigQuery-managed serverless infrastructure that auto-scales and removes the need to manage containers. It provides access to popular Python libraries, vectorized PyArrow processing, configurable container resources, concurrency controls, and streaming logs for observability. Billing is integrated with BigQuery SKUs and supports spend commitments and cost monitoring.

🤖 Google announces expanded Agentic Data Cloud capabilities, introducing new data agents and tools to enable conversational analytics and agent-driven workflows across BigQuery, Lakehouse, AlloyDB, Spanner, and Cloud SQL. The update includes Data Engineering, Data Science, Database Observability, Looker Dashboard, Data Insights, and Deep Research agents, plus developer toolkits like the Data Agent Kit and Managed MCP servers. These features aim to ground agents in real-time enterprise data with unified governance and near-100% accuracy for tasks such as NL-to-SQL conversions and automated pipeline maintenance.